Don’t rely on an AI chatbot alone to judge a medical or legal problem; before seeing a doctor, going to court or spending big money, find the source it cites and check it against the original

Free. Each check costs a few minutes to half an hour. Ask the AI to write out the statute, guideline or paper it is relying on, then open the original yourself and find that sentence. The habit to change is acting on something just because it sounds sure.

First medicine. A 2026 Nature Medicine randomized trial enrolled 1298 ordinary people in the UK; the protocol was preregistered. Each got two prepared cases and had to judge the disease and whether to go to emergency or watch at home. Some used an AI chatbot — GPT-4o, Llama 3, or Command R+. Controls used their usual methods, mostly web search or the NHS site. Handed a full case directly, AI named a relevant disease 94.9% of the time and picked the right care setting 56.3% of the time. Ordinary people using the same AI named a relevant disease under 34.5% of the time and picked the right setting under 44.2%. Controls recognized a relevant disease more often than AI users: odds ratio 1.76 (95% CI 1.45–2.13, confidence interval). For recognizing a serious red-flag illness, the odds ratio was 1.57 (1.28–1.92). Judging where to go was similar on both sides. The authors reviewed chat logs; problems were two: users did not tell all symptoms, and when AI named the right disease users did not take it. Next law. A U.S. research team in 2024 asked public models including ChatGPT 4 about U.S. federal case law. ChatGPT 4 had fabricated content in 58% of answers; Llama 2 up to 88%. Fabrication means content that does not match legal fact; the paper calls it “hallucination.” Models cannot tell which of their own lines are made up. When the user’s question carries a wrong legal premise, they often accept it wholesale. The same team in 2025 tested professional legal AI tools for lawyers. Tools that claim not to fabricate still had fabrication in 17% to 33% of answers. It looks worthwhile because AI is always on, free, and speaks with a doctor’s or lawyer’s certainty.

Bean AM, Payne RE, Parsons G, et al. (2026). Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 32(2), 609–615. https://doi.org/10.1038/s41591-025-04074-y；Dahl M, Magesh V, Suzgun M, Ho DE (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis, 16(1), 64–93. https://doi.org/10.1093/jla/laae003；Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE (2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 22(2), 216–242. https://doi.org/10.1111/jels.12413；Kalai AT, Nachum O, Vempala SS, Zhang E (2025). Why Language Models Hallucinate. arXiv:2509.04664（预印本，备注里那项）. https://arxiv.org/abs/2509.04664；反方：Ayers JW, Poliak A, Dredze M, et al. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183(6), 589. https://doi.org/10.1001/jamainternmed.2023.1838

Controversy. A 2023 study drew 195 patient questions from a U.S. forum and had clinicians score without seeing the source. 78.6% of scores rated ChatGPT’s answer better than a doctor’s. But that compared how well the answer was written, not whether the asker’s final judgment was right. The 2026 trial used 2024 models; newer ones may be stronger. Yet the same trial also found that AI’s solo exam score does not predict how ordinary people do with it. Why AI fabricates: OpenAI researchers explained in a 2025 non-peer-reviewed paper that training and scoring reward “guessing an answer”; saying “I don’t know” scores nothing, so when unsure it still speaks with certainty. The check method: have it write the statute name and article number, or the guideline or paper title; open the original on an official site and find that sentence; if you cannot find it, treat it as unsaid. When asking, do not put your conclusion in the question — ask “what does the law say in this situation,” not “have I already won.” Sudden face droop, one-sided weakness, crushing chest pain: do not ask AI first; call 120. See Section 13, Item 3 (face droop, one arm weak) and Section 13, Item 7 (crushing chest pain). No money for a lawyer: Section 7, Item 3 (apply for legal aid). Benefit set to small because only intermediate markers like “disease recognized or not” exist, with no death-reduction numbers. Beneficiaries are you and family.