ITL-424723421eb2
Watch the video

Key points
- A citation needs to support the exact clinical claim. Article existence, relevance, scientific accuracy, and translation fidelity are separate checks. [1]
- A real reference is only the beginning. A separate 2026 audit found no confirmed fabricated references among 4,979 citations, but did not test whether those references supported the clinical answers. [2]
- Agreement is not accuracy. In the anesthesiology study, AI–human agreement was 85.8% after merging two rating categories, but 47.5% when all three categories had to match. [1]
- An AI PASS does not remove the need for review. Of 296 citations left unflagged by the broader alert rule, 28 received a human FAIL rating. Human ratings were a comparison reference, not established truth. [1]
- Check the words that can change care. Population, dose, outcome, inequality signs, and translated clinical terms deserve direct comparison with the source passage. [1]
A physician preparing a teaching slide or clinical summary may receive a plausible answer accompanied by a genuine DOI and a second AI system's reassuring assessment. The decision is whether to use the attached claim. The useful next step is to open the source and locate the passage that actually supports it.
The 2026 study by Li and colleagues provides a practical reason to do this. It examined citations linked to clinical concepts and bilingual summaries in an anesthesiology knowledge base. Its central lesson concerns how to review evidence before using it; the study did not test improvements in clinical decisions or patient outcomes. [1]
Move from finding the article to checking the claim
The investigators sampled 570 citation records across 19 subspecialties. Each received one AI assessment and one assigned human assessment. The four domains were article existence, relevance to the assigned question, scientific accuracy of the summary, and preservation of clinical meaning between English and Chinese. [1]
These domains lead to different practical actions. A matching title and DOI establish the identity of a paper. Its methods establish which patients and interventions it studied. Its results establish what was observed. Its discussion may propose an explanation that the study did not directly test. A summary can preserve the bibliographic details while changing one of these relationships.

Figure 1. Four checks that answer different questions. The domains come from the study; the practical prompts are a clinical application of that framework. [1]
The separate 2026 study by Lotan and colleagues helps put this distinction in context. It examined 4,979 citations returned for 150 prompts across five specialties and found no confirmed fabricated references, while identifying three abstract-author attribution errors. However, the authors explicitly did not evaluate whether the cited material supported the clinical claims. Bibliographic verification and claim verification therefore answer different questions; a favorable result for one cannot supply the missing assessment of the other. [2]
Read an agreement percentage with its rating definitions
In the anesthesiology study, the overall ratings were PASS, CONDITIONAL, and FAIL. For the primary, post hoc binary comparison, PASS and CONDITIONAL were combined. Thus, an AI PASS paired with a human CONDITIONAL counted as agreement even though the assessments differed about whether qualification was needed. [1]
Under that binary definition, agreement was 489/570, or 85.8%. Exact agreement across all three categories was 271/570, or 47.5%. The difference comes from asking different questions of the same paired assessments. Neither percentage measures how often a citation was independently proven correct. [1]

Figure 2. The same ratings produce different agreement estimates when the categories change. Error bars show the reported 95% confidence intervals: 79.12–91.75% for binary agreement and 41.93–53.16% for three-category agreement. [1]
The study also reported markedly different chance-adjusted coefficients: binary Cohen's κ was 0.0645 and Gwet's AC1 was 0.8334. They use different chance-agreement calculations and respond differently to unequal category use. Choosing the more reassuring coefficient would obscure the underlying ratings. For the clinician using the output, the more useful question is what the disagreement concerned and whether the source resolves it. [1]
Each citation had only one human reviewer, and reviewers were tied to subspecialties. Some assessments were unblinded. Consequently, a human FAIL is a reason to examine the record, not an independent gold-standard demonstration that the AI was wrong. This qualification matters whenever the study's results are used to describe an AI tool's reliability. [1]
Use alerts to organize work, then examine what they leave out
The broader alert rule flagged both AI CONDITIONAL and FAIL ratings. It selected 274 of 570 citations and captured 46 of the 74 human FAIL ratings. Among the 296 unflagged citations, 28 still received a human FAIL. Conversely, only 46 of the 274 flagged citations were human FAIL: being flagged did not mean that an error had been established. [1]

Figure 3. What the broader alert rule selected and left outside its review list. Human not-FAIL includes both PASS and CONDITIONAL. This is a cross-classification of judgments, not a count of confirmed errors. [1]
Flagging AI FAIL alone created a shorter list, but captured only 5 of the 74 human FAIL ratings. The study did not measure review time or establish which strategy was most efficient. A practical use of alerts is to prioritize records for inspection while retaining appropriate scrutiny of unflagged, clinically consequential claims. An automated label by itself cannot close that decision. [1]
Make the last check clinically specific
The study's retained examples show why a generic “citation verified” message is inadequate. One cited paper addressed transversus abdominis plane block safety and median dose, while the assigned question concerned maximum dose. Another summary reversed the clinical meaning of a TAPSE threshold. In a Chinese summary, “stroke volume” was rendered with the term for cerebral stroke rather than the cardiac term “每搏量.” These examples illustrate failure modes; they do not estimate their frequency. [1]
Before carrying a claim into teaching or practice, identify the exact sentence being used. Then check the source passage for the population, intervention, comparator, outcome, timing, and qualifications that give the sentence meaning. For numerical statements, compare the denominator, units, direction of effect, and inequality sign. For bilingual material, compare clinical meaning rather than fluency alone. This is a practical application of the study's domains, not a separately validated clinical checklist. [1]
Repeated AI checking also needs interpretation. In a later assessment of 112 records, binary ratings repeated more often than exact three-category ratings, and some original FAIL ratings changed. Retrieval conditions were not held fixed, and the actual backend model could not be independently confirmed. A reassuring repeat answer therefore does not substitute for resolving the original source question. [1]
Three takeaways
- Verify the claim–source relationship. A real article and a correct DOI do not establish that the paper answers the question or supports the attached sentence.
- Use AI flags to guide attention. Check clinically consequential unflagged claims too, and treat disagreement as a prompt for source review.
- Preserve meaning through every transformation. Numbers, qualifying language, and translated terms must remain faithful when evidence becomes a summary, slide, or post.
Author disclosure: Juan Li is the first author of the anesthesiology citation-validation study discussed here.
References
- Li J, Huang S, Qiu Y, Chen W, Zhang Y, Zhao X, et al. Human-AI agreement in citation validation of an AI-generated subspecialty anesthesiology knowledge base: cross-sectional paired validation study. J Meta and Res Int. 2026;1:e105947. doi:10.2196/105947.
- Lotan T, Omar M, Barash Y, Kruskal JB, Ahmed M, Brook OR, et al. Reference quality of OpenEvidence across five medical specialties. npj Health Syst. 2026;3:87. doi:10.1038/s44401-026-00142-8.
