Patients with rare diseases share a common experience known as the 'diagnostic odyssey.' A November 2025 study in eBioMedicine reported that approximately 25% of patients wait 5–30 years for a diagnosis, and 40% receive an incorrect initial diagnosis. The U.S. National Institutes of Health’s Undiagnosed Diseases Network (UDN) provides more specific figures: among its 90 enrolled patients, the median age at symptom onset was 7.6 months, while the median duration to diagnosis was 7.6 years. The UDN accepts only patients who remain undiagnosed despite extensive prior evaluation—this constitutes the most difficult testbed for human diagnostic capability. In 2025, three studies successively deployed large language models (LLMs) into this testbed, delivering the most realistic performance metrics to date.
## 90 Patient Records: 13.3% vs. 5.6%
A study published in JAMA Network Open in August 2025 by Vanderbilt University Medical Center examined 90 UDN cases ultimately diagnosed between November 2016 and April 2024. Standardized clinical intake documents—comprising clinical history, family history, and summaries of prior evaluations—were fed separately to ChatGPT-4o and the smaller open-source model Llama 3.1 8B, prompting them to generate differential diagnoses. These were then compared against historical clinical review results for the same cases. ChatGPT-4o achieved a hit rate of 13.3% (95% confidence interval: 7.8%–21.9%), Llama 3.1 8B achieved 10.0%, and historical clinical reviews achieved only 5.6%; only ChatGPT-4o’s lead reached statistical significance (P = .001). When broadened to 'helpful diagnoses,' the rates rose to 23.3% and 16.7%, respectively. Cost figures are equally striking: roughly $0.03 per case, completed in five seconds. Inter-rater reliability among reviewers scoring these 90 cases exceeded 88%, meaning the metric used to assess model accuracy itself is robust—the gap between 13.3% and 5.6% is not attributable to scoring noise.
How should we interpret 13.3%? As an absolute value, it is low—over 85% of these diagnostic challenges remain missed. Yet the control group represents expert networks’ best efforts after exhaustive evaluation; 5.6% reflects their ceiling. Where human processes stall, the model uncovers new diagnostic clues at a cost of $0.03 per case—and also suggests next steps to validate its proposed diagnoses. Caveats must be stated clearly: only 90 cases were included, the confidence interval is wide, the control group was reconstructed retrospectively rather than tested head-to-head contemporaneously, and the 2.4-fold difference cannot be interpreted as a prospective expectation.
## 424 Structured Phenotypes: General-Purpose Models Outperform Specialized Tools
Another study from the same network (Studies in Health Technology and Informatics, August 2025) changed the input format: instead of free-text clinical records, UDN clinicians encoded each patient’s clinical features using Human Phenotype Ontology (HPO) terms. On 424 real patients, the study compared ChatGPT-4o, Llama 3.1 8B, and Exomiser—the phenotypically driven bioinformatics gold-standard tool, previously validated in the UK’s 100,000 Genomes Project. Results: ChatGPT-4o achieved a hit rate of 22.4%, Exomiser 13.9%, and Llama 3.1 8B 11.6%; combining ChatGPT-4o with Exomiser raised the hit rate to 30%, significantly exceeding either tool used alone.
Two discoveries here were unprecedented before 2025. First, on structured phenotype inputs, general-purpose LLMs outperformed specialized rare-disease tools. Second, the disease entities identified by LLMs and bioinformatics tools largely differed—hence the 30% combined hit rate exceeds the standalone 22.4%, indicating complementarity rather than redundancy. The study also found that age at onset significantly predicted accuracy: later onset correlated with higher accuracy, possibly because phenotypes become more differentiated with age. This, in turn, suggests that the youngest patients—whose presentations are least typical—still critically depend on deep, human-driven phenotyping. It must also be noted that the 424 HPO term inputs were carefully curated by UDN clinicians; the error introduced during extraction of HPO terms from raw clinical notes remains unmeasured.
## Ten Languages, 4,917 Case Cards: The Performance Gradient Nearly Vanishes
The eBioMedicine study pushed the question further into multilingual settings: researchers built 4,917 clinical case cards (covering 360 genetic disorders and 2,525 phenotypic features) from 706 published papers, leveraging official HPO translations and templated prompts to generate versions in English, Chinese, Japanese, French, German, Spanish, Italian, Dutch, Czech, and Turkish. GPT-4o achieved a top-1 hit rate of 19.9% in English and a top-3 hit rate of 27.0%; for the nine non-English languages, top-1 hit rates ranged from 16.9% to 20.6%, and top-3 rates from 25.4% to 28.6%—the maximum cross-language gap in top-1 hit rate was just three percentage points. Similarly flat performance was observed for Meditron3-70B, a medically fine-tuned model: top-3 hit rate was 20.9% in English and ranged from 19.9% to 24.0% across the other nine languages—no language lagged significantly.
This finding merits comparison with another well-established observation: when inputs are free-text clinical notes, AI performance degrades markedly in low-resource languages, exhibiting a steep language gradient; but when inputs use officially translated, structured ontology terms, the gradient nearly vanishes. In other words, the disparity lies not in the model’s multilingual 'intelligence', but in whether the input side possesses ontology infrastructure. For rare disease diagnosis, this implies that a machine-readable, structured phenotypic description can be reused across languages, across tools, and across institutions—the bottleneck is not model capability, but whether each patient has such a description generated. Of course, this study’s case cards derived from published case reports, which inherently represent more typical cases; publication bias warrants caution.
## Shared Structural Limitations Across All Three Studies
Laying the three studies side-by-side: 90 free-text records yielded 13.3% vs. 5.6% human performance; 424 structured phenotypes yielded 22.4% for LLMs alone and 30% for the LLM–bioinformatics combination; 4,917 multilingual case cards showed cross-language gaps within three percentage points. They share one critical structural limitation: all are retrospective studies, all test only on cases already solved by humans—the denominator is 'problems eventually resolved by humans'; performance on truly undiagnosed patients remains unknown and cannot be extrapolated. None measured whether integrating LLMs shortens time-to-diagnosis, let alone improves patient outcomes. All evaluated models were fixed-version snapshots—these numbers have an expiration date.
Thus, these figures do not support the claim 'just feed the medical record to ChatGPT and get a diagnosis'—13.3% to 22.4% means most cases will still be missed. Instead, they support a concrete new action: when diagnosis stalls, the primary care team should compile the patient’s clinical features into a structured phenotypic description and submit it to a multimodal combination of LLMs and bioinformatics tools to generate supplementary differential diagnoses for multidisciplinary discussion. The principle 'combination outperforms any single tool' should be explicitly embedded into rare disease center workflows—not treated as an optional add-on. Within this combination, LLMs drive divergent reasoning, specialized tools enforce strict matching, and clinicians make final judgments—none can be omitted.
One final issue worth discussing: when a $0.03 AI inference occasionally uncovers diagnostic clues that expert networks failed to identify over many years, should health insurers reimburse this step—this 'AI second opinion'? If so, who sets the standard—by hit rate, or by years of diagnostic delay avoided?