After a medical AI model is developed, it is typically validated on publicly released datasets published by others to demonstrate generalizability—a step termed 'independent external validation' in the literature. On November 12, 2026, JAMA Ophthalmology published a dataset audit revealing a problem with this step in glaucoma research: three public datasets all originated from the same laboratory yet had long been used across the literature as mutually 'independent' validation and fairness evaluation cohorts. Among them, 1,031 identical OCT three-dimensional scans appear simultaneously—under different identifiers—in two of the datasets. Training on one dataset and testing on 'the other' inflates AUC by 0.12. Between August and November 2026, six additional studies were published sequentially: external validation of the AIROGS Challenge-winning algorithm on the Rotterdam population cohort; a ten-year systematic review enumerating 63 studies; a fairness study across 12 centers; baseline description of the ophthalmology module in Germany’s national cohort; a structure-function systematic review covering 36 studies; and a preprocessing reproducibility assessment using three random seeds. Collectively, these seven studies address the same fundamental question: How solid is the data foundation for glaucoma AI—and how should new foundations be built?
## Audit: Identity Information Is Unnecessary—Fingerprints Suffice
The audit targeted Harvard-GDP (1,000 eyes), Harvard-GF (3,300 eyes), and Harvard-FairVision (10,000 eyes). The paper explicitly states that all three datasets originate from the same laboratory, yet they have long been used interchangeably in the literature as 'independent external validation' and fairness evaluation cohorts. The method comprised a cross-sectional dataset audit plus record linkage analysis, conducted in July 2026: record linkage was performed using 'fingerprints' derived from visual field values and demographic features; chance match rates were calibrated using within-group collision rates and permutation models; 1,302 candidate pairs underwent direct image-by-image comparison; no patient identity information was used at any stage. Results: Harvard-GF and Harvard-FairVision yielded 1,302 matched eyes, whereas the chance expectation was only 1.9 eyes; direct image comparison confirmed that 1,031 scans (79.2%, 95% CI 76.9%–81.4%) were identical OCT three-dimensional scans—as defined by the authors, i.e., cosine similarity ≥ 0.9999 and 88% of voxels identical. This implies that at least 31% of Harvard-GF’s 3,300 eyes also exist in FairVision (the authors note this is a lower bound); the repetition rate is nearly identical across FairVision’s training, validation, and test splits (78.5%–80.8%), meaning the splits themselves fail to isolate duplicates. Harvard-GDP and Harvard-GF share another 22 eyes that match perfectly across all 52 total deviation points and mean deviation values—and all 22 fall within GF’s validation split—but because GDP does not release OCT volumetric data, the authors label this evidence weaker. Three further facts: no documentation contains patient identifiers; no data release has ever disclosed overlap; among 105 papers citing these datasets, the authors obtained full texts for 64, and none reported overlap.
## Cost: AUC Inflated by 0.12; Fairness Assessment Distorted Accordingly
The final step of the audit was a controlled experiment: across five training runs, classifiers trained on one dataset and tested on 'the other' achieved AUC = 0.90 (95% CI 0.83–0.94), while 78.5% (74.0%–82.5%) of eyes in the test set were actually part of the training data; the true out-of-sample AUC was only 0.78 (0.72–0.83), an inflation of 0.12 (0.06–0.18). Fairness assessment is even more problematic: inflation occurs across every racial subgroup, and is larger in subgroups exhibiting lower out-of-sample performance (Spearman ρ = −0.57, permutation test P = .02). Thus, data leakage not only inflates overall performance but also flattens real inter-subgroup disparities—rendering fairness assessments built upon such benchmarks fundamentally distorted. The authors conclude: these datasets are not independent cohorts; treating them as independent benchmarks exaggerates reported performance via data leakage and non-independent validation, and may mask fairness gaps.
## When Sources Are Clean, What Do Honest Numbers Look Like?
On November 2, Ophthalmology published external validation of the AIROGS Challenge-winning algorithm on the Rotterdam Study: the algorithm was trained on 101,442 macular-centered color fundus photographs to identify referable suspected visual-field-defined glaucoma; the validation cohort was a prospective population cohort (three subcohorts, ages 50–80), comprising 19,088 eyes (9,837 individuals), of which 201 eyes (1.1%) were diagnosed with visual-field-defined glaucoma. Results included a full suite of metrics: sensitivity 70.6% (95% CI 63.8–76.8) at 95% specificity; ROC-AUC 0.911 (0.887–0.934); PR-AUC only 0.343 (0.274–0.427); participant-level sensitivity 68.8% and ROC-AUC 0.909; translated into screening operations, approximately 5.1 referrals per glaucoma case identified. Three details merit attention. First, missed diagnoses were not random: true positives exhibited significantly worse visual field damage than false negatives (median mean deviation −10.8 dB vs. −5.8 dB, P < 0.001); the model predominantly missed early-stage cases—the very group screening aims to capture. Second, among true positives with prior imaging, 33 of 61 eyes (54%) had already been flagged by the algorithm in earlier examinations—typically 4–7 years before visual field diagnosis. Third, imaging acquisition standards have real-world consequences: although the algorithm was trained solely on macular-centered photos, disc-centered photos demonstrated superior discriminative power in the 90%–100% high-specificity range (pAUC 0.860 vs. 0.802, P = 0.003); incomplete optic disc visualization was a major source of false positives, and excluding such images markedly improved positive predictive value. The authors’ conclusion is restrained: the algorithm achieved meaningful discriminative ability on an external population dataset, supporting its positioning as a scalable screening triage tool—deployment should be accompanied by explicit specification of imaging acquisition standards. This validation and the aforementioned audit constitute a methodological contrast (tasks and data differ, numbers are not directly comparable; the contrast lies in evaluation practice): the algorithm had never seen the validation cohort’s data, and metrics were reported comprehensively. Whether numbers look 'good' is secondary; what determines usability is whether sources are clean and metrics complete (this is the article’s summary).
## Ten-Year Inventory: The '≈0.95' World
An online Expert Review of Medical Devices systematic review (published November 9), following PRISMA guidelines, searched glaucoma AI literature from 2015 to 2026 (search protocol pre-registered), initially screening 3,758 records across PubMed, IEEE Xplore, ScienceDirect, and SpringerLink, and ultimately including 63 studies: dominant approaches apply deep learning to fundus photography, OCT, OCT angiography, visual fields, or multimodal data; in the reviewers’ words, 'many models achieved ROC AUC ≈ 0.95'. The expert opinion section is blunt: high reported accuracy alone cannot establish clinical utility; before scaling up, prospective multicenter validation, diverse populations, interpretable and reproducible models, standardized evaluation, and workflow integration are all indispensable. Reading this alongside the audit adds another layer: in the '≈0.95' world, no one has systematically quantified how many models rest on mutually non-independent benchmarks (this is the article’s judgment; the review did not analyze dataset overlap). A BMC Ophthalmology structure-function systematic review (including 36 studies) supplements technical limitations: models predicting visual field loss from OCT structural changes show large performance variation depending on task, dataset, OCT device, visual field modality, validation design, and metric definitions; performance consistently declines in advanced glaucoma; narrow scan coverage is a hard constraint—the authors argue that clinical deployment requires next-generation models capable of predicting pattern deviation probability maps from wide-angle volumetric OCT.
## New Data, Old Problems: Three Empirical Tests
Ophthalmology Science’s November issue describes Germany’s national cohort (NAKO) ophthalmology module: among 205,053 enrolled participants, 48,460 completed secondary ophthalmic examination (ages 19–74, mean 48.9 ± 12.5 years, 52.7% male), with standardized acquisition of nonmydriatic color fundus images, visual acuity, and ocular history. As a demonstration of data usability, standard deep learning architectures predicted age (MAE 2.96 years), sex (accuracy 0.84), and systolic/diastolic blood pressure (MAE 10.78/6.97 mmHg) from these fundus images—described by the authors as 'comparative performance to the state-of-the-art'. But equally important is the number 68.2%: the proportion of fundus images rated 'gradable' by four deep learning quality grading models; self-reported glaucoma prevalence was 2.0%. In other words, even in a brand-new, uniformly protocol-driven, population-based cohort, nearly one-third of images fail to meet analytical standards (this arithmetic follows from 68.2%; this is the article’s summary); data problems extend beyond 'Are these the same people?' to 'Can the images themselves be used?' (this is the article’s summary).
An online JAMIA fairness study (September 30) used electronic health records from 63,268 glaucoma patients across 12 centers, augmented with community-level social determinants of health (SDoH) features—namely, the Distressed Communities Index (DCI) and Rural–Urban Commuting Area (RUCA) codes—to train an XGBoost model predicting progression to incisional glaucoma surgery. Performance (AUROC) and fairness (generalized equalized odds, GEO—lower is fairer) were compared between two models—with and without SDoH—on an internal test set and three externally held test sets differing in population composition. Results: incorporating SDoH had negligible impact on prediction performance; fairness improved markedly on the internal set (GEO 0.098 vs. 0.216) and one mixed-population external site (0.172 vs. 0.378), but remained nearly unchanged at a highly urban, low-distress external site (0.491 vs. 0.504), and showed no improvement at a rural, high-distress external site (0.241 vs. 0.228). Authors conclude: community-level SDoH features can improve socioeconomic fairness without sacrificing accuracy—but gains do not automatically transfer to externally held cohorts with dissimilar population structures. Read alongside the audit: leaky benchmarks flatten subgroup disparities and distort fairness assessments; fairness improvements measured at a single site may vanish elsewhere. Fairness is not a property of the model alone, but of the entire 'model × data × population' pipeline (this is the article’s summary).
A Diagnostics preprocessing reproducibility assessment (September 7) shifts focus upstream: for optic disc/cup segmentation, five preprocessing pipelines (baseline, CLAHE, ROI cropping, ROI + CLAHE, enhanced CLAHE augmentation) were systematically re-evaluated across three public datasets (REFUGE, ORIGA, Drishti-GS), using a fixed segmentation model and three random seeds (42, 15, 89) per configuration. Results: CLAHE-enhanced augmentation performed best on REFUGE (disc Dice 0.9523 ± 0.0017, cup 0.8348 ± 0.0018); on ORIGA, disc Dice differences across all five configurations fell within 0.0067, with rankings reversing across metrics; on Drishti-GS, the earlier single-run experiment’s 'near-total failure' was traced to optimizer and loss function configuration—not the small 81-image training set. Two sobering notes: full-resolution CLAHE was the only choice consistently beneficial across all three datasets; ROI + CLAHE cropping centered on the centroid of gold-standard annotations—a step the authors themselves label 'oracle localization', which fully automated pipelines cannot rely upon. Even this seemingly trivial preprocessing step yields conclusions dependent on dataset and random seed; single-run rankings cannot sustain even segmentation tasks (this is the article’s summary).
## Three Judgments
The following are integrated judgments drawn from the seven studies—not conclusions of any single study. First, 'independent external validation' is a property of the data, not the paper: without auditing data provenance, collection timing, and mutual overlap, a validation set may simply be a renamed training set. The 1,302 matched eyes versus a chance expectation of 1.9, the 1,031 confirmed identical volumetric scans, and zero reports of overlap among 64 full-text papers citing these datasets constitute a complete negative case study; moreover, the audit requires no identity information—only modality-specific numerical fingerprints plus direct image comparison—a methodology replicable in any field with public benchmarks (this is the article’s judgment). Second, metrics must be read as a suite: within the same clean external validation, the 70.6% sensitivity at 95% specificity, 0.911 ROC-AUC, 0.343 PR-AUC, 5.1 referrals per case, and severity distribution of missed cases ('concentrated in early cases') collectively define the tool—any single metric over- or underestimates it; likewise, encountering a single-point AUC ≈ 0.95 warrants immediate questions about PR-AUC, calibration, disease prevalence, and dataset overlap. Third, data quality is continuous engineering—not a one-time investment: even in a new cohort, only 68.2% of images were gradable; preprocessing rankings drifted across datasets and seeds; fairness gains failed to transfer across sites; none of the three 'independent' datasets ever disclosed overlap in any release. Image quality, evaluation rigor, population representativeness, and source transparency—none can be achieved once and for all (this is the article’s judgment).
## What Different Readers Can Take Away
All recommendations below are strictly limited to the scope of this article. Model developers: make patient-level deduplication and data-provenance auditing standard steps before dataset splitting; run multiple random seeds and report metrics comprehensively; avoid ranking model capabilities using configurations involving 'oracle localization'. Dataset publishers: disclose provenance, collection periods, and known overlaps; the three audited datasets never disclosed overlap in any release—an institutional gap that can be systemically closed. Evaluators and journal readers: when encountering 'independent external validation', first ask about the source relationship between the two datasets; when seeing a single high AUC, first ask about metric definitions and population disease prevalence. Health systems and procurement agencies: fundus-based glaucoma screening AI requires full operational metrics—sensitivity, referral volume, gradable-rate thresholds, and acquisition standards—under local disease prevalence and image quality—not challenge-rankings. Patients and the public: screening tool metrics derive from specific populations, disease prevalences, and image qualities—your eyes do not automatically conform; arrangements for glaucoma screening and follow-up should be discussed with an ophthalmologist.
Final boundary statement: This article reads only summary-level material from the seven studies; the audit conclusions apply exclusively to the three datasets examined and cannot be extrapolated to 'public datasets generally'; 'identical OCT volumetric data' is defined per the authors’ criteria (cosine similarity ≥ 0.9999 and 88% identical voxels); 'flagged 4–7 years earlier' refers specifically to the subset of 61 true-positive eyes with prior imaging—not outcomes from a prospective screening trial; the decade-long review’s 'many models ≈ 0.95' is the authors’ qualitative summary, not a meta-analytic result; the NAKO paper’s age, sex, and blood pressure predictions serve only as data-usability demonstrations—not validation of a glaucoma screening tool; the JAMIA study’s outcome is 'progression to surgery', not diagnosis itself, and GEO is one specific fairness definition; the preprocessing reproducibility assessment covers only segmentation tasks and three datasets; conflicts of interest and funding information are taken solely from the abstracts cited—this article makes no claim of 'no conflict of interest'; all numbers are transcribed verbatim from the abstracts.
Source: JAMA Ophthalmology, published online November 12, 2026 (PMID 43018943, https://doi.org/10.1001/jamaophthalmol.2026.5155); Ophthalmology, published online November 2, 2026 (PMID 42971439); Expert Review of Medical Devices, published online November 9, 2026 (PMID 43000110); JAMIA, published online September 30, 2026 (PMID 42813648); Ophthalmology Science, November 2026 issue (PMID 42894708); BMC Ophthalmology, published online August 3, 2026 (PMID 42958112); Diagnostics, published online September 7, 2026 (PMID 42739310).