"Foundation models" have been pathology AI’s loudest selling point over the past two years: first learn a general set of "common sense" from massive volumes of unlabeled tissue images, then apply that knowledge across diverse diagnostic tasks—pretrain once, deploy everywhere. Yet from August to November this year, five studies (three peer-reviewed, two preprints) independently subjected this narrative to a reality check. The conclusion is not that "foundation models don’t work," but something more nuanced—and less exciting: it depends, and your evaluation methods have likely been overestimating them all along.
## Head-to-Head Battle Across 100,000 Biopsies
A study published in *Nature Communications* on October 26 dropped a heavyweight stone into this debate. The research team pitted a task-specific model—fully end-to-end trained for prostate biopsy grading—against two well-known pathology foundation models, using over 100,000 prostate core needle biopsies from 6,268 patients across 15 sites in 11 countries. As the paper itself states, this represents the largest external validation of AI for prostate cancer grading to date.
The results split cleanly in two. **Foundation models do help when data is scarce**—exactly their core strength, enabling rapid deployment with minimal annotation. But **when sufficient labeled data is available, the task-specific model catches up—and even overtakes foundation models in some scenarios**: In agreement with pathologist grading (measured by Cohen’s kappa, where 1 indicates perfect agreement), all three models achieved a high internal kappa of 0.91; on external cohorts, the task-specific model scored 0.87, while the two foundation models scored 0.86 and 0.85, respectively. The differences are small—but the direction is clear: task-specific training also significantly reduced misgrading, misdiagnosis of challenging morphologies, and technical variation—i.e., interference caused by differing slide preparation and scanning conditions across labs.
Two additional accounting items matter: The paper estimates that inference-stage energy consumption for foundation models can be up to 35× higher than for the task-specific model; cross-scanner variability is also greater. It should be noted that some authors hold shares and patents in related commercial companies (disclosed in the paper), and readers should weigh the conclusions accordingly. Still, the paper’s own conclusion is not "abandon foundation models," but rather that both paths deserve serious consideration—robustness, resource efficiency, and clinical deployability are what clinical pathology truly needs.
## Your Leaderboard Cannot Predict Generalization at *Your* Hospital
The second study targets the evaluation methodology itself. A benchmark published on August 13 in the journal *Journal of Imaging* systematically tested 10 mainstream pathology foundation models on two publicly available, multi-center, multi-scanner datasets: colorectal cancer (499 whole-slide images) and breast cancer (583 whole-slide images). The key innovation lies in design: beyond the usual practice of mixing all sources for cross-validation, researchers also performed "leave-one-source-out" testing—excluding all data from one center or one scanner during training, then evaluating on it exclusively.
The two approaches yielded different answers. **Conventional mixed validation may overestimate a model’s ability to generalize across centers and scanners**; only when sources are truly isolated does the generalization gap become visible—and inter-scanner differences prove harder to handle than inter-center differences, with performance loss concentrated on fine-grained tasks like epithelial tissue discrimination. Among the ten models, one emerged as most robust across all protocols—meaning that in real-world, multi-source settings, "which foundation model to choose" is itself a critical decision—not just picking the leaderboard champion.
A preprint posted to arXiv at the end of October dug even deeper into reliability. It evaluated eight foundation models across six datasets and six annotation budgets, conducting 864 probe-training runs (freezing the foundation model and training only lightweight classification heads on its features), plus 20 targeted transfer experiments across five hospitals. Several counterintuitive patterns emerged: When annotation budgets were maximal, the rule "models with stronger discriminative power also calibrate better" (calibration refers to alignment between a model’s self-reported confidence and its actual accuracy) held broadly. But **when annotation was slashed to just 1%, this rule broke down—and became dataset-dependent**. More uncomfortably, when a model’s discriminative power on a new environment fell below ~0.85, adding more source-domain annotations actually worsened calibration on the target environment—in 86%–88% of combinations. Common calibration correction techniques (e.g., temperature scaling) failed in 49 of 288 test cases; and most calibration errors here were under-confidence—not overconfidence.
## Hard Spots Are Concrete; Bias Is Fixable
What do clinical “hard spots” look like? A study published in *npj Digital Medicine* on October 21 offers a concrete profile: On a multi-center Barrett’s esophagus cohort (3,528 slides from 790 patients, spanning the full spectrum from normal tissue to esophageal adenocarcinoma), aggregating predictions from five foundation models most aligned with expert consensus yielded discriminative power of ~0.91 for both morphological subtyping and adenocarcinoma detection, and an adenocarcinoma detection accuracy of 0.959—impressive numbers. But **the benefit for dysplasia grading remained limited**. Dysplasia grading assesses how close a lesion is to malignancy—the very tier clinicians need most, and the one demanding greatest diagnostic skill. The authors’ conclusion is restrained: they affirm foundation models’ capabilities, yet stress that dysplasia assessment remains difficult—and external validation remains non-negotiable.
After the bad news, there’s good news too. A preprint posted to arXiv in early November shows that scanner- and staining-related biases encoded in foundation model representations—the major contributors to those generalization gaps discussed earlier—are **not incurable**: contrastive fine-tuning using paired multi-scanner, multi-stain slides (teaching the model that "images of the same tissue captured on different devices should yield similar representations") improved robustness and downstream task performance across tested models. Code is open-sourced; adapters and partial model weights are publicly released. Of course, this remains preprint-level evidence—segmentation and calibration tasks remain exceptions, and clinical standardization is still far off.
## My Assessment
Laying these five studies side-by-side, my assessment is:
First, "universality" is a matter of degree—not binary existence or absence. Foundation models win at cold starts; task-specific models win in precision, stability, and cost-efficiency once ample labeled data is available. For large hospitals with extensive annotation infrastructure, the real question isn’t "should we adopt foundation models?" but rather "what do I want to save?": annotation labor, energy consumption, or the hassle of adapting to diverse scanners.
Second, leaderboard rankings ≠ generalization at *your* hospital. The finding that mixed validation overestimates generalization is arguably the most actionable insight for procurement teams in this wave of evidence: when validating pathology AI, demand "leave-one-source-out" cross-center, cross-scanner test reports—not just leaderboard rankings.
Third, weaknesses are specific, so evaluation must be specific too. Dysplasia grading, epithelial discrimination, challenging morphologies—these are precisely the hard spots clinicians care about most. A generic "accuracy = 0.9" is meaningless; stratified performance metrics targeting these hard spots carry real value.
Fourth, bias is fixable—but don’t treat preprints as prescriptions. Contrastive fine-tuning points to an engineering-viable path, yet it remains unreviewed; meanwhile, the discovery that "adding annotations can worsen calibration" reminds us: remediation steps themselves require rigorous validation.
A question left for readers: When hospital pathology departments procure AI, should "cross-scanner validation reports" and energy consumption statements be mandatory in tender documents? And when the term "foundation model" itself commands a price premium, who decides whether it’s worth paying up to 35× more in energy cost?
Primary source: https://doi.org/10.1038/s41467-026-78996-7