On November 16, 2026, Langenbecks Archives of Surgery published a study from a German tertiary referral center: researchers compiled 30 postoperative gastrointestinal tumor cases (10 pancreatic, 10 gastric, and 10 colorectal) into structured scripts and ran each case three times independently using ChatGPT within a controlled guideline environment; treatment recommendations were generated via majority voting, then compared against both German clinical guidelines and actual multidisciplinary tumor board recommendations—guideline concordance was 70%, tumor board concordance was also 70%, and concordance with both simultaneously was only 56.7%. Even more striking numbers lie in reproducibility: among the 90 total runs, 25.6% yielded outputs discordant with guideline recommendations (of which 15.6% represented major deviations), and only 67.8% of case–guideline combinations produced identical outputs across all three runs.
This is the latest data point on "LLM-era oncology decision support." Before it lies a famous corpse: IBM Watson for Oncology. Only by placing these two generations of evidence side-by-side can we discern what this field has truly learned over the past decade.
## Watson’s story can now be told in full
Trained with participation from Memorial Sloan Kettering Cancer Center (MSKCC), Watson was deployed in 14 countries; in a 2018 media statement, IBM claimed it covered 13 cancer types and served over 84,000 patients across 230 hospitals. The turning point also came in 2018: on July 25, STAT News reported—based on a set of internal presentation slides—that IBM’s internal medical experts and clients had documented "multiple instances of unsafe and incorrect treatment recommendations." These slides originated from an internal presentation delivered in summer 2017 by Andrew Norden, then Deputy Chief Health Officer of IBM Watson Health. A more fundamental flaw lay in its training methodology: Watson was trained on a small number of "synthetic" (i.e., fictional) cancer cases—not real patient data—and its recommendations for each cancer type were derived from opinions of only a few experts—not from "guidelines or evidence." A physician at Jupiter Medical Center in Florida spoke even more bluntly to IBM executives (original English quote, translated here): "This product is garbage. We bought it only for publicity; most cases simply cannot be used with it." The report also confirmed: no patients died as a result of these erroneous recommendations.
On January 21, 2022, IBM announced the sale of Watson Health’s healthcare data and analytics assets to Francisco Partners (reported by Bloomberg to exceed $1 billion; neither party disclosed the transaction amount). Industry analysts commented in Healthcare IT News that this sale "essentially marks the end of the Watson Health experiment."
## Concordance is not the gold standard: Two stable patterns across data from four countries
Attributing Watson’s failure solely to "technological immaturity" is a misreading—the concordance studies conducted across countries reveal a far more complex pattern. A retrospective study at a university hospital in Busan, South Korea, involving 183 breast cancer patients (2016–2017), found Watson’s concordance with clinician recommendations was only 40.4%; the authors attributed the primary cause to differences between Korean and U.S. health insurance systems and clinical environments. A study in Northwest China (Ningxia/Qinghai) involving 463 breast cancer patients found concordance for postoperative adjuvant therapy was 80.2%, but plummeted to 50.5% for first-line treatment of advanced disease. A study of 322 gastric cancer patients (2015–2018) in Daegu, South Korea, showed overall concordance with tumor boards was 86.96%, but when stratified by stage: 96.9% for Stage I, 88.9% for Stage II, 90.9% for Stage III, and only 45.8% for Stage IV; multivariate analysis identified age, performance status, and Stage IV as significant predictors of discordance. A retrospective analysis of 55 consecutive urologic oncology cases (2017–2020) in Incheon, South Korea, reported an overall concordance rate of 92.7%, yet Watson failed to generate any appropriate options for three cases, and some recommendations were non-reimbursable under Korea’s national health insurance review system—researchers also observed that Watson’s introduction actually promoted multidisciplinary collaboration itself.
A systematic review published in Frontiers in Artificial Intelligence in July 2026 assembled these fragments into a panoramic view: 59 studies involving 23,158 patients showed AI decision-support concordance with multidisciplinary teams or clinical guidelines ranging from 23.2% to 99%—a span so wide that "concordance" itself carries almost no meaningful information; what truly determines performance is regional context, treatment accessibility, and case complexity. The review’s three major implementation barriers—technical limitations, socioeconomic and healthcare system constraints, and patient- and tumor-specific factors—map nearly one-to-one onto Watson’s original causes of failure.
From these four-country datasets, two stable patterns emerge. First, high concordance for simple cases, collapse for complex ones: AI aligns closely with human experts for early-stage, standard-pathway cases—but for Stage IV, advanced, or multi-morbid cases—the very scenarios most in need of decision support—concordance drops by half. Second, tumor boards themselves are not a gold standard: their recommendations inherently vary by region and institution, so "discordance" measured against local tumor boards does not necessarily indicate AI error. Watson’s era did not resolve these two issues—it merely exposed them.
## The LLM era: Someone has read this cautionary textbook
Emerging evidence from the LLM era shows the field has at least absorbed part of the lesson. Comparing the recent German ChatGPT study with Watson-era research reveals four notable differences. First, positioning has shifted: the authors conclude LLMs may serve in "guideline screening and pre-tumor-board preparation," and must operate under expert physician supervision—no one now claims replacement of tumor boards. Second, failure modes are transparent: Watson was a black box; errors could only be inferred retroactively through concordance studies; the ChatGPT study directly quantifies output instability (25.6% run-level discordance, 15.6% major deviations) and combats it via "three independent runs per case + majority voting"—instability has thus evolved from a hidden risk into a measured, engineered object. Third, evaluation design is more rigorous: prospective registration, distinction between major and minor deviations, and reporting of reproducibility. Fourth, benefit claims are more pragmatic: the systematic review confirms improvements in multidisciplinary team adherence to guidelines, identification of eligible clinical trial candidates, support for junior physicians, and triage of routine cases—none of which claim to "replace physicians."
## The final exam: Neither generation has submitted answers yet
What remains unlearned is equally clear. First, there is still no evidence linking AI use to patient-centered outcomes—the systematic review authors explicitly call for "larger prospective studies evaluating real-world integration, clinical impact, and patient outcomes"; however refined concordance studies become, they measure only "how much AI resembles experts," not "whether patients benefit." Second, regionalization problems remain unchanged: Watson failed in Korea due to reimbursement formularies and achieved only 40.4% concordance in Busan; today’s LLM studies remain exclusively single-center, single-country efforts—the German study’s 30 cases are smaller than any Watson-era study. Third, while majority voting suppresses some instability, the 67.8% full-concordance rate across three runs means: for the same patient and identical prompt, roughly one-third of the time, three queries yield non-identical answers—a still-unresolved engineering challenge in treatment decisions requiring signed accountability.
## How lessons translate into action: Three reader groups, three actions
For hospitals and oncology departments: the correct way to introduce LLMs into tumor boards begins with low-risk functions—pre-meeting literature and guideline verification, clinical trial screening, triage of routine cases; before deployment, conduct local guideline and local reimbursement formulary alignment validation; treat concordance rates as reference metrics—not acceptance criteria. For oncologists: your tumor board is not a gold standard; AI concordance figures must be interpreted stratified by disease stage and complexity; when using LLMs, require multiple runs and source verification; major deviations (e.g., the 15.6% run-level major discordances in the German study) must undergo manual adjudication. For the AI-in-healthcare industry: Watson’s full story—training on synthetic cases, substituting expert opinion for evidence, marketing outpacing validation, continuing rollout after internal detection of flaws—is the industry’s best cautionary textbook; ten years later, the resurgence of ChatGPT-style studies embedding "expert oversight," "prospective registration," and "quantification of failure modes" into their design signals that someone has indeed read the textbook. Yet the final exam question—patient outcomes—remains blank to this day.
Uncertainties are listed as usual: All Watson concordance studies were retrospective and small-sample (183, 463, 322, and 55 cases), and the "concordance" reference standard itself varied; the STAT report was based on internal slides, and both IBM and MSKCC issued rebuttals at the time—we present the report’s original wording with explicit attribution; the German ChatGPT study is a single-center, exploratory cross-sectional design (30 cases), with authors self-limiting conclusions to "potential roles in guideline screening and pre-meeting preparation"; LLM evidence relies on GPT-4–generation models, which have limited shelf life; no patient outcome data exist throughout.