Sepsis is a quintessential time-sensitive condition: each hour of treatment delay increases mortality risk, making algorithms that provide 'hours-ahead warning' inherently attractive to hospitals. In 2018, Epic Systems—the electronic health record (EHR) vendor whose software is used by hundreds of U.S. hospitals—released a proprietary sepsis prediction model within its EHR system: a penalized logistic regression model developed using data from approximately 405,000 encounters across three healthcare systems between 2013 and 2015. Hundreds of hospitals deployed it immediately. There was just one problem: the model was proprietary; its performance metrics existed only in vendor documentation, and no independent validation had ever been published. What unfolded over the next five years became a fully public case study of a black-box model colliding with external validation.
## 2021: Independent Validation Hits the Brakes
In August 2021, JAMA Internal Medicine published an independent external validation by a University of Michigan team: 27,697 adult inpatients and 38,455 hospitalizations at Michigan Medicine (with 2,552 sepsis cases, 7% of admissions), with model scores calculated prospectively every 15 minutes. Results: hospital-level AUC was only 0.63 (95% CI 0.62–0.64), significantly lower than the 0.76–0.83 range reported in vendor documentation.
At the vendor-recommended alert threshold (≥6 points), the model triggered alerts for 18% of hospitalizations; among the 2,552 sepsis cases, it identified only an additional 183 cases (7%)—i.e., cases missed by clinicians but caught by the model—while missing 1,709 cases (67%). Threshold performance: sensitivity 33%, specificity 83%, positive predictive value (PPV) 12%. The authors’ conclusion used unusually strong language: despite poor performance, this model remained widely adopted, raising fundamental concerns about national sepsis management. A contemporaneous editorial proposed a simple, reusable criterion: models with sensitivity + specificity < 1.5 (1 = coin flip; 2 = perfect) must be approached with caution before clinical deployment—the sum for this model was 1.16.
Ripples continued spreading. A December 2024 validation in the emergency departments of two county hospitals, covering 145,885 visits across all of 2023 (JAMIA Open), found: 7,183 alerts triggered (4.9%); 6-hour-window sensitivity was only 14.7%, PPV 7.6%; and alarm timing was even colder—the median lead time was 0 minutes, meaning half the alerts sounded *after* sepsis onset. The authors’ discussion comment was blunt: clinicians independently recognized and treated half the cases without any alerts, and random alerts achieved comparable specificity and negative predictive value.
## Vendor Response: Rebuilding the Model
Criticism did not remain on paper. In 2022, Epic released v2: switching to gradient-boosted trees, redefining the prediction target to align with the Sepsis-3 clinical consensus criteria (v1 had been criticized for using nonclinical labels such as antibiotic orders as endpoints—and even treating antibiotics themselves as predictors), and enabling local fine-tuning using each hospital’s own historical data. The vendor self-reported internal validation AUROC for v2 as 0.83–0.86—but these figures appeared only in conference presentations and were cited in later validation papers as personal communications, with no peer-reviewed documentation. As of August 2025, 95 healthcare organizations (representing 731 hospitals) were using v2 for real-time prediction, while over 100 others still used v1. And for more than three years after v2’s release, no independent multi-center external validation was published.
## 2026: Prospective Multi-Center Results
Finally, in February 2026, JAMA Network Open published this validation: four large academic medical centers collected real-time data during continuous deployment of v2 (from late August 2023 through March 2025), totaling 227,091 adult hospitalizations, including 7,401 sepsis cases (3.3%). The results require reading across three lines.
Discrimination was indeed repaired: encounter-level AUC ranged from 0.82 to 0.92 across the four sites; 12-hour prediction horizon AUC ranged from 0.75 to 0.85, with only slight degradation when benchmarked against clinician identification; median lead time ranged from 1.9 to 10.2 hours—within a clinically meaningful window.
Inter-institutional variation was enormous: the 0.10 AUC gap between 0.82 and 0.92 was not noise—it reflected heterogeneous patient populations and workflows across four real-world systems. The same model performed differently. Moreover, because v2 permits local fine-tuning, these four results effectively represent four distinct "localized models"—and cannot be used to infer expected performance for unadjusted deployments.
Alert burden and PPV remained unaddressed: PPV ranged only from 0.13 to 0.26—roughly one true positive per 4–8 alerts. The authors’ three recommendations to deployers all targeted workflow—not the model itself: conduct local validation prior to go-live; design false-positive management processes; implement alert-silencing strategies.
## Five-Year Arc Closes—Two Questions Remain Unanswered
Internal validation: 0.76–0.83; independent external validation: 0.63; county ED sensitivity: 14.7%; vendor rebuild; prospective multi-center: 0.82–0.92. This arc proves one thing: external validation is not hostility toward innovation—it is the only publicly visible mechanism that forces black-box models to evolve.
But two questions remain open. First, v1’s 0.63 and v2’s 0.82–0.92 are not directly comparable on the same endpoint—v1 used a composite definition combining CDC surveillance criteria and ICD coding, whereas v2 used Sepsis-3; label definitions themselves influence AUC, so cross-version comparisons must be made cautiously. Second, none of the studies over five years measured patient outcomes: mortality or length of stay remain unanswered. The final link between "earlier warning" and "fewer deaths"—a randomized trial—remains missing.
## Three Takeaways
First, for hospitals: before deploying any vendor-provided prediction model, conduct local retrospective validation; after deployment, perform continuous prospective monitoring; establish a permanent dashboard tracking AUC, sensitivity and PPV at operational thresholds, alert volume, lead time, and incremental detection beyond clinician identification; alert governance (silencing rules, tiered response protocols) is as critical as the model itself.
Second, for vendors: the trajectory from v1 to v2 demonstrates that external criticism can drive product iteration—but the industry waited three years for proactive, multi-center validation post-v2. Making "validate upon release" the default action is the shortest path to rebuilding trust.
Third, for everyone interpreting performance numbers: the statement "this model has AUC 0.83" carries different truth values in the 2018 vendor documentation, the 2021 independent validation, and the 2026 prospective validation—each bound to specific model version, validation design, endpoint definition, and population. Performance metrics stripped of these binding conditions are not knowledge—they are advertising.
One final question worth discussing: When a hospital’s EHR contains a vendor’s black-box prediction model—one never independently validated—should patients have the right to know of its existence and its true performance?
Source: External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Intern Med 2021. https://doi.org/10.1001/jamainternmed.2021.2626