Using ChatGPT to study medicine has, until recently, relied solely on surveys and anecdotes. Between 2025 and 2026, the first randomized controlled trials (RCTs) and a meta-analysis synthesizing 20 RCTs were published—marking the first time this field could speak with evidence. And that evidence paints a far more nuanced picture than either the hype (“AI tutors crush traditional teaching”) or the fear (“AI destroys deep learning”): coaching works—but feedback from AI remains questionable; short-term scores rise—but retention lacks supporting evidence.
## +0.7 Points, and a More Valuable Absenteeism Rate
The anchor is the ASCE trial published in NEJM AI, which its authors claim is the field’s first single-blind RCT: 247 fourth-year medical students in early clinical clerkships at two campuses of Montpellier-Nîmes Medical School (recruited November 2023–January 2024) were randomly assigned to either an AI training group (n=125) or a standard training group (n=122), with a 2.5-month intervention. The training platform used GPT-4 Turbo (version dated 2023-11-06) to simulate text-based patient interviews and deliver automated feedback. The primary endpoint was OSCE (Objective Structured Clinical Examination) scores administered by faculty: median score 11.4 in the intervention group versus 10.7 in the control group (out of 20; P=0.02; 95% CI 0–1.2)—a statistically significant but modest effect, with the lower bound of the confidence interval abutting zero.
More intriguing than the score gain are other figures: absenteeism was 0.8% in the intervention group versus 4.9% in the control group; pre-exam emotional preparedness was higher in the intervention group (P<0.001; CI 1–2), and stress levels lower (P=0.02; CI −1–0); among 459 process feedback responses, 74% of students who used the AI coach reported “feeling like they were interacting with a real patient.” This points to an easily overlooked judgment: the AI coach’s first-order effect may not be “teaching better,” but rather “enabling practice”—an infinitely patient, always-available, nonjudgmental simulated patient removes two major barriers: “can’t find a practice partner” and “too embarrassed to practice.”
Limitations must also be stated clearly: blinding covered only parts of the assessment phase; students knew they were using AI, so subjective endpoints such as emotional preparedness and perceived realism could not be blinded; the intervention lasted only 2.5 months, with no long-term follow-up; and the training platform was a commercial product—the researchers’ relationships with the platform are disclosed in the original publication.
## Meta-Analysis of 20 Trials: Effect Size Surprisingly Large, Certainty Only Moderate-to-Low
A systematic review published in BMC Medical Education in April 2026 pooled data from 20 RCTs involving 1,413 participants: after integrating generative AI into instruction, the pooled standardized mean difference (SMD)—a metric that places scores from different exams onto a common scale (where 1 represents one standard deviation)—for clinical skills reached 1.09 (95% CI 0.81–1.38; GRADE certainty moderate), and for knowledge acquisition it was 0.99 (0.49–1.49; GRADE certainty low).
An SMD near 1 is exceptionally large for an educational intervention. Yet GRADE assigns only moderate-to-low certainty, precisely because of how this number should be read: the majority of included trials had small sample sizes; endpoints were mostly short-term in-course assessments; and heterogeneity across studies was high. Large effect sizes paired with small trials constitute a classic combination carrying high risk of effect inflation. One commendable feature of this meta-analysis: it included negative results—not just selectively highlighting positive findings.
## Three Buckets of Cold Water
First bucket: When AI serves as a feedback provider, it did not outperform expert-crafted templates. A 2025 randomized crossover trial in Medical Teacher assigned German medical students to answer key-feature questions daily via an app; for half the questions they received generic feedback written by experts, and for the other half, personalized feedback generated instantly by ChatGPT. The endpoint was performance on a required final exam. Result: personalized AI feedback conferred no advantage over expert generic feedback (51.8±22.0% vs. 55.8±22.8%, p=0.06; the direction even slightly favored the expert group, though not significantly). Students who used the app frequently scored better—but this was merely an observational correlation. Inserting AI into the feedback loop does not automatically add value: expert-written generic feedback is pedagogically designed, whereas AI-generated personalized feedback varies in quality. Indeed, the aforementioned meta-analysis of 20 trials noted that AI feedback sometimes oversimplifies or deducts points for minor spelling errors—eroding its personalization benefit. One caveat: p=0.06 does not prove equivalence (the study did not conduct an equivalence test); it only means “no advantage was demonstrated.”
Second bucket: Short-term scores rose—but one week later, nothing remained. A small trial at Georgetown University School of Medicine (Cureus 2025) invited 198 first-year medical students; 33 enrolled and were randomly assigned to prepare for the same exam using ChatGPT-4.0 (n=10), external public resources such as Google or PubMed (n=12), or institutional lecture notes and textbooks (n=11). Immediate test scores: ChatGPT group 9.60±0.52, external-resources group 9.08±0.79—both significantly higher than the institutional-resources group’s 6.64±1.57 (p<0.001). But one week later, retested without any resources: scores were 6.20, 5.58, and 4.36 respectively—no significant between-group differences (p=0.118). Two striking details: ChatGPT conferred no advantage over “knowing how to search”; and “scoring well when resources are available” is fundamentally distinct from “remembering content when resources are absent.” This was a single-center, n=33 pilot with a very short one-week retention window—suggestive but insufficient for definitive conclusions. Still, it aligns with longstanding learning science: retrieval convenience ≠ encoding depth.
Third bucket: AI coaching versus peer role-play—each wins half. A 2025 pilot RCT in BMC Medical Education randomized 19 senior Korean medicine students to either an AI chatbot group (n=9; dual-layer architecture using GPT-4o and Claude 3.5, with scenario-specific responses and automated scoring feedback) or a peer role-play group (n=10; pairs practicing under faculty supervision). Participants then completed two OSCE stations (dizziness, shoulder pain). Total scores showed no significant difference—but subscores were complementary: the peer group scored higher on history-taking (74.4 vs. 66.2 in the dizziness station; Hedges’ g = −0.68), and rated higher on interaction authenticity and exam realism; the AI group showed promise in domains requiring clinical reasoning (e.g., patient education), and students valued its autonomy, repeatability, and structured feedback. With n=19, this pilot can only generate hypotheses; the population consisted of Korean medicine students—not Western clinical medicine trainees—so generalizability requires extreme caution. Yet the direction is clear: AI supplements practice volume and immediate structure; humans supply authenticity and communicative nuance.
## What’s Consistent—and What’s Missing
Laying all evidence side-by-side, consistent findings are clear: short-term exam scores improve almost uniformly; among various AI uses, “AI as practice partner / simulated patient” has the strongest evidence—not only the positive ASCE trial and the meta-analysis’s skill SMD of 1.09, but also another RCT on interview training showing that the GPT-simulated-patient group significantly outperformed the instructor-led role-play group in interviewing skills.
What’s conspicuously missing is equally clear: no support exists for retention or transfer—Georgetown’s one-week retention result was negative, and long-term data are entirely absent; evidence for “AI as feedback generator” is weakest; every study measured only exam performance—none assessed post-graduation clinical competence, let alone patient outcomes. Effect sizes must be interpreted against full-scale metrics: +0.7 points out of 20 carries far less meaning for individual students than it does for “accessibility in resource-constrained institutions”—Georgetown’s introduction posits AI might enhance accessibility for learners in under-resourced settings, but this remains a qualitative argument, not an empirically validated conclusion. Moreover, these trials are locked to specific models and platforms—GPT-4 Turbo, GPT-4o, Claude 3.5—versions current between 2023 and 2025; their conclusions have expiration dates.
## Three Takeaways
First, for medical students: Using AI as an infinitely available standardized patient and question generator is evidence-supported; do not rely on it as your sole source of feedback; and beware the illusion that “scoring well with AI” equals mastery—close the tool during self-testing, and remember: one week later, in a resource-free retest, the three groups showed no significant differences.
Second, for medical schools: The absenteeism gap—0.8% vs. 4.9%—suggests AI coaching’s greatest value may lie in reducing the marginal cost of practice opportunities to nearly zero—a structural advantage for institutions strained by low faculty–student ratios and limited budgets for standardized patients. Procurement should require platforms to provide granular process data on practice volume and performance, making “did they practice?” visible before “did they learn?” can even be asked.
Third, for those following this field: An SMD of 1.09 cannot be generalized as “AI teaching universally boosts outcomes by one standard deviation”; the true void lies beyond exams—retention, transfer, post-graduation competence, and patient outcomes remain unmeasured.
One final question worth debating: If AI coaching’s biggest contribution is “making practice happen”—not “teaching better”—should medical schools formally mandate “AI practice duration” in curricula? Or would that simply replace one form of coercion with another—undermining students’ intrinsic motivation to “practice just a little longer”?
Source: AI-Standardized Clinical Examination Training on OSCE Performance. NEJM AI. https://doi.org/10.1056/AIoa2500066