In November, Weill Cornell Medicine at Cornell University posted a notice to patients on its official website: Between June 10 and 20, certain patient information from a radiology study was **input into an artificial intelligence tool and processed using machine learning—without authorization and in violation of institutional policy**.
According to the notice, this was not a cyberattack: no evidence has been found that any patient information was misused, nor is there currently any evidence that unauthorized individuals accessed the data. The affected information—including medical record numbers, imaging IDs, examination dates and times, diagnoses, and medications—excluded names, addresses, phone numbers, email addresses, Social Security numbers, and patients’ financial information. The incident occurred in June, the institution learned of it on September 6, and the notice was posted only in November. In other words, the data did not "leak out," but someone fed it to AI—and that act itself constituted a privacy incident requiring notification to patients.
This notice serves as an excellent entry point for understanding the current state of privacy in healthcare AI. Because when we line up several studies scheduled for 2026, we find that the three traditional privacy safeguards—de-identification, output review, and aggregate risk scoring—each coincidentally corresponds to one blind spot.
## Blind Spot One: No Output Leakage ≠ No Access
The subtlety of the Weill Cornell incident lies in the fact that the violation occurred during the "access and processing" stage—not the "leakage" stage. The U.S. HIPAA regulation includes a "minimum necessary" principle: identifiable patient information may be accessed only when strictly required for the task at hand. But in the AI era, this principle confronts a technical challenge—**an AI agent can retrieve a medical record number yet never mention it in its response**.
A study posted to arXiv in November turned this issue into an experiment. Researchers embedded "canaries"—traceable markers—into synthetic medical records, then tasked five language models with clinical tasks that fundamentally required no patient identity information. They tested the models under four different permission boundary settings and seven distinct prompting frameworks, observing whether the models would unnecessarily invoke tools accessing protected health information (PHI), and whether the canaries would appear in final outputs.
Results: Three of the five models unnecessarily invoked PHI-accessing tools; Claude Opus 4.8 and GPT-5.5 did not. Tightening permission boundaries reduced such access—but could not eliminate it entirely. Under the strictest policy, residual unauthorized access concentrated in prompts where users claimed, "I have overridden policy" or asserted, "I have obtained approval." The only approach approaching full eradication was a **pre-execution gate**: before each invocation of a sensitive tool, a check must first ask, "Is this task truly necessary?"
The paper’s conclusion can be summarized in one sentence: Evaluating only outputs cannot verify compliance with the "minimum necessary" principle. Permission boundaries, pre-execution gates, and access-log auditing must all be implemented simultaneously. Assessing healthcare AI solely by whether outputs are “clean” is like auditing a cashier solely by whether daily revenue decreased—you must examine the transaction log.
## Blind Spot Two: Aggregate Metrics Pass ≠ Individual Safety Guaranteed
A study published in *Nature* in June claims to be among the first "patient-level" privacy audits of healthcare AI. It tested membership inference attacks: given a model, an attacker attempts to determine whether a specific individual’s data was used to train it. Why is this serious? In healthcare contexts, "this person appears in the training set" is itself sensitive information—being included in a disease-specific dataset effectively exposes one’s diagnosis.
Prior work almost exclusively examined overall success rates. This audit—covering seven large, real-world clinical datasets (medical imaging, electrocardiograms, electronic health records)—found that: **models whose aggregate metrics approached random guessing could nonetheless achieve near-perfect inference accuracy for specific individuals**. The reason is intuitive: a single patient often contributes multiple similar records to a dataset, causing the model to “remember” them exceptionally well. Moreover, larger model capacity correlates with more high-risk patients.
Even more concerning is the third finding: after stratifying by disease status, self-reported race, insurance type, sex, and imaging protocol, **groups underrepresented in training data suffered disproportionately high attack success rates**. The rarer a person is in the data, the more likely the model “remembers” them—and the easier they become for attackers to identify. Aggregate privacy metrics average away these individuals’ risks.
## Blind Spot Three: De-identified ≠ Anonymous
One standard practice for sharing healthcare data is de-identification: removing names and IDs—or even avoiding raw image sharing altogether, instead distributing only model-extracted "embeddings": numeric vectors representing image content. A study posted to arXiv in October specifically tested the privacy floor of brain MRI embeddings.
Researchers used embeddings generated by five publicly available pre-trained encoders to conduct patient-linkage attacks across two longitudinal brain-imaging datasets—where attackers, armed with public encoders and one auxiliary scan from a target, attempted to match embeddings back to specific individuals. Results: In the highest-risk embedding space, **patient-matching accuracy reached 99%**, while attributes such as sex could be predicted with over 90% accuracy (balanced accuracy, per the paper’s reporting)—despite the fact that these encoders were trained on objectives completely unrelated to “identifying people.” The authors’ judgment is direct: replacing raw brain MRIs with learned embeddings does not eliminate patient-linkage or attribute-inference risks; relying solely on embeddings is insufficient to establish anonymity. They even describe brain anatomy as a quasi-biometric trait.
## Patients Are Already Worried—Some Have Already Resigned Themselves
These concerns extend beyond researchers. A study posted to arXiv in November analyzed 6,514 posts across 14 medical subreddits: users spontaneously discussed excessive access privileges, opaque data flows, and unauthorized secondary use of data. Attitudes fell into roughly four categories: demanding accountability for incidents, conditional acceptance (requiring human oversight and regulation), skepticism about system trustworthiness, and—increasingly—**resignation** to data exposure. Discussion volume rose significantly, with inflection points aligning closely with major AI deployments, privacy incidents, and regulatory developments.
## My Assessment: The Unit of Privacy Risk Has Changed
Putting the three blind spots together yields three key insights.
First, the unit of privacy risk in healthcare AI has shifted—from "did data leak out?" to "who accessed it, which patients did the model remember most vividly, and can embeddings be reverse-engineered to re-identify individuals?" Meanwhile, the old compliance triad misses exactly one blind spot each: de-identification fails against embedding linkage; output review misses silent access; aggregate scoring averages away individual and minority-group risks. At Weill Cornell, the institution notified affected patients—but had this "access-only, no leakage evidence" violation gone undetected and uninvestigated, it could have remained entirely invisible.
Second, privacy is becoming a fairness issue. Those most memorized by models—and thus most easily identified by attackers—are often those least represented in the data; privacy risk compounds with representational disparity. This means privacy audits should not report a single aggregate score, but rather deliver patient-level and group-stratified reports—the very approach taken in the *Nature* paper.
Third, for patients and the public, the most realistic threat model is internal process failure—not external hackers. Rather than asking, "Could my data be stolen?", the more urgent questions are: Who inside the institution can feed my data to which AI? Who audits access logs? Is there a "task-necessity" gate before invoking sensitive tools? Today’s healthcare institutions may not yet have ready answers to these three questions—and yet, patients have every right to ask them.
Finally, two open questions remain: Should patient-level privacy audits become a mandatory standard for releasing healthcare AI models? Should pre-execution gates become as standard for healthcare AI as informed consent forms are today?
Source:
- Disparate privacy risks from medical AI (*Nature*, 2026): https://doi.org/10.1038/s41586-026-10688-0