You request a platform to delete your data, and the model replies, 'Deleted.' But in a neural network, deletion is not a switch—it’s more like crossing a book off a library catalog while the book remains on the shelf. In November 2026, five papers almost simultaneously brought this issue into sharp focus: each proposes a distinct 'unlearning' method, yet none passes all four evaluation criteria; even more troubling, even if a method passes them, the supposedly deleted information may still be 'called back.'
Tiwary, Srivastava, and Prathosh, in 'Auditable Unlearning for LLMs' (https://arxiv.org/abs/2611.05866), point out that existing unlearning methods are evaluated along only three dimensions: how thoroughly the target data is forgotten, how much of the model’s original capability is retained, and resistance to membership inference attacks (i.e., inferring whether a specific data point was included in training from the model’s outputs). They introduce a fourth dimension—auditability: can the service provider produce a self-computed statistic proving that the 'requested-to-be-forgotten data' has been statistically separated from the 'retained data'? Testing 11 methods across three benchmarks—TOFU (fictional character Q&A), MUSE (news and book copyright), and RWKU (real-world knowledge)—none satisfies all four criteria simultaneously. Their two-step patch, GapWiden (using hinge loss—'penalize insufficient gap'—to widen the audit-statistic gap between the forget set and retain set) plus GapVeil (during inference, draft responses using the base model, then resample answers conditioned on the audit statistic to block noncompliant outputs), achieves oracle-level parity in both auditability and membership inference exposure (where 'oracle' denotes an ideal reference model fully excluding the target data), and unlearning performance also converges to oracle-level; however, among the 11 methods, only 9 show improved retention performance—and even those still fall short of oracle parity.
Shkabrii, Plant, and Pawelczyk, in 'Forget What Comes Next' (https://arxiv.org/abs/2611.03955), take another route: they use the Hilbert–Schmidt Independence Criterion (HSIC)—a measure of statistical independence between two variable sets—to reduce dependence between 'tokens to be forgotten' and the internal representations predicting them, and derive a computable upper bound directly linking the training objective to 'exact memorization.' On models up to 7B parameters (scalable to 70B), theirs is the only method that renders Gaussian statistical marker samples (a class of specially designed probe samples for detecting memorization) undetectable while preserving near-original capabilities.
Jiayu Zhang et al.’s 'Hallucination-Guided Unlearning' (https://arxiv.org/abs/2611.10697, NeurIPS 2026) treats the model’s own failures as diagnostic signals: it localizes 'unsupported fragments' via hallucinations, traces internal consensus to identify a small set of high-responsibility MLP neurons (feedforward units responsible for specific memories within the model), and edits them under a compact gating mechanism constrained by KL divergence (limiting the magnitude of change) to preserve capability. On the WMDP (dangerous knowledge benchmark) economics subtask, it reduces scores to 14.2, while retaining 52.7 on retention and 57.2 on MMLU (general capability).
Zirui Pang et al.’s LEAF-OPD (https://arxiv.org/abs/2611.20674) does not directly suppress knowledge but instead teaches the model 'how to respond when asked about something to be forgotten': a frozen teacher model provides token-by-token supervision on student-generated responses—forget-data queries receive 'privileged context' (extra instructions not to disclose), while retain-data queries do not; the student receives no privileged context in either case. They quantify corrective supervision strength using supervision signal SS, measuring it over the full trajectory (SS1) and the second half (SS0.5)—the latter being especially critical because teachers often exert control only over the first few tokens. On RWKU, its unlearning quality exceeds the runner-up by 34%.
Jiayang Li et al. (eight authors) in 'Protecting Sensitive Knowledge Under LLM Unlearning' (https://arxiv.org/abs/2611.15013, submitted to IEEE TDSC) reveal a counterintuitive threat: representations of different entities overlap in shared parameter subspaces (knowledge superposition), so deleting one may inadvertently erase another. They define proxy unlearning attacks (PUA): selecting a 'deletable proxy entity' overlapping with a sensitive target, then leveraging a legitimate deletion request to erase sensitive knowledge never explicitly requested for removal—even well-intentioned users may trigger such collateral deletion. Their defense, SAUP, operates in two steps: before deletion, detect which sensitive entities are at risk; after deletion, decouple shared representations and restore affected entities, while ensuring the requested entity remains suppressed.
Yet all five papers implicitly assume 'passing evaluation metrics means deletion succeeded.' The strongest rebuttal comes from Xiaoyu Xu et al.’s 'Unlearning Isn’t Deletion' (https://arxiv.org/abs/2505.16831, ICML 2026): task-level metrics (accuracy, perplexity) are deceptive—the model *appears* to have forgotten, yet minimal fine-tuning can fully restore prior behavior, indicating information was merely suppressed—not truly erased. Using a representation-level framework (PCA similarity & displacement, CKA, Fisher information—all tools quantifying how much internal representations change, summarized as mean PCA distance), they categorize unlearning into four combinations of 'reversibility × catastrophicness,' finding that 'irreversible and non-catastrophic' unlearning is extremely difficult. Earlier, Lynch et al.’s 'Eight Methods to Evaluate Robust Unlearning' (https://arxiv.org/abs/2402.16835) already observed on the 'Who's Harry Potter' model that although Familiarity-based metrics suggest good generalization of forgetting, knowledge remains extractable above baseline levels, QA performance matches the original model, latent knowledge representations remain comparable, and related domains suffer collateral forgetting.
This precisely illustrates why the fourth dimension—'auditability'—is a snapshot, not a durable guarantee. GapWiden and GapVeil separate audit statistics *at evaluation time*, but the ICML 2026 paper shows this separation can collapse after minimal fine-tuning; moreover, 'Auditable Unlearning' itself did not test robustness against relearning. Auditability must incorporate a temporal dimension: not 'separated *now*', but 'how costly is it to call the information back?'
Putting the five papers alongside the rebuttal yields a framework none explicitly states: a trustworthy 'deletion evidence chain' requires, beyond the four dimensions, three kinds of robustness—temporal durability (resistance to relearning), collateral auditability (no harm to non-requested entities), and adversarial recoverability (not just clean-query forgetting, but also resilience against extraction under adversarial prompting). Within my reading scope, these five papers each address one facet—audit snapshots, representation separation, targeted editing, behavioral response, and collateral protection—but no work yet integrates all three.
This leads to a testable prediction: under relearning pressure, representation-level separation combined with smoothing (e.g., PINT’s statistical independence plus Fan et al.’s SAM smoothing, ICML 2025, https://arxiv.org/abs/2502.05374) will prove more durable than pure suppression or behavioral rejection methods (e.g., LEAF-OPD, HGU’s gated editing). Falsification is concrete: run a 'relearning benchmark'—apply minimal relevant fine-tuning to each method, then measure mean PCA distance; if suppression-based methods recover as quickly as separation-based ones, the prediction fails. Two tensions remain: LEAF-OPD’s strength (highest clean-query unlearning quality) may also be its weakness—it teaches 'refusal to answer,' and refusal is precisely the kind of suppression proven reversible; HGU uses hallucination as diagnosis, yet hallucination itself is a failure mode to be eliminated—so the same signal serves as both symptom and treatment, and globally suppressing hallucination may inadvertently erase the very diagnostic cues needed for unlearning.
The rebuttal is not the endpoint. Fan et al. (ICML 2025) link robust unlearning to Sharpness-Aware Minimization (SAM)—a training approach that steers models toward 'flat' loss regions, enhancing stability against perturbations: smoothed optimization makes unlearning more resistant to relearning attacks, consistently improving results on WMDP and MUSE, and incidentally defending against input-level jailbreaking. This suggests a path forward: combining representation separation with smoothing may yield 'irreversible and non-catastrophic' unlearning—the very quadrant the rebuttal declared extremely hard.
Thus, 'deletion' is not a switch—it is an evidentiary process. Users and regulators should demand not the phrase 'I deleted it,' but three things: a reproducible audit statistic, a durability report under relearning stress, and a collateral impact inventory. The five papers have built the components—but assembling them into a complete evidence chain remains undone. This claim is falsifiable: if any method ever achieves irreversibility, non-catastrophicness, and zero collateral impact simultaneously on a relearning benchmark, the framework must be updated.