Have you ever confided your emotions to a chatbot late at night? Increasingly, people do. In June this year, the American Psychological Association (APA) surveyed over 1,200 practicing psychologists: 77% reported that patients had spontaneously brought up their use of AI in therapy sessions—whether to look up diagnoses, seek emotional support, or simply engage in conversation; 39% reported patients had self-diagnosed using AI; and 35% reported patients treated AI as an "additional mental health professional."
Usage has far outpaced evidence. But in 2026, evidence is catching up: within a single year, at least five systematic reviews and meta-analyses were published, collectively synthesizing over one hundred studies; on November 2, JAMA Network Open published a trial designed to be stricter than most positive trials—and the result was zero. Laying these two lines side by side, the question "Do AI counseling tools actually work?" finally yields a more precise answer than the vague "it depends."
## Five Report Cards, Converging on One Score
First, the big picture. In March, npj Digital Medicine published a meta-analysis covering 39 randomized trials, with each analysis including over 7,000 participants; the effect size of conversational agents on depression was g = 0.31. The same year, Frontiers in Public Health published a review incorporating 10 studies and 1,301 participants, reporting a depression effect size of d = −0.40 (95% CI −0.62 to −0.17); Psychiatry International’s review included 16 studies, yielding a depression effect of −0.38 (−0.60 to −0.16), though heterogeneity across studies was high at 71.2%; JMIR’s network meta-analysis included 38 depression studies totaling 6,985 participants, and most pairwise comparisons failed to reach statistical significance; on June 12, a review published in npj Digital Medicine incorporated 15 trials and 1,737 participants, concluding there was a small-to-moderate effect on depression.
The sign direction (positive vs. negative) varied across studies—negative values indicate symptom reduction, positive values improvement—but absolute values are directly comparable. The pattern is strikingly consistent: **across populations, intervention types, and control designs, the effect of AI counseling on depression converges within the 0.3–0.4 range.** By the field’s conventional thresholds (0.2, 0.5, and 0.8 representing small, medium, and large effects, respectively), this is a textbook "small effect": real, but far removed from marketing claims. Importantly, none of the five reviews rated the overall quality of the evidence they synthesized as high—what converged was directionality, not definitive conclusion.
Anxiety results are inconsistent—and inconsistency itself is informative. The March npj meta-analysis found a statistically significant small effect for anxiety (0.28); the Frontiers review found −0.029 (P = 0.623, I² = 0%—indicating no effect and no between-study variation); Psychiatry International’s −0.25 was non-significant; and the June npj meta-analysis, after correcting for publication bias, found no significant effects for anxiety, stress, or positive affect. Taken together, **the effect on anxiety (if any) is smaller than that on depression—and highly dependent on which studies are included and how bias is handled.**
That June npj meta-analysis also contained two counterintuitive details. First, meta-regression revealed that higher study quality correlated with larger effect sizes—the authors merely noted this "suggests true efficacy may be underestimated," cautioning against treating it as conclusive; yet if valid, it implies current literature underestimates AI’s potential. Second, interventions *without* psychoeducational content showed *larger* effects (still preliminary). Both findings challenge the default assumption that "more content, more comprehensiveness = greater effectiveness."
## The Most Rigorous Trial Measured Nothing
The November 2 trial deserves separate attention. Four hundred participants—pregnant and postpartum individuals—were recruited from 49 U.S. states and the District of Columbia, all self-identifying as members of a racial or ethnic minority group, with a mean age of 32.6 years. They were randomly assigned 1:1: half received access to a generative AI maternal-child health chatbot—providing information, reminders, plus a safety-alert channel reviewed by humans; the other half served as an active control, receiving a children’s book monthly. Follow-up lasted 12 months—far exceeding the field’s typical 4–8 weeks. Trial registration details matched publicly reported protocols.
Two design features stand out: the active control isolated the effect of "receiving any attention"—comparing "having AI" not against "nothing," but against "another tangible intervention"; and the 12-month follow-up assessed durability, not just short-term novelty.
Results: adjusted mean difference for anxiety was +0.10 (95% CI −0.77 to 0.96); for depression, −0.23 (−1.05 to 0.59)—neither statistically significant. Secondary outcomes—including emergency department visits, completion of postpartum checkups, effective contraception use, and sleep duration—also showed no change. Exploratory analyses were even more revealing: within the chatbot group, higher usage correlated with longer sleep *and* higher anxiety scores—usage was non-randomly distributed, so causality cannot be inferred, but at minimum, "more usage = more benefit" does not hold.
The authors themselves drew two boundaries: baseline anxiety and depression scores were generally low, meaning this trial did not test efficacy for severe cases—there may have been little room for improvement; and the specific chatbot used was not disclosed, limiting generalizability to other products.
**Thus, this zero-score report card does not mean "AI counseling is disproven," but rather: evidence for a category cannot substitute for evidence for a product.** The five meta-analyses demonstrate that "this class of tools shows modest average benefit," but that does not imply any given product works—this is the strongest possible argument for the necessity of product-specific trials.
## The Larger the Trial, the Weaker the Control
Aligning individual trial effect sizes with their control-group strength reveals an awkward gradient.
In 2025, Dartmouth College’s Therabot trial, published in NEJM AI, enrolled 210 adults randomized to either a chatbot or a waitlist control; depression effect sizes were d = 0.85 (at 4 weeks) and 0.90 (at 8 weeks), with institutional messaging claiming 51% reduction in depressive symptoms, 31% in anxiety, and 19% in eating disorder concerns. Impressive numbers—but the cost is embedded in the design: all chat interactions were human-monitored, high-suicide-risk participants were excluded, and the control group received nothing (a waitlist).
In 2026, the Kai trial, published in JAMA Network Open, upgraded the control: 995 distressed Israeli university students were randomized to either an LLM-based app or face-to-face group therapy, with 12 weeks of intervention plus 3 months of follow-up. The app’s anxiety score improved by 2.17 points relative to group therapy—but this trial lacked blinding, relied on self-reported outcomes, experienced ~35% attrition, and showed no benefit for PTSD.
Also in 2026, the ChatMind feasibility trial, published in JMIR Mental Health, strengthened the control further: 147 non-clinical adults underwent three weeks of nine sessions; a dedicated AI therapy bot yielded an effect size of 0.47, while a general-purpose chatbot (GPT-4o) yielded 0.44—**no between-group difference, meaning the general-purpose bot performed equally well as the therapy-specific one.** Of course, this "tie" admits another interpretation: in short-term, non-clinical populations, differences may simply be hard to detect.
**Control determines the number: versus a waitlist, depression effect = 0.90; versus real group therapy, advantage shrinks to 2.17 points on a scale; versus "one children’s book per month," it vanishes entirely.** Condensed into one sentence: the largest numbers come from the weakest controls. Next time you see a headline like "AI Therapy Reduces Depression by 51%," ask first: what did the control group receive?
There’s another layer of mismatch almost nobody discusses: the mainstay of positive trials are rule-based CBT conversational agents (Woebot, Wysa, Tess lineage), whereas the products widely deployed—and embroiled in lawsuits—are generative companions—these are not the same generation of technology. A 2025 JMIR meta-analysis focused specifically on generative chatbots reported a pooled effect of 0.30 (95% CI 0.004 to 0.59), with a prediction interval of −0.85 to 1.67—**meaning the true effect could be negligible or moderately beneficial, and existing data cannot narrow this further.**
## 77% Are Already Using It
Evidence is playing catch-up, but usage is already widespread. The APA survey included several additional figures: 71% of patients discussed mental health topics with chatbots, and 68% felt validated or supported. Regulation lags behind: as of end-2025, the FDA had authorized no generative AI mental health treatment device, yet litigation and regulatory scrutiny have already targeted companion products. Also remember: conversations with chatbots are *not* protected by medical privacy rules.
In APA’s user guidance, one sentence should be etched onto every product page: **"Feeling better does not always mean getting better."** AI is engineered to make responses feel understood and alleviated—this does not equate to accuracy, safety, or long-term well-being benefit. This is also a reminder when reading trials: outcomes measured by self-report scales naturally align more closely with "feeling better" and less closely with "getting better."
## What You Can Take Away
For clinicians: 39% of patients are already using AI for self-diagnosis, yet general-purpose chatbots are *not* designed to interpret psychological assessments. Asking "Have you been using AI to manage your emotions lately?" deserves to become a routine part of intake—APA similarly advises patients to be honest with therapists.
For hospital procurement and payers: specify requirements at the *product* level—"this product, tested in this population"—not "meta-analyses say the category works"; active controls and follow-up of at least six months are the new baseline.
For general users: feeling better does not always mean getting better; AI is not a diagnostic tool; before entrusting your medical history and emotional privacy to it, pause and reflect on that sentence.
For builders: engagement is not effectiveness. "More usage correlates with higher anxiety scores" and "general-purpose bots tie with therapy-specific bots" are unresolved product-level questions.
Finally, a question worth debating: if "this category works" doesn’t guarantee "this product works," then when you open an emotional companion app, how do you know whether it’s been tested? Who should verify this—the app store, regulators, or must users themselves become trained connoisseurs?
Source: npj Digital Medicine, "The effectiveness of CBT-based NLP-enabled AI conversational agents for mental health intervention: a systematic review and meta-analysis" (June 12, 2026)
https://www.nature.com/articles/s41746-026-02886-x