Human vs AI: Who's Better at Cognitive Behavioral Therapy

Medscape

23-05-2025

LOS ANGELES — Artificial intelligence (AI) falls short of human therapists when it comes to empathy and emotional connection in the delivery of cognitive behavioral therapy (CBT), initial results of a new pilot study suggested.
However, the results showed that AI performed well in providing a structured therapeutic approach.
'While AI may offer structured CBT components and serve as a supplementary or triage tool, it lacks the nuance and flexibility to serve as a stand-alone therapy,' study investigator Esha Aneja, a fourth-year medical student at California Northstate College of Medicine, Elk Grove, California, told Medscape Medical News .
'Physicians and therapists should view AI as a potential adjunct, not an alternative.' 'Human oversight, ethical safeguards, and empathy remain essential to safe and effective mental health care.'
The findings were presented on May 17 at the American Psychiatric Association (APA) 2025 Annual Meeting.
Demand for CBT Outstripping Supply
Currently, there aren't enough psychiatric professionals in the United States — or globally — to meet the growing demand for CBT.
Patients frequently face delays in accessing care, so more are turning to AI tools like ChatGPT to address their mental health needs, said Aneja.
However, she noted in her presentation that large language model (LLM)–based AI chatbots for text-based therapy are still largely theoretical in psychiatric literature.
While LLMs have been integrated into electronic health records for diagnostic purposes, the ability of AI to execute CBT remains understudied.
The goal of the study was to compare the effectiveness of therapy delivered by humans with AI.
Experts familiar with CBT principles using the Cognitive Therapy Rating Scale (CTRS) compared a human therapist with an AI model (ChatGPT-3.5) in responding to a third-party patient presenting with a specific mental health concern.
CTRS is a gold-standard observational tool for assessing the quality and fidelity of CBT sessions. It evaluates multiple domains, each rated on a 0-6 scale, with higher scores reflecting more skilled therapeutic delivery.
Both the human therapist, who conducted the session over Zoom, and the AI therapist, ChatGPT-3.5 (the most current version at the time), interacted with the patient solely via text chat. Reviewers received transcripts of each session but were blinded to whether the responses came from a human or AI.
The study surveyed 75 reviewers to compare the quality of human-based and ChatGPT-3.5-based interactions with patients. Participants included medical students, social work students, psychiatric residents, and board-certified psychiatrists.
Humans Win the Day
The human therapist outperformed ChatGPT-3.5 across all domains. Areas where the differences in mean CTRS scores were statistically significant included feedback (4.48 vs 3.03), collaboration (4.91 vs 3.84), pacing (4.60 vs 3.67), and guided discovery (0.35 vs 3.45), as well as 'focus on key cognitive behaviors' and 'application of CBT techniques' ( P = .001 for all).
Areas where the ratings were similar between the two groups included agenda setting, understanding, interpersonal effectiveness, and strategies for change.
When it came to therapeutic approach and empathy, respondents disagreed on whether the human therapist demonstrated enough empathy, Aneja reported.
'Some praised their warmth and responsiveness, while others felt the therapist focused too much on technique and missed emotional cues,' she said. 'In contrast, AI was more uniformly described as 'robotic' or 'surface-level' in its empathy, with little variation.'
While AI may become 'cognitively empathetic' in the future and therefore able to respond more appropriately, 'emotional or embodied empathy, the kind that comes from shared human experience, is beyond its current capabilities,' said Aneja.
And, even in the areas that were more compatible with AI such as structure and agenda, respondents felt AI was 'too wordy' and 'robotic' and included 'a lot of lecturing,' she added. They also noted AI lacked personalized recommendations with respect to patient understanding and tailored approaches.
While the researchers suspected AI might fall short, this new study 'quantifies and contextualizes those limitations in a real-world CBT framework,' said Aneja.
AI could 'definitely' be used as a screening tool in psychiatry, particularly when patients can't get to see a provider in a timely manner, she said. It could 'look for things like suicidality or situations where urgent attention is important.'
However, therapists should keep the tool's limitations in mind, especially the empathy component, she added.
Weighing in on these results, Howard Liu, MD, chair of the Department of Psychiatry at the University of Nebraska Medical Center, Omaha, Nebraska, and chair of the APA Council on Communications, Washington, DC, called the study 'fascinating,' especially with the backdrop of psychiatrist shortages across the country.
However, he stressed the importance of informing patients when using AI. 'Different health systems have different policies about whether you can, in fact, feed in protected health information into these systems,' he pointed out.
Philip R. Muskin, MD, professor of psychiatry at the Columbia University Irving Medical Center, New York City, said he was not surprised by the findings overall or the comments about the 'lecture-like quality' of the AI 'therapist.'
'Human responses vary, even when rigidly following a CBT agenda,' he told Medscape Medical News .
'Reading about therapy, which is essentially what the AI software does, isn't comparable to a therapist who has read training materials but has incorporated the information through human interaction.'