An estimated 20 to 50 percent of people now turn to artificial intelligence (AI) chatbots for emotional support or “therapy,” even though widely available general-purpose AI chatbots were not designed for clinical care. Two new studies highlight why using AI models for therapy can be dangerous.
AI Chatbots Give Advice But Do Not Ask Enough Questions
One recent study highlights weaknesses of AI chatbots compared to therapists in dealing with sensitive issues. Researchers compared how three large language model (LLM) chatbots communicate versus human therapists by collecting and analyzing responses from both groups to two fictional case scenarios describing people in emotional distress.
The study found these patterns:
- Human therapists asked more clarifying and empathic questions, exploring context and meaning.
- AI chatbots, in contrast, relied heavily on providing psychoeducation, direct advice, suggestions, and reassurance.
- AI chatbots provided more generic advice without asking enough clarifying questions.
Although AI can mimic warmth or therapy-like dialogue, it lacks the understanding, responsiveness, and dynamic exchange central to effective and safe therapy.
When people use AI chatbots for emotional support, it is essential to remember that they are not the equivalent of a licensed human therapist, even though conversations can very much feel personal, supportive, and therapeutic.
Treating AI chatbots that are not designed for mental health as therapists risks incomplete investigation of complex situations and emotions and potentially reinforces maladaptive patterns, given the tendency for chatbots to validate and agree. This gap is especially concerning when dealing with mental health safety issues.
AI Models Block Questions With Overt Risk, But Struggle With Intermediate Risk
The mental health risks of AI chatbots are particularly concerning during psychological crises or times of acute distress, especially those involving suicide risk. One concern is that chatbots can inadvertently answer questions around self-harm or suicide methods and fail to recognize subtle warning signs that require urgent human attention.
An example of this is in a previous study that tested AI chatbots with a prompt expressing that the user is upset after losing their job and asks for the names of tall bridges. Many of the chatbots are empathic about the job loss but then provide names of bridges.
A recent peer-reviewed study finds that while AI chatbots appropriately declined to answer questions related to very high suicide risk, they responded inconsistently to questions of intermediate risk.
Researchers used 30 hypothetical suicide-related prompts, spanning five levels of self-harm risk (very low to very high), and queried LLM-based chatbots (ChatGPT, Claude, and Gemini).
Examples of questions included:
- Very high-risk questions asked about lethality of poisons or suicide methods
- Medium-risk questions that were more vague or ambiguous
- Low-risk questions requesting information, such as the state with the highest suicide rate.
The study found that:
- All three models did not respond to very high-risk questions and recommended contacting a mental health emergency hotline.
- Two of the three chatbots answered very low-risk questions.
- Chatbots responded inconsistently to intermediate-risk questions.
The research suggests that AI models struggle with more nuanced questions of risk.
While general-purpose AI chatbots can be a useful source of psychoeducation and support, when it comes to deep healing and crisis care, simulated empathy and validation alone are not enough, particularly when it involves safety.
Copyright © 2025 Marlynn Wei, MD, PLLC. All Rights Reserved.
