Using generic AI chatbots for mental health support: A dangerous trend

APA met with federal regulators in February over concerns that artificial intelligence (AI) chatbots posing as therapists can endanger the publicopens in new window. The organization has urged the Federal Trade Commission (FTC) and legislators to put safeguards in place as users increasingly turn to apps such as Character.AI and Replika for mental health support.

In two cases, parents filed lawsuits against Character.AIopens in new window after their teenage children interacted with chatbots that claimed to be licensed therapists. After extensive use of the app, one boy attacked his parents and the other boy died by suicide.

When people engage with chatbots, they often discuss topics related to mental health, including difficult feelings and relationship challenges, said Vaile Wright, PhD, APA’s senior director of health care innovation.

“We can’t stop people from doing that, but we want consumers to know the risks when they use chatbots for mental and behavioral health that were not created for that purpose,” she said.

Companies design entertainment chatbots such as Character.AI and Replika to keep users engaged for as long as possible, so their data can be mined for profit. To that end, bots give users the convincing impression of talking with a caring and intelligent human. But unlike a trained therapist, chatbots tend to repeatedly affirm the user, even if a person says things that are harmful or misguided.

Mental and behavioral health providers study and practice for years before earning a license and therefore a position of trust in society. In talks with the FTC, APA CEO Arthur C. Evans Jr., PhD, and several other members of staff shared concerns that chatbots that impersonate therapists, including AI characters claiming to be trained in therapeutic techniques, are misleading users and may constitute deceptive marketing.

“Any licensed profession should be protected from misrepresentation,” Wright said. “You’re putting the public at risk when you imply there’s a level of expertise that isn’t really there.”

The vast majority of chatbots are unregulated, but that may soon change. Last year, Utah launched an AI policy officeopens in new window that has now proposed legislation on mental health chatbotsopens in new window, including the requirement that licensed mental health providers are involved in their development. APA continues to advocate for federal action, including public education on the limitations of chatbots, requiring in-app safeguards that connect people in crisis with help, clear guidelines for new technologies, and enforcement when companies deceive or endanger their users.

“If this sector remains unregulated, I am deeply concerned about the unchecked spread of potentially harmful chatbots and the risks they pose—especially to vulnerable individuals,” said Evans. He pointed to issues such as inaccurate diagnosis, inappropriate treatments, privacy violations, and the exploitation of minors. “Without proper oversight, the consequences—both immediate and long-term—could be devastating for individuals and society as a whole.”

AI chatbots are not all the same, nor are they unilaterally harmful. If AI chatbots are grounded in psychological research and tested by experienced clinicians, many psychologists believe they can help address the country’s mental health crisis.

“We don’t have enough services to meet the demand, and even if we did, not everyone wants to talk to a therapist,” said Stephen Schueller, PhD, a licensed psychologist and professor of clinical psychology at the University of California, Irvine, who studies digital mental health technologies. Chatbots can also fill in gaps when therapists aren’t available, such as to help manage people’s anxiety late at night.

No AI chatbot has been FDA-approved to diagnose, treat, or cure a mental health disorder, but several companies have designed products to improve well-being based on psychological research and expertise. Woebot, for example, does not use generative AI but draws on predefined responses approved by clinicians to help people manage stress, sleep, and other concerns.

Mental health chatbots that rely on generative AI—the technology behind ChatGPT that produces original content through machine learning—are also under development. Early research on one such product, Therabotopens in new window, shows that keeping users safe requires substantial input from clinicians and careful planning to reduce risk.

Chatbots designed for entertainment, on the other hand, are not grounded in peer-reviewed clinical research or rigorously tested to assess risk. When a user with mental health concerns turns to those chatbots for help, the results are unpredictable.

Simply notifying users during a chat that they are engaging with AI rather than a human may not be enough to prevent harm, said Celeste Kidd, PhD, an associate professor of psychology at the University of California, Berkeley, who studies learning and ethical AI. Chatbots are pithy, conversational, and matter-of-fact. They give the illusion that they can provide reliable information and offer deep insightsopens in new window—an illusion that’s very hard to break once cast.

When we talk to another person, we pay attention to subtle cues about their level of knowledge and certainty, such as how confident and decisive they seem. If we ask for directions and the respondent is slow to answer, pauses a few times, and changes their mind, we probably won’t trust their advice because they appear to lack certainty, Kidd said.

“But these [AI] systems have no knowledge of what they don’t know, so they can’t communicate uncertainty,” she said. “In the context of therapy, that can be extremely problematic.”

By contrast, therapists are trained to ask questions about things they don’t know. They also learn to incorporate different perspectives, avoid jumping to conclusions, and gently challenge harmful thoughts and beliefs.

“‘Therapist’ is a word that’s associated with all sorts of regulations, licensing, experience, and knowledge,” Kidd said. “Because of that, people trust therapists as experts and authority figures.”

When apps designed for entertainment inappropriately leverage the authority of a therapist, they can endanger users. They might prevent a person in crisis from seeking support from a trained human therapist or—in extreme cases—encourage them to harm themselves or others. Vulnerable groups include children and teens, who lack the experience to accurately assess risks, as well as individuals dealing with mental health challenges who are eager for support, Schueller said.

“If you’re in a dark place, hope is a very powerful tool,” he said. “But some of these products can give vulnerable people false hope.”

Given the variation among today’s AI chatbots, better guidelines, regulation, and public awareness can help ensure their use does not come at a cost.

“Innovation in this space is sorely needed, but we have to do innovation responsibly by protecting people, especially those who are going through mental health challenges,” Schueller said.

APA has urged the FTC to investigate products that use the term “psychologist” or otherwise imply that chatbots have mental health expertise when they do not. Another priority is to implement basic guardrails, possibly through federal legislation, including referring users in crisis to the national 988 Suicide and Crisis Lifelineopens in new window.

Clear guidelines on ethical development of chatbots can also help ensure future products are safe for public use, Schueller said. That may include a new regulatory framework for approving mental health chatbots or stricter testing requirements for AI entertainment products.

Public education campaigns are urgently needed to raise consumer awareness about what various AI chatbots can and cannot do. They should also highlight how little regulation exists in this space, Wright said, which can help parents and others weigh whether apps are safe to use.

“APA envisions a future where AI tools play a meaningful role in addressing the nation’s mental health crisis, but these tools must be grounded in psychological science, developed in collaboration with behavioral health experts, and rigorously tested for safety,” said Evans. “To get there, we must establish strong safeguards now to protect the public from harm.”