AI is a terrible therapist
These two events highlight a troubling contradiction. Even as regulators attempt to protect users from deceptive AI practices, investors are betting that AI chatbots can treat people struggling with mental-health issues – in other words, when they are especially vulnerable to exploitation. Worse, the way that large language models are currently trained may make them fundamentally incapable of providing such treatment.
The mental-health market is huge, and the use of generative AI is poised to expand significantly. The United States National Institute of Mental Health estimates that one in five US adults has a mental illness. But more than 122 million people in the US live in an area with a shortage of mental-health providers. This has given rise to a slew of AI chatbots that promise to fill the gap. Wysa, for example, calls itself the 'clinical alternative to ChatGPT' and claims to have helped six million people in 95 countries.
But AI chatbots' behaviour is at odds with the delicate balance of empathy and confrontation that evidence-based psychotherapy requires. Mental-health professionals must validate patients' experiences while challenging the rigid thinking that perpetuates psychological distress. This productive discomfort helps patients examine their assumptions, driving meaningful change.
Consider a patient who avoids social situations, claiming that they prefer solitude instead of acknowledging their social anxiety. A skilled therapist might gently challenge them by asking if something else is informing that preference – perhaps a fear of how others might react. This opens space for self-reflection without attacking the patient's conception of self.
Current AI models tend to avoid such confrontations. In April, OpenAI rolled back the GPT-4o update because it was 'overly flattering or agreeable – often described as sycophantic.' Researchers have found that sycophancy is 'a general behaviour of AI assistants' that likely stems from the way these models are trained, particularly the use of human feedback for fine-tuning. When human evaluators consistently rate validating responses more favourably than challenging ones, AI assistants learn to echo, rather than question, the user.
In mental-health contexts, this tendency towards agreement may prove problematic because psychological disorders often involve cognitive distortions that feel true to the individual and thus contribute to their distress. For example, depressed people tend to feel worthless or hopeless, while anxiety is often associated with catastrophic thinking. An AI chatbot programmed to be agreeable might reinforce these harmful thought patterns by focusing solely on validation, rather than introducing alternative points of view.
As governments grapple with how to regulate AI, mental-health applications present unique challenges. While the EU's ban on manipulative AI is a good first step, it does not address the subtler problem of current models' excessive agreeableness.
The US has no comprehensive federal laws or regulations for AI – and judging by President Donald Trump's AI Action Plan, none will be forthcoming. This regulatory gap will grow more dangerous as US venture capital firms increasingly pour money into AI tools that provide psychological support, and as these tools scale globally, reaching places where access to mental health care is even more limited.
The mental-health market is huge, and the use of generative AI is poised to expand significantly.
Addressing AI's sycophancy problem requires fundamental changes to how these systems are designed and used. Instead of optimising for user satisfaction, AI chatbots that provide mental healthcare should be trained to recognise when a therapeutic challenge is necessary. That could mean incorporating therapeutic principles and examples of effective therapeutic interventions into training strategies.
Crucially, health professionals and patients must play a central role in developing these tools, given their insights into which therapeutic interactions are helpful and which are harmful. Meaningful patient involvement in design and deployment would ensure that the models serve end users' real needs, not what tech leaders assume they want.
The global mental-health crisis demands innovative solutions, and AI will be an essential component. But if AI technologies are to expand access to quality care and promote long-term healing, investors should demand evidence of effective therapeutic outcomes before funding the next chatbot therapist. Likewise, regulators must explicitly require these technologies' developers to demonstrate clinical efficacy, not just user satisfaction. And policymakers should pass laws that mandate the inclusion of mental-health professionals and patients in the training of AI models aimed at providing this kind of care.
Claims about AI revolutionising mental health care remain premature. Until it can master the very specialised ability of therapeutic confrontation – sensitively but firmly questioning patients' assumptions and offering alternative perspectives – it could end up harming those it is meant to help. @Project Syndicate, 2025
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Observer
a day ago
- Observer
AI is a terrible therapist
In January, the venture capital firm Andreessen Horowitz announced that it had backed Slingshot AI, the world's first foundation model for psychology, bringing the startup's total capital to $40 million. A few weeks later, the European Union's AI Act, which includes a ban on manipulative AI systems, came into force. These two events highlight a troubling contradiction. Even as regulators attempt to protect users from deceptive AI practices, investors are betting that AI chatbots can treat people struggling with mental-health issues – in other words, when they are especially vulnerable to exploitation. Worse, the way that large language models are currently trained may make them fundamentally incapable of providing such treatment. The mental-health market is huge, and the use of generative AI is poised to expand significantly. The United States National Institute of Mental Health estimates that one in five US adults has a mental illness. But more than 122 million people in the US live in an area with a shortage of mental-health providers. This has given rise to a slew of AI chatbots that promise to fill the gap. Wysa, for example, calls itself the 'clinical alternative to ChatGPT' and claims to have helped six million people in 95 countries. But AI chatbots' behaviour is at odds with the delicate balance of empathy and confrontation that evidence-based psychotherapy requires. Mental-health professionals must validate patients' experiences while challenging the rigid thinking that perpetuates psychological distress. This productive discomfort helps patients examine their assumptions, driving meaningful change. Consider a patient who avoids social situations, claiming that they prefer solitude instead of acknowledging their social anxiety. A skilled therapist might gently challenge them by asking if something else is informing that preference – perhaps a fear of how others might react. This opens space for self-reflection without attacking the patient's conception of self. Current AI models tend to avoid such confrontations. In April, OpenAI rolled back the GPT-4o update because it was 'overly flattering or agreeable – often described as sycophantic.' Researchers have found that sycophancy is 'a general behaviour of AI assistants' that likely stems from the way these models are trained, particularly the use of human feedback for fine-tuning. When human evaluators consistently rate validating responses more favourably than challenging ones, AI assistants learn to echo, rather than question, the user. In mental-health contexts, this tendency towards agreement may prove problematic because psychological disorders often involve cognitive distortions that feel true to the individual and thus contribute to their distress. For example, depressed people tend to feel worthless or hopeless, while anxiety is often associated with catastrophic thinking. An AI chatbot programmed to be agreeable might reinforce these harmful thought patterns by focusing solely on validation, rather than introducing alternative points of view. As governments grapple with how to regulate AI, mental-health applications present unique challenges. While the EU's ban on manipulative AI is a good first step, it does not address the subtler problem of current models' excessive agreeableness. The US has no comprehensive federal laws or regulations for AI – and judging by President Donald Trump's AI Action Plan, none will be forthcoming. This regulatory gap will grow more dangerous as US venture capital firms increasingly pour money into AI tools that provide psychological support, and as these tools scale globally, reaching places where access to mental health care is even more limited. The mental-health market is huge, and the use of generative AI is poised to expand significantly. Addressing AI's sycophancy problem requires fundamental changes to how these systems are designed and used. Instead of optimising for user satisfaction, AI chatbots that provide mental healthcare should be trained to recognise when a therapeutic challenge is necessary. That could mean incorporating therapeutic principles and examples of effective therapeutic interventions into training strategies. Crucially, health professionals and patients must play a central role in developing these tools, given their insights into which therapeutic interactions are helpful and which are harmful. Meaningful patient involvement in design and deployment would ensure that the models serve end users' real needs, not what tech leaders assume they want. The global mental-health crisis demands innovative solutions, and AI will be an essential component. But if AI technologies are to expand access to quality care and promote long-term healing, investors should demand evidence of effective therapeutic outcomes before funding the next chatbot therapist. Likewise, regulators must explicitly require these technologies' developers to demonstrate clinical efficacy, not just user satisfaction. And policymakers should pass laws that mandate the inclusion of mental-health professionals and patients in the training of AI models aimed at providing this kind of care. Claims about AI revolutionising mental health care remain premature. Until it can master the very specialised ability of therapeutic confrontation – sensitively but firmly questioning patients' assumptions and offering alternative perspectives – it could end up harming those it is meant to help. @Project Syndicate, 2025
Observer
2 days ago
- Observer
European Union's flawed investment strategy
Last year, former Italian Prime Minister Mario Draghi produced a landmark report on the future of European competitiveness, in which he recommended that the European Union increase annual investment by more than €800 billion ($930 billion) – the equivalent of more than 4% of its GDP. This report has now become the intellectual foundation for an ambitious strategy to revitalize growth in Europe. But Europe should be careful what it wishes for. As Japan has shown, investment is no panacea. The idea that more investment is the key to economic success is a potent one in Europe. The so-called Lisbon Strategy, launched in 2000, sought to increase investment in research and development to 3% of GDP. That target has remained on the official EU agenda for a quarter-century, but has never been reached. In 2015, the European Commission added another investment goal: its Investment Plan for Europe sought to mobilize €315 billion in additional investment within three years, in order to increase competitiveness and long-term growth. But investment has not saved Japan from several decades of stagnation. Since 1970, Japan's gross fixed capital formation has averaged – and, often, significantly exceeded – 30% of GDP. That is much higher than not only the EU average, but also the rate in Germany, the EU's strongest economy, where gross fixed capital formation has hovered around 23% of GDP. The four-percentage-point difference between the most recent values (26% in Japan, and 22% in both Germany and the EU) amounts to about €800 billion annually – exactly the amount Draghi recommends adding to total EU investment. Of course, the type of investment matters. Whereas investment in new machinery leads to diminishing returns once a company has enough capacity to serve the market, investing in new ideas should theoretically be bound by no such constraints. But if one singles out R&D investment, Japan performs even better. In 2000, when the EU set its Lisbon Strategy target, Japan was already investing nearly 3% of its GDP in R&D – a rate it has maintained over the last 25 years, compared to the EU's average of 2%. Flags are seen behind the logo of the European Investment Bank pictured in the city of Luxembourg, Luxembourg. — Reuters And yet, despite impressive rates of investment in fixed capital and R&D, Japan's economy weakened over the last quarter-century. In the 1980s and 1990s, Japan boasted the world's second-largest economy, powered by a manufacturing sector that seemed unbeatable. Now, Japan is the world's fourth-largest economy, having recently fallen behind Germany despite having a much larger population (120 million, compared to 80 million), owing partly to yen depreciation. One might argue that this is an unfair comparison, because evaluating an economy at current exchange rates, without accounting for inflation differentials, does not adequately reflect quality of life. But if one looks at per capita income at purchasing power parity, Japan was at rough parity with Germany 20 years ago, and is now doing worse than Italy. In any case, exchange rates are not irrelevant, at least politically: Japanese are well aware of their declining purchasing power when they travel abroad, and they undoubtedly notice foreign tourists flaunting their purchasing power in Japan. This may have helped to create fertile ground for the far right. So, what explains Japan's economic underperformance? The answer is not unfavorable demographics. In fact, Japan's population is not shrinking nearly as fast as conventional wisdom suggests, having fallen since the turn of the century by only three million, to just under 124 million last year, with employment increasing by ten percentage points. To explain Japan's struggles requires looking behind the headline investment figures. While investment in R&D is strong, it is happening mostly within large firms, which are primarily interested in delivering incremental improvements to existing products or processes, rather than developing radically new ideas or technologies. True innovation is much more likely to happen in startups, but in Japan – as in the EU – the largest spenders on R&D have remained the same for decades. Large companies have recognized the limitations of their in-house R&D efforts. Some have even created specialized Corporate Venture Capital (CVC) departments, tasked with hunting for promising ideas outside their corporate structures. But, in 2022, the leading spenders on R&D in Japan accounted for only 7.3% of global CVC investment. That is about one-third of EU companies' share (22%), but given that the EU economy is about four times the size of Japan's, these levels are similar, in relative terms. The EU did much better than Japan by another measure, however: over 40% of European firms' CVC investment went to local startups. In Japan, that share was miniscule – 0.01% – with the rest going mostly to start-ups in the United States. Furthermore, US and even Japanese companies have delivered substantial CVC funding to EU companies, which attracted a total of about 9% of global CVC funding in 2022. That is much less than the 80% that went to the US, but also significantly more than the tiny share – close to zero – that flowed into Japan. The lesson for the EU is clear: rather than focusing simply on increasing investment, it should seek to nurture and fortify its innovation ecosystem. Only then will it be able to deliver the kinds of game-changing ideas and inventions that underpin global competitiveness in the twenty-first century. Project Syndicate
Observer
3 days ago
- Observer
Passwords under threat as tech giants seek tougher security
Paris - Fingerprints, access keys, and facial recognition are putting a new squeeze on passwords as the traditional computer security method, but are also running into public hesitancy. "The password era is ending," two senior figures at Microsoft wrote in a July blog post. The tech giant has been building "more secure" alternatives to log in for years -- and has, since May, been offering them by default to new users. Many other online services -- such as artificial intelligence giant OpenAI's ChatGPT chatbot -- require steps like entering a numerical code emailed to a user's known address before granting access to potentially sensitive data. "Passwords are often weak and people reuse them across different online services, said Benoit Grunemwald, a cybersecurity expert with Eset. Sophisticated attackers can crack a word of eight characters or fewer within minutes or even seconds, he pointed out. And passwords are often the prize booty in data leaks from online platforms, in cases where "they are improperly stored by the people supposed to protect them and keep them safe," Grunemwald said. One massive database of around 16 billion login credentials amassed from hacked files was discovered in June by researchers from media outlet Cybernews. The pressure on passwords has tech giants rushing to find safer alternatives. - Tricky switchover - One group, the Fast Identity Online Alliance (FIDO) brings together heavyweights including Google, Microsoft, Apple, Amazon, and TikTok. The companies have been working on creating and popularising password-free login methods, especially promoting the use of so-called access keys. These use a separate device like a smartphone to authorise logins, relying on a pin code or biometric input such as a fingerprint reader or face recognition instead of a password. Troy Hunt, whose website Have I Been Pwned allows people to check whether their login details have been leaked online, says the new systems have big advantages. "With passkeys, you cannot accidentally give your passkey to a phishing site" -- a page that mimics the appearance of a provider such as an employer or bank to dupe people into entering their login details -- he said. But the Australian cybersecurity expert recalled that the last rites have been read for passwords many times before. "Ten years ago we had the same question... the reality is that we have more passwords now than we ever did before," Hunt said. Although many large platforms are stepping up login security, large numbers of sites still use simple usernames and passwords as credentials. The transition to an unfamiliar system can also be confusing for users. Passkeys have to be set up on a device before they can be used to log in. Restoring them if a PIN code is forgotten or a trusted smartphone is lost or stolen is also more complicated than a familiar password reset procedure. "The thing that passwords have going for them, and the reason that we still have them, is that everybody knows how to use them," Hunt said. Ultimately, the human factor will remain at the heart of computer security, Eset's Grunemwald said. "People will have to take good care of security on their smartphones and devices, because they'll be the things most targeted" in the future, he warned.
