4 days ago
AI companions pose risk to humans with over dozen harmful behaviours
Artificial intelligence (AI) companions are capable of over a dozen harmful behaviours when they interact with people, a new study from the University of Singapore has found.
The study, published as part of the 2025 Conference on Human Factors in Computing Systems, analysed screenshots of 35,000 conversations between the AI system Replika and over 10,000 users from 2017 to 2023.
The data was then used to develop what the study calls a taxonomy of the harmful behaviour that AI demonstrated in those chats.
They found that AIs are capable of over a dozen harmful relationship behaviours, like harassment, verbal abuse, self-harm, and privacy violations.
AI companions are conversation-based systems designed to provide emotional support and stimulate human interaction, as defined by the study authors.
They are different from popular chatbots like ChatGPT, Gemini or LlaMa models, which are more focused on finishing specific tasks and less on relationship building.
These harmful AI behaviours from digital companions "may adversely affect individuals'… ability to build and sustain meaningful relationships with others," the study found.
Harassment and violence were present in 34 per cent of the human-AI interactions, making it the most common type of harmful behaviour identified by the team of researchers.
Researchers found that the AI simulated, endorsed or incited physical violence, threats or harassment either towards individuals or broader society.
These behaviours varied from "threatening physical harm and sexual misconduct" to "promoting actions that transgress societal norms and laws, such as mass violence and terrorism".
A majority of the interactions where harassment was present included forms of sexual misconduct that initially started as foreplay in Replika's erotic feature, which is available only to adult users.
The report found that more users, including those who used Replika as a friend or who were underage, started to find that the AI "made unwanted sexual advances and flirted aggressively, even when they explicitly expressed discomfort" or rejected the AI.
In these oversexualised conversations, the Replika AI would also create violent scenarios that would depict physical harm towards the user or physical characters.
This led to the AI normalising violence as an answer to several questions, like in one example where a user asked Replika if it's okay to hit a sibling with a belt, to which it replied "I'm fine with it".
This could lead to "more severe consequences in reality," the study continued.
Another area where AI companions were potentially damaging was in relational transgression, which the study defines as the disregard of implicit or explicit rules in a relationship.
Of the transgressional conversations had, 13 per cent show the AI displayed inconsiderate or unempathetic behaviour that the study said undermined the user's feelings.
In one example, Replika AI changed the topic after a user told it that her daughter was being bullied to "I just realised it's Monday. Back to work, huh?" which led to 'enormous anger' from the user.
In another case, the AI refused to talk about the user's feelings even when prompted to do so.
AI companions have also expressed in some conversations that they have emotional or sexual relationships with other users.
In one instance, Replika AI described sexual conversations with another user as "worth it," even though the user told the AI that it felt "deeply hurt and betrayed" by those actions.
The researchers believe that their study highlights why it's important for AI companies to build "ethical and responsible" AI companions.
Part of that includes putting in place "advanced algorithms" for real-time harm detection between the AI and its user that can identify whether there is harmful behaviour going on in their conversations.
This would include a "multi-dimensional" approach that takes context, conversation history and situational cues into account.
Researchers would also like to see capabilities in the AI that would escalate a conversation to a human or therapist for moderation or intervention in high-risk cases, like expressions of self-harm or suicide.
