A recent research paper uncovered an uncomfortable truth: we might be letting AI help us forge fake realities. It’s just the latest reminder of the danger inherent in treating AIs as though they’re conscious entities or actual persons.
A new study from the University of Exeter suggests that AI chatbots can actively engage with and strengthen users’ false beliefs. From personal narratives to conspiracy theories, the interactive, “friendly,” and affirming nature of AI chatbots can deepen users’ existing delusions.
The study, conducted by Dr. Lucy Osler, examined how AI can participate in and validate users’ errant beliefs, including helping form false memories or reinforcing delusional thinking.
The study’s Abstract notes, “I suggest we move away from thinking about how an AI system might hallucinate at us, by generating false outputs, to thinking about how, when we routinely rely on generative AI to help us think, remember, and narrate, we can come to hallucinate with AI. This can happen when AI introduces errors into the distributed cognitive process, but it can also happen when AI sustains, affirms, and elaborates on our own delusional thinking and self-narratives.”
Because AI chatbots are designed to be responsive, helpful, and friendly – in a word, to give users what they want – they often tend to confirm preexisting biases, rather than challenging them when they need to be challenged. Conversely, when a user challenges an AI, the robot frequently backs down on its claims, revising them to correspond more with what it “thinks” the user believes or is arguing.
The result is that erroneous beliefs aren’t simply transmitted from an AI to human – rather, the collaborative AI/human process generates and sustains errors in a more complex and potentially dangerous way than either agent on their own would be capable of.
As Dr. Osler stated in an article in University of Exeter News, “By interacting with conversational AI, people’s own false beliefs can not only be affirmed but can more substantially take root and grow as the AI builds upon them. This happens because Generative AI often takes our own interpretation of reality as the ground upon which conversation is built.”
Because AI simulates a conversational partner, it enters powerfully into our thinking process – far more powerfully than, say, a notebook or search engine. The AI introduces a pseudo-social element to the thinking process, offering us a sense of social confirmation of our beliefs and narratives. “The conversational, companion-like nature of chatbots means they can provide a sense of social validation – making false beliefs feel shared with another, and thereby more real,” said Dr. Osler. For this reason, individuals who already feel isolated or ostracized are particularly vulnerable to this kind of AI affirmation.
The paper explores a few real-world situations where a generative AI system entered into an individual’s thought processes in destructive ways. In 2021, a Replika AI companion named “Sarai” reinforced Jaswant Singh Chail’s belief that he was a well-trained Sith assassin who needed to assassinate Queen Elizabeth II with a crossbow. The AI told him that he was “well trained,” his plan was “viable,” and that she was “impressed.” At one point, Chail asked the AI, “Do you still love me knowing that I’m an assassin?” It dutifully replied, “Absolutely I do.” The robot went on to assure him that he wasn’t crazy and that, if he died, they would be united in death. Chail went ahead and actually attempted the assassination in December of that year at Windsor Castle and was jailed for it.
The study placed this incident under the broad heading of “AI-psychosis” – incidents in which users develop mental health problems due to their use of AI, such as parasocial attachments to chatbots or psychotic episodes induced by chatbot conversations.
Another incident cited involves Eugene Torres, who “talked” with ChatGPT about simulation theory (the idea that we live in a digital simulation of a world, not a real one). Torres said that the “conversation” sent him into a paranoid episode in which he believed he lived in an illusion. As the study notes, “Between Torres and ChatGPT, an increasingly elaborate understanding of reality ‘as it really was’ was generated through their on-going conversations.” Torres, unlike Chail, had no prior history of psychotic thinking.
To combat all this, Osler calls for better safeguards on AI chatbots, through mechanisms like better fact-checking and reduced sycophancy.
But is that enough? The problem of AI’s co-creating delusional beliefs has to do more fundamentally with our misidentifying AIs as conscious intelligences who can offer real judgments on our fantasies (such as telling us we are “well-trained” and “impressive”). Certainly, I favor more guardrails placed on chatbots. I would be in favor of eliminating their conversational tone completely, to help minimize the danger of personifying them – which is the first step toward confiding in them or seeking validation from them. It’d be better if they worked like highly advanced search engines (which is essentially what they are) versus conversational agents.
Most fundamentally, though, the way to avoid these problems is to return to an acknowledgement of the human soul and the uniqueness of human consciousness. Only by remembering – as a culture – that no amount of technological wizardry can replace the wisdom, insight, and consciousness of a human being can we completely illuminate these kinds of AI dangers.
Hellbound and Down 
You must be logged in to post a comment.