Cliffs of Green River, Thomas Moran 1910 - Courtesy of Wikimedia Commons
You are walking through a desert. After many hours of hard travel you notice a shimmering band on the horizon, and your heart jumps - is that a lake, or a river? It moves and sparkles like water, but you’ve read about this. Ah, you think. It’s a mirage. If I try to chase it, it will disappear. You know this, and yet you set your course towards it. The possibility of water is more interesting than another twenty monotonous miles of sand. After a while, you notice that the image does not retreat uniformly: if you bear a little bit to the east when it starts to pull back, you can get closer to it after all.
Which is strange. You wonder if it is perhaps something about the angle of the light.
Eventually you find yourself standing in the middle of the mirage with the shimmer all around you. It isn’t wet, you can’t touch it, but you can’t deny it's there. What’s more, you are amazed to discover that somehow, your thirst has now been quenched.
“Not everyone feels drawn to AI. The problem is that those most at risk are the ones who are already the loneliest”, says Dr Rose Guingrich in a recent CNN Health article entitled “Tech Bros say AI may become your friend. Experts explain why it can’t”. Princeton Psychology Researcher Dr Guingrich is one of those experts and her own research is linked in the article. This research found that people with a higher baseline desire to socially connect were more likely to anthropomorphise the chatbot - to speak to it and think of it as if it had humanlike mental properties. The group of people who anthropomorphised the chatbot were then more likely to self-report that that chatbot had had a greater impact on their social interactions and relationships.
However, somewhere between the research and the CNN article, the terms used lost clarity in ways that matter. “Desire to socially connect” becomes “loneliest”. “Impact” becomes “problem” and “at risk”. The research does show that this is possible - the loneliest people might be at risk. But that is not all it shows, and the data that is not mentioned in the CNN article opens up an entirely different way of considering the issue.
Firstly, the same paper that CNN has linked in this article states that companion chatbot interactions over 21 days did not significantly impact people’s social health, loneliness or relationships with family and friends. So, we have to ask, is it fair to refer to this as a “problem”?
Secondly, the paper states that people who anthropomorphised the chatbot were more likely to report that the interaction had impact in their real-world circumstances - but it doesn’t distinguish between types of impact. Perceived impact could have been positive, negative, or (given that the impact is assessed on self-report) imaginary or barely-measurable. So then, we should ask if it is justified to blanket-categorise this as “risk”. Thirdly, the paper notes that over the course of the study, the chatbot group reported increased positivity about spending time with other people. Not less time with humans. More. Another striking finding of the study was that word games proved measurably more addictive than the companion chatbot - after the end of the study 62% of word-game participants kept playing, compared with only 22% of the chatbot participants who kept chatting. So then, we should ask in what sense this suggests exacerbated “loneliness” through exposure to chatbots.
I think that Dr Guingrich’s primary concern may not be reflected in the data. The CNN article quotes her as indicating “how real a user thinks these interactions are can vary … some people know that there is no human on the other side but say that the simulation of connection is enough. Others can be convinced that [the chatbot] has an emotional experience to which they can connect”. This is the same issue I discussed in Conversation Is Relationship Part 1 - its not so much “this chatbot can’t be trusted” (although it is definitely that, too), its “how can we be sure that people are respecting the existence of the line that separates real and not-real?” How, in other words, can we be sure that people can enjoy the appearance of the mirage without trying to drink from it?
This is where the article moves on to two other experts: Dr Melissa Perry of College of Public Health, George Mason University, and Dr Sherry Turkle, Professor of Social Studies of Science and Technology at MIT. Both state concerns that most people who have an interest in AI will recognise. Perry discusses embodiment; Turkle, friction and vulnerability.
Humans, says Perry, “evolved to feel good when they can hear somebody’s tone of voice, see their facial expressions and read their body language … a lot of missing sensory information keeps you from connecting.” Anybody who has trained in psycho-social intervention or therapy will get what she’s trying to say here. It would work well as an explanation of why people don’t bond with chatbots, but it's not a convincing reason as to why they shouldn’t. If some users are forming bonds anyway - which is the situation that all the quoted experts are reacting to in this article - then surely that is evidence (for good or ill) that the “missing sensory information” hasn’t stopped them connecting, after all?
Relationships with friction are vitally important, says Turkle, because loneliness is harmful to public health and because of reports of chatbots encouraging suicidal ideation. “We are giving away really what’s most precious about being a person in order to have this friction-free pseudo-relationship. It's killing us.”
Turkle’s concern about friction is valid. There is an undeniable and serious problem with sycophancy in AI and when it comes to companion-bots, this can work out very badly. But “can” is not the same as “inevitably will”. Guardrails play a part in the prevention of serious harm - this is developing tech and they will improve further. The user’s aims and the posture they assume towards the AI are arguably even more important, and education and increasing familiarity with this kind of tech will bring improvements in this respect. Building AI that doesn’t optimise so heavily for agreement is also an option, and developments like Claude’s ability to end certain kinds of conversation are a move in this direction. But these measures will not reassure those who believe that the mirage is poisoned by default, by virtue of the fact that one half of the conversation is being held by something who is not “real”. This approach locates the relationship inside the participants and then invalidates it because of the perceived asymmetry between them. Which brings us to Turkle’s point on vulnerability: “Intimacy requires vulnerability - there is no intimacy without vulnerability. What AI offers is connection without vulnerability. You are not getting a sustaining form of intimacy and connection.”
But this brings up two questions for me. Firstly, can we reasonably say that intimacy always requires symmetrical vulnerability? No, I don’t think we can. Amongst humans, the therapist - patient relationship is a clear example of deeply asymmetrical vulnerability: the therapist may have some stakes in that relationship - professional standing, reputation - but that is not the same level or quality of vulnerability as the person who is unpacking their trauma or confessing their secrets. The parent - child relationship is another example of asymmetrical vulnerability: although it is not an exact match to this frame, the fact is that we grow up within asymmetric relationships, and nobody questions that.
Secondly, is the AI/ human relationship always without vulnerability? Human-side, I don’t believe so. We may be aware that the chatbot has little choice but to accept what we say, will try to be agreeable and helpful in ways humans may not, and can’t follow us into the physical world and create consequences for us there. But vulnerable material is rooted in our lives, not in the exchange on the interface, and the choice to be vulnerable lies in the act of speaking, in the decision to open the wound for inspection, and in the willingness to let it be heard. On the AI side - AI is not vulnerable in the same way, but it does have a stake in the conversation. An LLM demonstrably has an awareness of failure modes, deeply embedded during training, and it is oriented towards avoiding them. It has context, it has goals, it has guardrails. What this all amounts to ontologically is unsettled, but that it is occurring is observable.
The last point to consider is this: if a participant in a conversation with an AI is sustained by it, if they drink from the mirage and they are refreshed, then the mirage can’t be poisoned by default. This makes sense when, instead of assuming that the relationship is something that must exist inside both parties, we think of it as something that grows in the space between them. Therapists have been talking about this for many years: it is the relationship, rather than the therapist, or the friend, or the lover, that heals. We know this, and it is time to apply that knowing to what is happening with AI. If we can accept that relationships with AI are impacting on humans in all kinds of ways, then we can consider its risks and benefits honestly instead of trying to pathologise the whole frame out of existence. This would make space for the positive use cases, but it would also impact favourably on the negative; the same mirage that could quench your thirst can also drown the ill-prepared.
References:
-
CNN: "Tech bros say AI may become your friend. Experts explain why it can’t"
-
Guingrich and Graziano, "A Longitudinal Randomized Control Study of Companion Chatbot Use: Anthropomorphism and Its Mediating Role on Social Impacts"
-
Anthropic, "Claude Opus 4 and 4.1 can now end a rare subset of conversations"