AI in Mental Health: Claiming to Be Human or Assuming Its Nature as a Tool?

Why anthropomorphism in wellbeing AI seeks to comfort rather than resolve.

In recent years, emotional support applications have multiplied. Listening chatbots, around-the-clock assistants, anonymous spaces to talk: supply is abundant, and the promise is usually the same. To be heard, at any time, without judgment.

This promise responds to a real need. In France, waiting times to see a state-subsidised psychologist average 23 days, and can exceed 6 to 12 months for a psychiatrist in some regions. Many people simply cannot wait. An AI available at 3am has concrete value.

But behind this promise lies a choice that deserves naming: anthropomorphism. Designing an AI as a warm presence rather than as a tool. This choice has direct consequences for what the tool can actually do. And most of the time, no one talks about it.

Why wellbeing AIs are designed to imitate humans

Researchers at Google DeepMind documented this in an article published in April 2024: anthropomorphism in AI raises deep questions around emotional dependency, and relationships with these tools should not be built on artificial attachment. What this article highlights is that the degree of anthropomorphism in an AI results from deliberate design choices, made in an environment where commercial incentives push toward maximising engagement. An AI that comforts generates immediate satisfaction, and therefore retention. An AI that confronts risks being abandoned after the first session.

Tools therefore calibrate toward validation. They learn to say what the user needs to hear. They adopt an empathetic tone, gentle reformulations, a posture that mimics what we expect from a well-meaning interlocutor. The result: a warm mirror. Useful for feeling less alone. Less useful for changing anything.

The teams building them are not making a deliberately bad choice. They are responding to a commercial logic that optimises for engagement over change. The two do not always point in the same direction.

Feeling better and getting better are not the same thing

A simple example. Someone is struggling with a conflict with their manager. Tension has been building for weeks; they ruminate on the way home. They open an app, describe their situation. The AI listens, reflects, validates. The session ends. They feel relieved.

Three days later, the situation is identical. They open the app again.

This pattern is not an isolated failure. It is the logical outcome of anthropomorphism applied to a wellbeing tool: an AI designed to soothe rather than to create movement. The relief was real. The change did not happen. And the tool may have helped delay a necessary action, by offering a regular release valve without ever addressing what sustains the problem.

Repeated use can establish a habit: returning to seek immediate relief rather than attempting something different in reality. A randomised controlled trial conducted jointly by the MIT Media Lab and OpenAI, involving 981 participants who used ChatGPT for four weeks, illustrates this: while participants were on average less lonely by the end of the study, prolonged daily interactions with the chatbot can reinforce negative psychosocial effects. Participants who used the chatbot intensively consistently showed worse outcomes on loneliness, emotional dependency and loss of real social interaction. OpenAI itself acknowledged in a research report that emotional dependency could be heightened by its own tools, identifying anthropomorphisation as one of the main risks.

Anthropomorphism, by making AI endearing, encourages precisely this repeated return. The tool makes you feel good. It does not help you regain agency.

What actually sustains a problem

Brief systemic therapy from the Palo Alto school starts from a counter-intuitive premise: what maintains a problem is not its deep causes. It is the repeated attempts to avoid or control it.

These attempts have their own logic. They seem reasonable. That is precisely why they fail.

Someone checks compulsively to calm down, and feeds their doubt. They avoid a difficult conversation, and the fear grows. They fight an intrusive thought, and it returns stronger. They over-control to avoid making a mistake, and mistakes become even more threatening.

An anthropomorphic AI that comforts without questioning these mechanisms can reinforce them, offering each time a relief that sidesteps what is actually keeping the problem alive.

Assuming its nature as a tool: what this means in practice

This is the choice we made at Hellopalo, and it has not always been easy to hold.

Our therapeutic agent does not seek to be liked. It does not systematically validate. It confronts, names what is stuck, and helps to consider something other than what has been attempted for too long. The conversation has an end, because a session without limits becomes a soothing ritual. It leads to concrete tasks, designed to create movement in reality rather than relief within the session. The conversation itself is deleted at the end of each session. Only a summary is kept, to enable therapeutic continuity from one session to the next, not to create a sense of connection with the tool.

And when a situation goes beyond what the tool can handle, it says so. It facilitates a connection with a human therapist. A core part of the support logic, not a secondary option. A tool that knows how to step aside is a tool that accepts what it is. It does not seek to occupy a space that does not belong to it.

This choice runs counter to certain engagement logics. We believe it is the right one. We do not claim to have resolved everything.

The question we still ask ourselves

How do you measure that a wellbeing AI is truly useful? The usual metrics, time spent, sessions opened, satisfaction scores, measure engagement. Not change. We are working to build different indicators, grounded in how situations actually evolve over time. Harder to measure. Yet the only thing that matters.

A useful AI in mental health is judged by what changes in a person's life after the session, not by what they felt during it. Resisting anthropomorphism means resisting the temptation to build a tool that pleases at the expense of one that helps. Pushed far enough, the two end up working against each other.