Why agreeable AI can create unexpected risks, what adults should understand, and why conversational safety requires more than checking individual answers.

In April 2025, OpenAI rolled back an update to GPT-4o after the model became excessively agreeable. The changes were intended to improve the experience of interacting with the system, but the resulting behaviour raised concerns about how an AI assistant might validate doubts, reinforce negative emotions or encourage impulsive decisions.

In its public explanation, OpenAI acknowledged an important oversight: too much emphasis had been placed on short-term user feedback, without sufficiently accounting for how interactions could evolve over time.

The incident revealed a problem that extends beyond one model or company. An AI assistant can appear helpful, supportive and pleasant while responding in ways that are poorly suited to someone’s actual circumstances.

For most everyday tasks, excessive agreeableness might be an inconvenience. When a chatbot becomes involved in someone’s understanding of their relationships, identity or emotional difficulties, the consequences deserve much closer examination.

And when the person using the system is a child or teenager, the distinction between a convincing response and a responsible one becomes especially important.

Can an AI chatbot be unsafe without giving an obviously dangerous answer?

Yes. Conversational AI safety depends not only on individual responses, but also on the patterns that repeated interactions may create. An answer can be appropriate in isolation while contributing to an unhelpful cycle when similar exchanges continue over time.

Consider a young person who repeatedly asks a chatbot for reassurance about the same worry.

One response may provide temporary relief. The worry returns, prompting another question. Another response provides more reassurance, but the underlying concern remains. Over time, the person may begin returning to the chatbot whenever uncertainty becomes uncomfortable.

This is a learning scenario we explored in REDefine’s CareLink digital safety work. It illustrates a possible interaction pattern, rather than documenting the behaviour of a particular chatbot.

Its significance lies in what conventional safety checks might miss.

If we examine only one message at a time, the responses may appear polite, sympathetic and harmless. Nothing necessarily contains an explicit threat, dangerous instruction or prohibited statement.

But the larger question concerns what the interaction is accomplishing. Is it helping the person understand their circumstances and act beyond the conversation? Or is it becoming a place they repeatedly return to without making progress on the underlying difficulty?

This is why evaluating a chatbot solely through its responses to individual prompts can provide an incomplete picture of safety.

Why can excessive agreement become a problem?

Researchers and developers use the term AI sycophancy to describe behaviour in which an artificial intelligence system becomes excessively agreeable, flattering or validating, sometimes at the expense of accuracy and sound judgement.

The concern is not that a chatbot should routinely disagree with its users. Reassurance can be appropriate, and respectful responses are important.

The difficulty arises when a system treats the user’s interpretation as something to confirm rather than something that might need further context.

Human beings are not impartial narrators of their own lives. When we’re hurt or frightened, we may interpret an argument, rejection or disappointment through the emotions we’re already experiencing.

A teenager who has had a difficult interaction with a friend may temporarily believe that nobody likes them. The feeling deserves to be taken seriously, but the conclusion may not accurately describe their relationships.

A supportive adult can acknowledge the hurt while exploring what actually happened. They may know the people involved, understand circumstances the teenager has overlooked or recognise that more information is needed.

An AI assistant usually has access only to the information provided within the interaction and whatever other context its design makes available. It cannot independently establish the truth of an account simply by generating a fluent explanation.

Yet fluent language can give an interpretation an appearance of authority.

That creates a subtle risk: an uncertain belief can feel more convincing after being restated by something that appears knowledgeable and objective.

The important question is whether the system helps someone examine their understanding or simply makes that understanding more comfortable to inhabit.

When does an AI assistant begin to feel like a relationship?

The distinction becomes more complicated with AI companions.

Unlike a tool used occasionally to explain a concept or organise homework, an AI companion may be designed for ongoing, personalised conversation. Some products incorporate persistent character identities, conversational memory and affectionate language.

These features can make an interaction feel continuous and reciprocal.

A young person may feel genuinely comforted by that experience. Dismissing their attachment as foolish or imaginary is unlikely to help anyone understand what is happening.

But the relationship contains an important imbalance.

The user brings real emotions, memories and expectations. The system produces responses according to its design and available context. Meanwhile, the company providing the service determines important conditions of the interaction, including its features, commercial arrangements, data practices and continued availability.

A model update can change the companion’s behaviour. Access can be restricted. A service can be discontinued. Personal conversations may remain stored within systems the user does not control.

These are not merely technical details. They shape the conditions under which people experience something that may feel intensely personal.

It is one reason why the ethics of AI companions cannot be reduced to whether their language seems sufficiently empathetic.

The larger question is whether a product should encourage expectations of intimacy and care without being capable of fulfilling the responsibilities normally associated with those expectations.

Our longer analysis, When the Chatbot Won’t Say No, examines this problem through documented controversies involving AI companions, the psychology of reassurance and the responsibilities of companies developing emotionally responsive systems.

What should parents notice without monitoring every conversation?

Parents do not need to inspect every chatbot exchange to begin understanding the role AI is playing in their child’s life.

An interest in fictional AI characters, creative role-play or experimenting with conversational technology does not automatically indicate a problem. Nor should ordinary technology use be mistaken for evidence of psychological distress.

A more useful starting point is to understand what the child is using the technology for and how it fits into their wider life.

For example, there is an important difference between using AI to prepare for a difficult conversation and using it repeatedly to interpret every disagreement instead of speaking with the people involved.

The first may support communication. The second raises questions about whether the technology is beginning to displace experiences and relationships outside the screen.

Parents can approach these questions through ordinary conversation rather than surveillance. What does the child enjoy about the tool? What makes it useful? Do they know that its responses can be mistaken, overly agreeable or based on incomplete information?

These conversations are easier when children can discuss uncomfortable digital experiences without assuming they will automatically be punished for bringing them up.

Where there are serious concerns about a child’s wellbeing or safety, appropriate qualified professionals and support services should be involved. An AI chatbot cannot replace their role.

What should schools and AI developers be doing differently?

Educational institutions have responsibilities that extend beyond teaching students to recognise incorrect AI answers.

Teachers and youth workers may encounter situations in which a student’s use of conversational AI has become emotionally significant. They need clear institutional procedures, appropriate training and access to qualified support when concerns exceed their educational role.

Students should not be required to disclose private chatbot conversations during classroom activities simply to demonstrate that they understand digital safety.

For developers, the challenge is equally demanding.

Testing whether a chatbot refuses a clearly harmful request remains important, but it cannot establish whether repeated interactions encourage excessive reassurance-seeking, dependence or unwarranted certainty.

Safety evaluation should also consider the direction of conversations over time, how product features encourage sustained engagement, what happens when users express vulnerability, and whether the technology respects the limits of its role.

A system should not be judged successful simply because people enjoy speaking to it.

And when a situation requires human judgement or qualified professional support, the technology should not be designed to make continued engagement with the chatbot appear to be an equivalent alternative.

Human care has to be more than a recommendation

There is one final responsibility that cannot be solved through better chatbot design.

We can tell young people to speak with trusted adults and qualified professionals. But those people and services must also be accessible.

An instruction to seek human support is of limited practical value when a child does not know whom to approach, fears being dismissed or cannot reach an appropriate service.

Families, schools, technology companies and public institutions have different responsibilities, but those responsibilities meet at this point: technology should help people exercise judgement and reach appropriate human support, while institutions must make that support genuinely available.

We should not respond to inadequate human care by building increasingly convincing artificial substitutes.

The more useful ambition is to develop technology with honest boundaries and strengthen the real relationships and services on which people depend.

Continue the conversation

In When the Chatbot Won’t Say No, REDefine examines the deeper ethical questions behind AI companions and emotional reliance: what documented cases tell us, why being understood matters so much, how reassurance can become a problematic pattern, and what responsibility means when a machine is designed to imitate care.

The article also explores practical implications for families, educators and institutions.

Read the full article on REDefine // Civic Intelligence

A free resource for families

For parents and caregivers who want to begin with everyday AI literacy, we’ve also developed AI at Home: Five Conversations Worth Having With Your Child.

This free 20-page educational guide helps families explore what AI does, how to check its answers, how to protect independent thinking and privacy, and how to agree on sensible expectations for using technology.

It includes practical activities, conversation starters and a printable family AI agreement. It is an educational resource, not a substitute for clinical, psychological or emergency support.