A recommendation system can correctly record that someone watched six videos about anxiety and still misunderstand why they watched them.

A school database can correctly record that a student missed several classes while telling us almost nothing about whether the student has disengaged from education.

A chatbot can receive an accurate account of a disagreement while knowing only what one participant chose to tell it.

None of these systems necessarily contains false information.

The problem appears in the next step: what the system infers from the information it has.

This distinction matters because more and more digital systems do not simply store what people do. They classify, predict, recommend, prioritise and respond. The difference between a recorded signal and the meaning assigned to that signal can therefore affect what somebody sees, which opportunities become available, how an institution responds or even how a person begins to understand their own situation.

We call this distance between the observable information and the conclusion drawn from it the inference gap.

Understanding that gap gives organisations, educators, families and technology teams a more useful way to think about responsible AI than asking only whether the data is accurate or the software is functioning correctly.

What is the inference gap?

The inference gap is the distance between what a system can actually observe and what it concludes that observation means.

Consider a recommendation platform.

It may know that a user watched a video for forty seconds, replayed part of it and then opened another video on the same subject.

Those are observations.

From those observations, the system may infer that the person is interested in the subject and should receive more of it.

That is an interpretation.

The distinction becomes important because the same behaviour can have several meanings. Someone may watch content about depression because they are struggling themselves, because they are worried about a friend, because they are studying the subject or because the video disturbed them enough that they watched twice.

The platform sees behaviour. It does not directly see intention.

Digital systems need to make these kinds of simplifications. No recommendation engine, chatbot, school platform or administrative database can know everything about the person using it.

Responsible design therefore cannot depend on eliminating inference.

It has to address what happens when an inference is incomplete or wrong.

Why accurate data can still produce a bad decision

Organisations often treat data quality and decision quality as though they were the same problem.

They are not.

A system can contain perfectly accurate records and still support an inappropriate conclusion.

Imagine a student who has repeatedly missed university classes. An attendance system can record every absence correctly. If the institution assumes those absences indicate lack of interest, however, it may act on something the data never established.

The missing information may be that the student is caring for an ill parent, dealing with unstable housing, working additional hours or facing another circumstance the attendance record cannot represent.

The original data does not suddenly become false.

Its meaning changes when context arrives.

AI makes this problem particularly important because automated systems can move very quickly from observation to interpretation to action.

A classification can determine which information is shown next. A recommendation can create the behaviour from which another recommendation is generated. A chatbot can respond to one person’s description of an event and influence how that person interprets what happened.

This creates a feedback loop.

The system is no longer merely observing the world. Its interpretation begins helping to shape the next part of it.

Four questions for evaluating an automated inference

A useful assessment should therefore look beyond the accuracy of the original data.

When a digital system is interpreting people, four questions deserve attention.

1. What does the system actually know?

Separate observable information from interpretation.

A platform may know that somebody clicked, watched, searched or stopped using a service. It does not automatically know the motivation behind the behaviour.

A chatbot may know what a user has written. It does not automatically know whether the account is complete, whether another person would describe the situation differently or what relevant information has been omitted.

An institutional system may know that a form is incomplete. That does not necessarily tell us why the person could not complete it.

This sounds obvious when stated explicitly. In practice, the boundary can disappear surprisingly quickly once an inference is displayed as a score, recommendation, category or polished natural-language response.

2. What has the system inferred?

The next step is to identify what meaning has been attached to the signal.

Did repeated viewing become “interest”?

Did absence become “disengagement”?

Did a pattern in historical data become a prediction about somebody’s future behaviour?

Did one person’s account become an apparent assessment of another?

The goal is not to prohibit interpretation. Most useful digital systems rely on it.

The goal is to make the interpretation visible enough that people can distinguish what is known from what has been concluded.

3. What happens because of that inference?

This is where a seemingly minor interpretation becomes a governance issue.

If the inference changes nothing consequential, an error may be relatively easy to tolerate.

If it determines what a child repeatedly encounters online, influences access to a service, affects an institutional response or becomes part of a sensitive personal decision, the standard should be much higher.

The question is therefore not simply whether the model is correct most of the time.

It is also:

What power does the inference have?

The greater the consequence, the more important it becomes to examine uncertainty, context and routes for human judgement.

4. Can the person correct the system?

This may be the most important test.

People will be misunderstood by automated systems. No amount of training data will remove that possibility entirely.

Responsible technology therefore needs mechanisms through which a person can say:

That is not what this means.

For a recommendation system, that might mean an explicit preference such as Not interested genuinely changes future recommendations rather than being outweighed by previous viewing behaviour.

For an institutional system, it may mean staff can recognise circumstances that do not fit predefined categories and have an authorised route for correcting the process.

For an AI-supported decision, it may mean a human can review the underlying evidence rather than treating the model’s conclusion as the evidence itself.

For a conversational AI, it may mean the system acknowledges that it does not possess enough information to judge another person or a complex situation reliably.

The objective is not endless human override.

It is contestability: the ability to challenge an interpretation when the system’s representation and lived reality diverge.

Why better personalisation does not solve the whole problem

A common response to inaccurate inference is to collect more information.

Sometimes that helps.

More context may improve a recommendation. Better data may reduce classification errors. A more capable model may recognise ambiguity that a simpler system missed.

But there is a limit to this solution.

The idea that technological systems should understand us better can easily become an argument for collecting increasingly detailed information about people.

That creates another problem.

How much should a platform need to know about a teenager before it can recommend videos responsibly? How much intimate context should a chatbot require before deciding whether its reassurance is appropriate? How much information should an institution collect merely to avoid treating every unusual circumstance as an error?

Responsible technology should not depend on perfect surveillance.

Sometimes the better solution is not to know more about the person.

It is to make the system less certain about what it thinks it knows.

The difference between prediction and understanding

AI systems are increasingly capable of making useful predictions from very limited information.

That capability is valuable.

But prediction and understanding are not interchangeable.

A system may become very good at predicting which video somebody will watch next without knowing why the person is watching.

It may generate an emotionally appropriate response without possessing the responsibilities we normally associate with care.

It may identify a pattern that correlates with a particular outcome while remaining unable to explain an individual person’s circumstances.

This is why responsible AI needs something more than accuracy metrics.

We also need to ask whether the system communicates uncertainty appropriately, whether its conclusions can be challenged and whether the organisation deploying it understands the difference between a useful prediction and a sufficient basis for judgement.

That distinction becomes especially important when technology enters education, health, public services, employment or emotionally sensitive situations.

A practical test for organisations

When reviewing an AI system, automated workflow or digital service, take one consequential decision and trace it backwards.

Start with what happens to the person.

Then ask:

What decision or recommendation produced that outcome?

What inference produced the decision?

What signal produced the inference?

What relevant information could the signal not contain?

How would somebody demonstrate that the interpretation was wrong?

Who has the authority to correct it?

This exercise often reveals something ordinary technology assessments miss.

The weakness may not be inaccurate data or defective software.

It may be an assumption sitting quietly between the two.

That assumption might concern what engagement means, what a normal user looks like, what counts as risk, which circumstances deserve an exception or which behaviour should be interpreted as preference.

Once encoded into a functioning system, such assumptions can become surprisingly difficult to see.

They begin to look like properties of reality rather than decisions somebody once made.

Helping children understand the difference

The same distinction can be taught without asking children to understand machine-learning mathematics.

A useful starting question is simply:

What did the system see, and what did it decide that meant?

That question turns a recommendation feed into something children can investigate rather than merely consume.

REDefine created The Feed Detective for exactly this purpose. The free family learning resource uses animated investigations, activities, a printable field guide and a family digital agreement to help children and adults explore how recommendation systems interpret behaviour and what happens when those interpretations are wrong.

We have also developed AI at Home: Five Conversations Worth Having With Your Child, which helps families discuss AI answers, privacy, independent thinking, boundaries and the difference between a convincing response and reliable understanding.

Both resources are designed to build judgement without turning family digital life into surveillance.

The right to remain more complicated than the model

Digital systems will always simplify reality.

Forms need fields. Databases need categories. Recommendation systems need signals. AI models need inputs.

The problem begins when the simplified representation becomes more authoritative than the person it represents.

That is why the future of responsible technology cannot depend solely on systems becoming increasingly good at interpreting us.

It must also preserve our ability to remain more complicated than their interpretation.

A teenager should be able to stop a recommendation pattern without proving why they watched the original video.

A student should be able to explain circumstances that an attendance record cannot contain.

A person affected by an automated decision should have some meaningful route to challenge the conclusion rather than being told that the system has already considered the available information.

Because accurate data and accurate understanding are not the same thing.

And when the two diverge, the most important question may not be whether the system can eventually learn more about us.

It may be whether we still retain enough agency to tell it:

That is not what this means.


This article develops one practical idea from our latest REDefine weekly essay, The Signal Is Not the Person, which connects AI safety in schools, TikTok-style recommendation systems, emotional AI, institutional classification and the wider question of what happens when systems begin acting on incomplete interpretations of people.

The full essay also includes two Writer Spotlights and looks ahead to two questions we will examine next: what happens when digital systems cannot represent someone’s reality, and whether people should have a right to remain outside somebody else’s technology.