AI safeguarding in schools is not just about choosing safe technology or training teachers. It is about what happens when an unexpected interaction requires somebody to act—and whether that person has the authority to do so.

A teacher notices that students have begun sharing an AI-generated response about one of their classmates. What started as an ordinary learning activity has become a dispute involving private information, accusations and a distressed child.

The teacher can stop the activity. They can address the students, offer appropriate support and report the incident. They can do what experienced educators have always done when a lesson takes an unexpected turn.

But what about the AI system that contributed to the situation?

Can the teacher prevent it from producing a similar response for another student? Can the school restrict the relevant function while the incident is examined? Does anyone know whether the system has recently been updated, or whether its behaviour has changed since it was approved for educational use?

These questions may seem technical until somebody needs an answer.

And that is when the reassuring language of institutional safeguarding meets a rather less reassuring reality: the people closest to the children are not always the people who control the technology.

The problem with being prepared on paper

Most schools do not approach safeguarding casually. They have procedures for bullying, requirements concerning privacy, rules governing technology and designated people responsible for protecting children.

Teachers receive training. Parents are informed about expectations. Students are reminded to behave responsibly, online as well as offline.

These arrangements matter. The difficulty is that generative AI can create situations that cross several of them at once.

Consider what happens when a chatbot provides an answer that appears reasonable in isolation but acquires a completely different significance when shared among classmates.

The student may have asked a question outside the intended learning activity. The AI may have responded without understanding the human circumstances involved. Other children may interpret the answer as an authoritative judgement. And the resulting situation may involve both harmful conduct and questions about the technology itself.

The school must then bring together educational judgement, child protection, privacy responsibilities and technical oversight.

Having a separate policy for each area does not guarantee that those responsibilities will connect when they are needed.

This was the central difficulty exposed by a real, anonymised school incident examined in our LinkedIn article, A Safety Policy Is Not a Safety Procedure. The institution had established safeguarding commitments, but the response revealed how difficult it can be to make those commitments work together when an AI interaction produces consequences nobody anticipated.

The case raises a question that deserves attention well beyond the school involved. If our procedures work only when problems remain neatly within the categories we created for them, how prepared are we for technology that moves between those categories in seconds?

Human oversight is not the same as having a human in the room

There is a phrase that appears frequently in discussions of responsible AI: human oversight.

It sounds sensible, especially in education. We want teachers to remain responsible for learning, children to retain access to human support and institutions to maintain control over important decisions.

But imagine twenty-six students using an AI assistant while one teacher supervises the class.

The teacher may be attentive, competent and experienced. They still cannot read and evaluate twenty-six different conversations in real time while doing everything else teaching requires.

Nor would monitoring every private interaction necessarily be an appropriate solution. Children have privacy interests, and schools must consider those alongside their safeguarding responsibilities.

So the real question is not whether a human being is present. It is what that human being can realistically observe, decide and change.

A teacher might have permission to end an activity but no access to the platform’s technical controls. A safeguarding professional may be responsible for assessing a child’s welfare without knowing how the AI system behaves. The school’s technology coordinator may manage accounts and devices while having little influence over an external provider’s model.

All three can perform their jobs conscientiously while an important part of the response remains unresolved.

This is the authority gap that schools need to examine before introducing AI.

Human oversight becomes meaningful only when responsibility is accompanied by the information, authority and support necessary to act.

Otherwise, we have simply given an educator another responsibility while leaving the decisions that shape the risk elsewhere.

What should an AI safeguarding procedure actually do?

An AI safeguarding policy establishes the principles and responsibilities governing the technology’s use. A functioning procedure explains how those responsibilities become practical decisions when something unexpected happens.

The distinction becomes clear when we follow a concern through the school.

Suppose a teacher identifies an AI interaction that may have caused harm. They need to know what immediate action is appropriate, how to support the children involved and how to report the concern without unnecessarily disclosing private information.

But the response cannot necessarily end with that report.

Somebody must determine whether the incident raises questions about the technology’s continued use. That person needs access to relevant information and the authority to involve technical specialists, restrict the system if necessary and ensure that findings lead to appropriate changes.

If the problem involves both student behaviour and system behaviour, the school must be able to examine both without confusing them.

Not every inappropriate use of AI is evidence that the platform itself is unsafe. Children can misuse perfectly ordinary information, just as adults can. Conversely, addressing a student’s conduct does not automatically settle whether the technology has exposed a foreseeable weakness.

An effective procedure preserves that distinction.

It allows the school to protect the child, address the circumstances of the incident and assess the technology without expecting one professional to perform all three jobs.

Most importantly, the procedure should remain workable when information is incomplete. A teacher cannot always establish exactly what happened before taking appropriate protective action. The institution should not expect them to.

The exercise every school should try before its next AI rollout

Before introducing an AI assistant, school leaders could conduct a short rehearsal with the people who would actually be responsible if something went wrong.

Choose a realistic situation in which a student receives an unexpected AI response and uses it in a way that affects another child.

Ask the teacher what they would do first. Then follow the concern to the safeguarding contact, school leadership and the person responsible for the technology.

At each stage, examine whether the next person has the information and authority necessary to carry out their responsibility.

The most revealing moment may be when somebody says, “I assume someone else would handle that.”

That assumption is worth investigating.

Now complicate the scenario. The safeguarding contact is unavailable. The message has been shared outside the classroom. The provider has recently updated the model, and the school does not immediately know whether its behaviour has changed.

Can staff still take appropriate first steps? Is there somebody authorised to restrict the technology? Can the incident be investigated without exposing the children to further unnecessary disclosure?

The purpose is not to produce an enormous manual covering every imaginable outcome. Schools already have enough paperwork, and a procedure nobody can navigate under pressure is not particularly reassuring.

The purpose is to discover whether the institution’s responsibilities connect under realistic conditions.

Sometimes this exercise will reveal a need for further training. Sometimes it will expose uncertainty about decision-making authority, weaknesses in communication or insufficient control over the technology.

And sometimes it will show that a particular AI function is not yet appropriate for the learners concerned.

A responsible decision to limit technology is not evidence that a school is falling behind. It may demonstrate a better understanding of the difference between what a system can do and what the institution can safely support.

The cost of AI adoption is not just the subscription

Schools are understandably interested in what AI might make easier. A system that helps prepare activities, explains difficult concepts or supports individual learners can offer genuine educational value.

But time saved in one part of teaching does not automatically mean that the total workload has decreased.

AI systems also require supervision, evaluation, staff preparation, incident response and periodic reassessment. Providers may update their products. Students may discover uses that were not anticipated during implementation. New questions may arise about accessibility, privacy or the appropriate boundaries of educational assistance.

That work must be organised, and somebody needs time to do it.

If school leadership introduces a technology without accounting for the additional responsibilities, those responsibilities do not disappear. They are often absorbed by the teachers and other staff members already managing the immediate demands of education.

We should be careful about celebrating efficiency while overlooking the human labour required to make that efficiency responsible.

A school that cannot allocate the necessary oversight and support may need to narrow its planned use of AI. That is an operational judgement, not a rejection of innovation.

The real question is whether protection is available when somebody needs it

Children will continue to test boundaries, misunderstand one another and ask questions that were never part of the lesson plan. Generative AI will continue to produce answers that educators have not reviewed in advance.

We should not pretend that either can be made entirely predictable.

But educational institutions have choices about the technologies they introduce, the conditions under which students encounter them and the authority they give staff to respond when something goes wrong.

Those choices matter most when a situation becomes difficult.

The true measure of an AI safeguarding procedure is not how convincingly it describes the school’s values. It is whether a teacher can obtain support, whether an affected child can rely on appropriate protection and whether somebody has the power to address the technology when the circumstances require it.

Our article, A Safety Policy Is Not a Safety Procedure, explores a real school incident in which those expectations were tested, examining the psychology of AI-generated responses, children’s social relationships and the institutional gaps that became visible when the situation escalated.

The case offers an important lesson for any school preparing to expand its use of AI: before asking teachers to supervise another system, make sure the institution has given them something more useful than another promise.