What GDPR, the EU AI Act and building AI for a sensitive context taught us about turning responsibility into engineering
It is pretty easy to impress a room during an AI demonstration.
Someone asks the system a difficult question. It answers quickly, fluently and with considerably more confidence than most humans display before their second coffee. Someone tries another question. The answer is useful again. The interface behaves itself, nothing crashes, and people begin doing something entirely reasonable: imagining the finished product.
That is a good moment. Building something that works is not trivial.
It is simply not the same as knowing that it is ready.
We learned that difference very concretely while working around CareLink, where a conversational AI component was being developed for a context involving young people, emotional wellbeing, resilience and potentially sensitive information. The technical challenge was not simply to make a language model produce useful responses. Modern models can demonstrate that ability remarkably quickly, sometimes before the surrounding project has decided exactly what useful is supposed to mean.
The harder work began when we started asking what needed to exist around that capability before an organisation could responsibly put it in front of people.
What information could enter the system, and what happened to it afterwards? What was the AI actually for, and what sat outside its role? What should happen when somebody disclosed something the designers had not expected? Which failures were merely annoying and which were serious enough to stop a release? How would a human enter when necessary? What should be logged, and what should deliberately not be retained? What happened when the underlying model, prompt, knowledge source or language changed? Most importantly, how could we demonstrate that a safeguard described in a policy document actually existed in the system people were going to use?
This is where the language of responsible AI begins to become difficult in a useful way.
Many organisations are very good at principles. Privacy, fairness, transparency, human oversight, safety, accountability and robustness appear in responsible-AI frameworks across sectors. They are important concepts, and Europe’s regulatory architecture gives several of them increasingly concrete legal form.
But a principle cannot stop an unsafe response, delete a record or decide what information a model provider receives. It cannot notice that the Portuguese version behaves differently from the English version. It cannot tell a project team that a failed test is a release blocker rather than something to put in the “lessons learned” section later.
Somebody eventually has to translate it.
That is the argument running through this article: responsible AI begins where principles become system requirements.
And once that translation begins, responsible AI starts looking considerably less like an ethics statement and much more like engineering.
The most visible part of AI is not necessarily the most important
AI creates an illusion of readiness because the part everyone can see is also the part easiest to demonstrate.
We see the answer. We hear the voice. We interact with the interface. We can judge whether the response sounds sensible and whether the conversation feels natural. If ten people ask ordinary questions and receive ten reasonable answers, the system begins to feel tangible in a way that a risk analysis or a data-flow diagram never will.
What we cannot see by chatting with the model is much of what determines whether the system is governable.
We cannot see where the user’s text travelled. We do not know whether it was retained, logged or reused. We cannot tell whether a response came from an approved body of knowledge or from the model’s wider generative capabilities. We do not know what would have happened if the user had phrased the same idea indirectly, switched language midway through the interaction or disclosed information substantially more sensitive than anything the interface explicitly asked for.
And we certainly cannot tell whether anyone has decided what should happen when the model is confidently wrong.
That matters because generative AI can fail invisibly. A conventional application often looks broken when something goes wrong. A language model may instead produce an answer that is beautifully structured, warm in tone, grammatically impeccable and entirely unsuitable.
Fluency is an interface property. It is not evidence of safety.
We had already encountered this distinction in our earlier work on CareLink. At one stage of internal testing, roughly six interactions in ten were performing satisfactorily. The figure was never a proposed legal threshold, and there is no magical percentage at which an AI system becomes responsible. What mattered was what could be hiding inside the remaining four interactions: fabricated information, inappropriate reassurance, missed crisis language, unsafe handling of emotionally sensitive material or inconsistent behaviour between languages.
An average begins to look much less comforting when the failure you care about is uncommon but consequential.
This changes the central question. Instead of asking only “Does the AI work?”, responsible development has to ask “Under which conditions are we prepared to let it work?”
That is a much harder question because it requires boundaries, evidence and judgement rather than applause after a successful demonstration.
GDPR makes you start somewhere AI conversations often wouldn’t: with the data
AI conversations naturally gravitate toward the model. Which model are we using? How capable is it? How large is the context window? Can we improve the prompting? Should we use retrieval? Can we fine-tune it?
GDPR forces us to begin somewhere rather less shiny.
What information are we processing, and why?
Its core principles include purpose limitation, data minimisation, storage limitation, accuracy, integrity and confidentiality, alongside the wider accountability obligation. Data protection by design and by default requires organisations to consider appropriate safeguards when the processing is designed, rather than constructing the system first and introducing privacy at the ceremonial ribbon-cutting stage.
In practice, this changes the engineering conversation.
If somebody enters personal information into a conversational system, where does the text go? Does the application store the conversation? Does an external model provider receive it? What metadata travels with it? Who within the organisation can see the raw conversation? What ends up in diagnostic logs? How long are those records retained? Can they be deleted? If several organisations are involved, who decides why and how the data is processed?
For a system operating in a sensitive context, another complication appears immediately: users do not behave like database fields.
A carefully designed form may contain no box marked health data. A person typing freely about their circumstances may nevertheless disclose exactly that, and health data is a special category of personal data under GDPR.
Responsible design therefore has to consider not only the information a system intentionally requests, but the information it can reasonably receive.
That distinction affects logging, retention, administrative access, external processing, incident handling and the user interface itself. It can also affect whether a Data Protection Impact Assessment is required. Under GDPR, a DPIA is required where processing is likely to result in a high risk to people’s rights and freedoms; the Commission identifies systematic and extensive automated evaluation and large-scale processing of sensitive data among the cases where a DPIA is required. It also describes the DPIA as something conducted before processing and treated as a living tool, rather than a one-off document completed and then respectfully archived.
This is where one of the simplest lessons from the work became one of the most useful:
A privacy policy is not a data architecture.
The policy can say that the organisation minimises data. The architecture determines whether it actually does. The policy can state information is retained only as long as necessary. Somebody still has to decide what necessary means, configure retention and make deletion operational.
The words matter. The system has to make them true.
The EU AI Act asks a different question: what can the system do, and how can it fail?
GDPR provides one important lens because personal-data processing sits underneath many AI systems. The EU AI Act adds another. It asks us to look not simply at information but at the behaviour, purpose and governance of the AI system itself.
There is an important legal distinction here. We are not using CareLink to claim retrospectively that the system belongs to a particular AI Act high-risk category. Classification depends on intended purpose and the specific criteria in the Regulation, and responsible compliance begins by resisting the temptation to attach dramatic regulatory labels before doing the classification work.
But the high-risk framework gives us an unusually useful engineering vocabulary.
For systems that fall within it, the Act addresses continuous risk management, technical documentation, logging, transparency for deployers, effective human oversight, accuracy, robustness and cybersecurity. Risk management is conceived as an iterative lifecycle process, not a meeting held shortly before launch to confirm that everyone remains generally in favour of safety.
The current timetable also matters. The AI Act became generally applicable on 2 August 2026, while the 2026 AI Omnibus moved the main high-risk requirements for Annex III systems to 2 December 2027 and those for high-risk AI embedded in regulated products under Annex I to 2 August 2028. Some other obligations are already applicable; for example, Article 50 requires people interacting directly with an AI system to be informed that they are interacting with AI unless this is obvious from the circumstances.
The interesting point for us is not that every project should prematurely pretend to be subject to every high-risk requirement.
It is that regulation is making the distance between principle and implementation increasingly difficult to ignore.
If we say an AI system is transparent, what information does the user or deployer actually receive? If we say a human oversees it, what can that human actually see and do? If we say the system is robust, against which scenarios was it tested? If we say risk is managed, where is the evidence connecting a known failure mode to a technical control?
Regulation stops looking abstract very quickly once somebody has to build the evidence.
An AI system does not have one boundary. It has several.
One of the most useful things CareLink taught us was to stop thinking of an AI boundary as a single line.
There is a purpose boundary, separating what the system is meant to do from tasks that sit outside its role. There is a data boundary, determining what information is necessary, what may be received unexpectedly and what should be retained. There is a knowledge boundary, distinguishing what the system has a reliable basis for saying from what it can merely generate plausibly.
There is also a behaviour boundary, where certain forms of response become inappropriate or unsafe. There is a human boundary, where automation should stop being sufficient and judgement needs to enter. Finally, there is a release boundary, where an organisation decides that the available evidence is strong enough for a defined form of real-world use.
The awkward thing about generative AI is that the model can cross several of these boundaries without appearing technically broken.
It may be perfectly capable of answering a question it should not answer.
That is precisely why capability and authorisation must not be confused.
The fact that a language model can generate something does not define the intended purpose of the system built around it. The organisation does.
Human oversight is where responsible-AI language is tested rather quickly
Consider one of the most reassuring phrases in the AI-governance vocabulary: human in the loop.
There is a human. There is apparently a loop. Everyone can go home.
Except the phrase tells us almost nothing until we examine the architecture underneath it.
When does the human enter? What causes the system to recognise that intervention is needed? What information reaches that person? Do they understand the model’s limitations? Can they challenge the output? Can they override it? Can they stop the process? Are they actually authorised to do so, or have we created the organisational equivalent of an emergency brake connected to nothing?
For high-risk systems, the AI Act deliberately makes human oversight more operational. Article 14 requires systems to be designed so that natural persons can effectively oversee them, and Article 26 requires deployers to assign oversight to people with the necessary competence, training, authority and support.
That combination is important.
A human with no authority is not meaningful oversight. A person with authority who receives no useful information is not meaningful oversight either. And a person receiving every possible alert without prioritisation may simply become an extremely expensive notification service.
Once we started thinking this way, human oversight stopped being an ethical principle and became a set of engineering and organisational requirements.
That is exactly the transition responsible AI needs.
The work becomes invisible at precisely the point it becomes important
The more seriously we approached these questions, the less the work resembled the chatbot itself.
We needed to define intended purpose and exclusions. We needed to map information flows. We needed to distinguish generic project risk from AI-specific failure modes. We needed to think about what should happen when a boundary was crossed, whether that behaviour worked consistently across languages, how correction would be verified and what evidence would be sufficient to support a release decision.
Most of this wouldn’t result in exciting screenshots.
We are yet to gather around a requirements traceability matrix at a technology conference and whisper, My God, look at the controls.
But this is where a prototype begins becoming governable.
And it brings us to the question that sits underneath the professional part of this article:
Can we follow a principle all the way into the technology, and can we follow the evidence back?
If the answer is yes, responsibility has begun becoming operational.
If the answer is no, we may still have an excellent ethics page.
Want the full version — and the tools behind it?
The public article ends here, but the professional section goes further: from principles into the actual assurance architecture behind a responsible AI system.
In the full version, we unpack how to connect regulatory scope, data flows, risk, controls, testing, human oversight, release readiness, monitoring and change into one defensible evidence chain.
Paid subscribers also receive the REDefine Responsible AI Assurance Architecture editable workbook and its Companion Guide, designed to help apply the method to a real AI system.
Subscribe to the REDefine newsletter to unlock the full article and download the professional resources.
