Who is accountable when an AI system gets it wrong?

An AI system produces an output. Someone acts on it. The output turns out to be wrong. Who carries that? The question is not hypothetical. It is landing on desks across UK workplaces right now, and most organisations have not answered it clearly enough to protect the people working inside them.

This article is about accountability in practice, not ethics in the abstract. It is about where oversight sits in a workflow, what human in the loop actually requires of you, and what you should check before an AI-assisted output leaves your desk. If you are using AI tools at work, or managing people who do, these questions are already your responsibility whether or not anyone has told you so.

The gap between using AI and being accountable for it

Most professionals who use AI tools at work fall into one of two positions. Either they assume the tool is reliable enough that they do not need to check its outputs rigorously, or they assume that because the tool produced the output, the tool is somehow responsible for it. Both positions are wrong, and in a regulated or professional context, both carry real risk.

UK law does not recognise an AI system as a legal person. It cannot be held liable. Liability flows to the human beings and organisations in the chain: the person who deployed the system, the person who reviewed the output, the person who made the decision based on it, and the organisation that sanctioned the workflow. When something goes wrong, investigators and regulators look for the point where a human being could have caught the error and did not.

That point is almost always closer to the end user than anyone expected.

Where oversight actually sits in a workflow

The phrase human in the loop has been used so often it has started to lose meaning. In some organisations it means a person clicked approve before the output was sent. That is not oversight. That is a rubber stamp with extra steps.

Genuine human oversight in an AI workflow requires three things.

• The person reviewing the output must understand enough about how the system works to know what kinds of errors it is likely to make.
• They must have enough domain knowledge to spot when an output is wrong even if it looks plausible.
• They must have the authority, the time and the organisational culture to reject or escalate an output without pressure to accept it and move on.

If any of those three conditions is missing, the loop exists on paper but not in practice. The person clicking approve becomes, in effect, a courier for the AI system's output rather than a reviewer of it. And if that output causes harm, the courier is still accountable, because their name is on the sign-off.

This is one of the issues that higher education is beginning to grapple with seriously, and it is worth reading how universities are recontextualising AI literacy for their own learners in AI in UK Universities: How Higher Education Is Adapting and What It Means for Students, because the same shift is happening in workplace learning and for the same reasons.

What the Certificate teaches that the Award does not

The NCFE Level 3 Certificate in the Application of Artificial Intelligence contains a unit that the shorter Award does not: Unit 05, Ensuring Responsible and Ethical AI Practices. This is the unit where accountability stops being a concept and becomes a checklist of practical decisions.

Unit 05 works through questions that every AI user in a professional setting needs to be able to answer before an output goes anywhere.

• Who authorised the use of this AI system for this task, and on what basis?
• What data was the system trained on, and does any of it create bias risk for this particular output?
• What is the consequence if this output is wrong, and is the review process proportionate to that consequence?
• Has the output been checked against a source that is independent of the AI system itself?
• Is there a record of who reviewed the output, when, and what they checked?

These are not philosophical questions. They are audit questions. They are the questions a line manager, a regulator or a legal team will ask after something goes wrong. Unit 05 gives learners the framework to build answers into their normal working practice, not to reach for after an incident.

If you want to see the full unit-by-unit breakdown of what the Certificate covers and who it is designed for, the NCFE Level 3 Certificate in Cyber Security Practices guide shows how a similarly structured NCFE qualification maps its units to real workplace demands, and the Certificate in AI follows the same design logic. The Certificate qualification page itself sets out the unit structure and assessment approach for learners ready to commit.

Three types of AI error and who carries each one

Not all AI errors are the same, and the accountability question has a different answer depending on which type of error occurred. It is useful to distinguish between them.

Hallucination

A large language model generates a fact that does not exist: a case reference, a statistic, a product specification. The output reads confidently. A professional passes it on without checking. The recipient acts on it. This error sits squarely with the person who passed on the output. The model's tendency to hallucinate is documented, widely known and entirely foreseeable. Foreseeable errors are the hardest to defend against in any professional or regulatory context.

Bias in the output

An AI system produces a recommendation that systematically disadvantages a particular group, because the training data reflected historical patterns of disadvantage. The person using the system did not create the bias, but if they did not check for it and the output affected real people, the question of accountability becomes complicated quickly. Under the Equality Act 2010, an employer cannot outsource responsibility for discriminatory outcomes to a software tool.

Correct output, wrong context

This is the subtlest failure. The AI system produces an output that is accurate for the general case but wrong for the specific situation in front of you. A legal summary that applies in England but not in Scotland. A dosage guideline for an adult population applied to a paediatric case. A financial calculation that assumes a tax regime that changed last April. The model did exactly what it was designed to do. The professional applied it to a context the model could not assess. The professional carries this one entirely.

The most dangerous AI output is not the one that looks obviously wrong. It is the one that looks exactly right, in a context where it is subtly not. Building the habit of asking what context this system cannot see is the single most transferable skill an AI-literate professional can develop.

What to check before an output leaves your desk

The following is not a comprehensive audit framework. It is a minimum standard for a professional working with AI-generated content in a UK workplace context, drawing on the principles covered in Unit 05 of the Certificate.

Check	What you are actually asking	Why it matters
Source independence	Can I verify this against something that did not come from the same system or the same training data?	Circular verification catches nothing. A model trained on an error will reproduce it confidently.
Contextual fit	Does this output apply to the specific organisation, jurisdiction, role or individual it is being used for?	General-purpose models produce general-purpose answers. Professional contexts are rarely general.
Consequence proportionality	Is the level of checking I am doing proportionate to the harm if this is wrong?	A low-stakes internal summary needs less scrutiny than a customer-facing recommendation or a regulatory submission.
Bias exposure	Could this output reflect patterns in historical data that disadvantage a group?	Equality Act 2010 liability does not transfer to the AI vendor.
Record of review	Is there a record that a named person reviewed this output and what they checked?	Without a record, the review did not happen in any meaningful regulatory or legal sense.
Authorisation trail	Was this AI system sanctioned for use in this context by someone with the authority to sanction it?	Shadow AI use, tools adopted without IT or governance approval, creates liability that the organisation may not be aware of until it is too late.

The organisation's role and the individual's role

Accountability for AI outputs is not solely an individual question. Organisations that deploy AI systems have obligations that sit above the individual user. They include selecting systems that are appropriate for the task, providing training that is adequate for the level of risk involved, setting governance policies that are specific enough to be followed, and creating a culture in which a professional can raise concerns about an AI output without it being treated as an obstruction.

Where those organisational obligations are not met, individual accountability does not disappear, but it is shared and it can be mitigated. A professional who followed a documented procedure, checked what they were required to check, recorded what they did, and escalated what they were unsure about is in a substantially better position than one who accepted an output because it looked fine and time was short.

The question of who is responsible for AI outputs in the workplace is therefore both structural and personal. The structural part is the organisation's job. The personal part is yours, and it is yours regardless of whether the organisation has sorted out the structural part.

Why this matters more in some sectors than others

In sectors where professional regulation applies, the stakes attached to AI accountability are higher and more immediate. Healthcare, legal services, financial advice, education, social care, engineering and architecture all have regulators who expect practitioners to exercise professional judgement. An AI tool does not hold a professional registration. The practitioner does. If an AI-assisted output causes harm and the practitioner's defence is that the tool said so, that defence is likely to make the situation worse rather than better in front of a fitness-to-practise panel or a professional indemnity insurer.

The same principle applies in less formally regulated roles. A data analyst who relies on an AI tool to produce summaries that are then used in business decisions has a responsibility to understand the limits of that tool. Understanding what questions to ask about a dataset, and what an AI summary might be missing or misrepresenting, is precisely what qualifications at the right level are designed to build. The NCFE Level 2 Certificate in Data Analysis guide explains how that foundational layer of analytical thinking is structured for learners who are working with data but have not yet formalised their skills.

The Certificate versus the Award: a practical distinction

The NCFE Level 3 Award in the Application of Artificial Intelligence covers how AI systems work, where they are used and the broad landscape of AI tools in professional settings. It is a solid foundation and suitable for many learners.

The Certificate extends that foundation into the territory of organisational responsibility. Unit 05 is where the Award stops and the Certificate continues. For a learner who manages a team that uses AI tools, who works in a regulated sector, or who is in a role where AI outputs inform decisions that affect other people, the Certificate is not an optional upgrade. It is the relevant qualification for the role they are in.

The practical argument is simple: if your name appears on a process that uses AI, or if you approve outputs that others act on, you need the skills that Unit 05 builds. The Award tells you what AI is. The Certificate tells you what to do when it goes wrong and how to structure your practice so that you can demonstrate you did everything that was reasonably expected of you.