What Happens When AI Gets It Wrong?

A customer asks a support assistant whether she can return a discounted item. The assistant, trying to be helpful, tells her yes: within 60 days, for a full refund. No such policy exists. She takes a screenshot. Two days later she is in the support queue holding it, and an agent has to decide whether to honor a promise nobody at the company ever made.

The first two articles in this series covered whether a problem needs AI and what the system can see. This one starts from a premise every honest AI project has to accept: the system will sometimes be wrong. Not because of a bug you can fix once, but because probabilistic systems produce errors as a normal operating condition. What separates a demo from a product is what happens next.

Reliability engineers have a name for the right mindset: assume failure. Design so that errors are detected quickly, contained narrowly, and fixed permanently. The rest of this article is what that looks like when the component that fails is a language model.

Failures come in different kinds

“The AI was wrong” covers several different problems, and each has its own detector and its own fix. Six are worth separating:

• Plausible but unsupported. The answer reads well and isn’t backed by anything in the sources. This is the failure people usually mean by hallucination.

• Misread intent. The model answers a different question from the one asked, often because the request was ambiguous, a long input was truncated, or key information was buried deep in a long prompt, where models can use it less reliably.

• Contract violations. Output that doesn’t match the expected format, a tool call with invalid arguments, a response cut off mid-sentence. These are the easy ones, because they are loud.

• Overreach. The system takes an action beyond what was intended: sends the email, issues the refund, edits the record. These are the dangerous ones, because they have side effects.

• Drift. It worked last month. Then the provider updated the model, someone edited a prompt, the source documents changed, or customers started asking about something new. Nothing crashed. Quality just moved.

• Manipulation. A user, or a document the system reads, deliberately steers it: a customer coaxing a discount out of a chatbot, or hidden instructions inside a retrieved file.

Stale or wrong retrieved context, the subject of the last article, shows up in production as the first kind: a fluent answer that happens to be wrong. Frequency tells you less than you’d expect about which of these matters most. Consequence does, and that is the question most teams skip.

Rank by reversibility, not by frequency

The common instinct is to fix the most frequent errors first. For AI features, a better ordering is by what an error costs and whether it can be undone. A typo in a drafted email that a person reviews costs seconds. A wrong refund issued automatically costs money. A wrong message sent to ten thousand customers costs trust that no refund repairs.

Three questions classify any action the system can take:

• Can it be undone? A draft can be discarded. A sent email, a payment, a deletion, or a published statement can’t.

• How wide is the blast radius? One user’s session, or every customer at once?

• How fast does damage accumulate? An error caught within an hour is a different incident from one caught after a month.

The answers sort capabilities into rough tiers. Suggestions a person sees before anything happens carry the lowest risk. Reversible actions such as tagging, drafting, or updating internal records carry moderate risk and should come with an undo and a log. Irreversible or external actions, such as payments, customer-facing messages, deletions, and legal or medical statements, need an explicit approval step or shouldn’t be automated at all. The principle underneath is that autonomy should be proportional to reversibility. Which decisions an AI should be allowed to make is the subject of the next article. Here, the point is that good design starts with this ranking.

Detect it before your customers do

Most AI failures are silent, so detection has to be built in layers, cheapest first.

Deterministic checks. Code is the best error detector available, because it doesn’t share the model’s blind spots. Validate the schema. Check values against allowed lists and sane ranges: a refund above the policy cap, a date in the past, an order number that doesn’t exist. Scan output for things it must never contain, such as personal data or promises outside approved language. These checks are fast, free, and loud.

Grounding checks. For answers built from sources, test whether each claim is actually supported by the passages the system retrieved: verify citations, run an entailment model, or use a second model as a judge. Treat the judge as a triage tool, not as the truth. Model judges have known biases, including a preference for fluent and longer answers, so calibrate them against human labels before trusting their scores.

Disagreement, not self-reported confidence. Asking a model how sure it is rarely produces a reliable number, since stated confidence is often poorly calibrated. Observable signals work better: repeated samples that disagree with each other, weak retrieval scores, repeated validation failures. Route anything with those signals to review.

Production monitoring. Track rates over time: validation failures, “couldn’t find an answer,” escalations to humans, thumbs-down, the edits people make to drafts, latency, and cost. The absolute number matters less than a sudden change in it, because change is how drift announces itself. And read a stratified sample of real conversations every week: some random, some low-scoring, some flagged. No dashboard replaces reading the transcripts.

Canaries. Keep a fixed set of known questions and run them against production on a schedule. When a model update or a prompt edit quietly changes behavior, the canaries notice before users do.

User reports. People catch errors no automated check will. Make flagging a wrong answer a single click, and make sure those reports reach someone who reads them.

Contain it: make one failure small

Detection tells you something went wrong. Containment decides how much it matters. A handful of design choices do most of the work:

• Least privilege. Anything the model can do, a bad output or a malicious input can also do. Give the system read-only access where possible, scoped credentials, spending and rate limits, and an allow-list of the actions it may request.

• Confirmation on consequential actions. The model prepares; a person or a deterministic rule approves. Show the approver the exact action (“Refund $240 to order 8841”) instead of the model’s description of it, so the review checks the thing that will actually happen.

• Undo and audit. Prefer drafts to sends and soft deletes to deletes, and keep a log detailed enough to reconstruct what happened and reverse it.

• Circuit breakers. When failure signals spike, such as validation failures, unusual refund volume, or a flood of near-identical requests, the system should degrade automatically instead of waiting for someone to notice.

• A kill switch, and something to fall back to. One tested switch that turns the AI path off is only useful if the old process still works. Teams routinely dismantle the manual workflow when the AI launches, then discover during an incident that there is nothing to revert to. Keep it alive.

• Graceful degradation. Decide in advance what the system does when it can’t be trusted: show plain search results instead of a generated answer, hand off to a person, or say plainly that it isn’t sure.

• Staged rollout. Run the system in shadow mode first, producing answers that are logged but not shown, and compare them with what people actually did. Then release to internal users, then to a small slice of traffic. Prompt, model, and retrieval changes get the same treatment as code deploys: tested against the evaluation set, rolled out gradually, with a rollback ready.

Be honest with users about what they’re talking to

Tell people when they’re dealing with AI, show sources, and keep a human reachable. This isn’t only courtesy. In 2024 a Canadian tribunal held Air Canada responsible for wrong information its website chatbot gave a customer about bereavement fares, and rejected the airline’s argument that the chatbot was a separate entity answerable for its own statements (Moffatt v. Air Canada, 2024 BCCRT 149). The ruling came from a small-claims forum, but the principle is one every deployer should plan around: users are entitled to treat what your system says as coming from you. A line of fine print saying “AI can make mistakes” doesn’t transfer that responsibility. Design on the assumption that you own the output.

When it fails anyway, run it like an incident

It will. What distinguishes mature teams is the response.

1. Contain. Flip the switch or degrade the feature. Stop the damage before you understand it.

2. Scope. Use the logs to find which users, which time window, and which outputs were affected. This is where recording inputs, retrieved passages, and versions pays for itself.

3. Remediate. Correct affected customers and records proactively instead of waiting for complaints. Depending on what was said or done, legal or regulatory obligations may apply, so involve the right people early.

4. Find the cause. Place the failure in the categories above. It is rarely “the model was bad.” More often it is a retrieval gap, a prompt change, a missing validation, or an action nobody gated.

5. Leave a permanent mark. Add the failing case to the evaluation set and build the detector or gate that would have caught it. A failure that leaves no test behind will happen again.

Keep the review blameless and the findings written down. Teams that treat the first incident as an embarrassment tend to have a worse second one.

Putting it together

Picture the support assistant from the opening, designed with these ideas. It drafts replies but doesn’t send them. Any refund it proposes is checked by code against the order record and the policy cap. Anything under the cap goes to an agent as a one-click approval showing the exact amount, and anything above it escalates. Replies that make claims not found in the retrieved policy text are blocked and routed to a person. Twenty canary questions run every morning. Validation failures and escalation rates are charted, and a spike pages someone. A tested switch sends all conversations to the human queue, which still exists. And every wrong answer found in production becomes a test.

That assistant will still be wrong sometimes. The difference is that the screenshot never reaches the customer.

Start by assuming it will fail

At PraneoTech, we ask the same question at the start of every AI feature: what is the worst plausible output, and what would it cost? Then we build so that the answer is boring.

Reliable AI isn’t a model that never fails. It’s a system where failures are caught quickly, contained narrowly, and never repeated.

In the next article, we’ll look at the decision underneath all of this: which choices an AI should be allowed to make, which should stay with people, and how to tell the difference.