Would You Let AI Make That Decision?

Three requests land on the same team in one afternoon. Tag incoming support tickets by topic. Approve refunds up to $50 without a human looking. Reject loan applications that score below a threshold.

All three amount to “let the AI decide,” and almost nobody treats them the same. Most people’s instincts say yes to the first, maybe to the second, and no to the third. Those instincts are mostly right. The work is to turn them into criteria a team can apply consistently, and defend when someone asks why.

The previous article in this series assumed the system will sometimes be wrong and focused on catching and containing errors. This one steps back to the decision underneath: how much authority a system should have, who stays accountable, and how that authority should grow.

A decision is more than a prediction

Most decisions bundle three things: a prediction (what is likely to happen), a judgment (what we want, and how to weigh competing goals), and an action. Models are strongest at the first. Will this customer churn? Is this invoice likely fraudulent? Does this scan likely show an abnormality? Those questions have answers that can be checked against reality.

The second part is different. How much risk of a bad loan is acceptable in order to widen access to credit? How should a falsely flagged customer be weighed against a missed fraud? No dataset contains the answer, because these are choices about values, and someone has to own them. A common mistake is letting a model’s output quietly absorb a judgment. A threshold picked by a data scientist on a Tuesday becomes the company’s policy on who gets approved.

The practical consequence is to separate the three on paper. Let a model supply the prediction. Let accountable people set the thresholds and trade-offs in the open. Let code carry out the action. Then “should AI decide?” becomes a narrower question: which of these pieces, and how much of it.

Authority is a ladder, not a switch

“Human in the loop” gets used as if the choice were binary. There are at least five rungs:

1. Informs. The system surfaces information, and a person decides.

2. Recommends. The system proposes an answer. A person decides and may override it.

3. Decides, with a veto window. The system acts unless a person steps in within a set time. This is sometimes called being on the loop.

4. Decides, with sampled audits. The system acts, and people review a sample afterward and look for patterns.

5. Decides, unsupervised. No routine review.

The reversibility ranking from the last article sets the ceiling: irreversible actions shouldn’t sit above the second rung without strong reasons. The questions below choose the rung within that ceiling. The fifth rung is rarer than the hype suggests, and it fits only a narrow slice of work that is high in volume, low in stakes, easy to reverse, and measured continuously.

Seven questions before you hand over a decision

1. Who bears the consequence, and how much? A mis-tagged ticket is an inconvenience. A rejected applicant, a denied claim, or a frozen account can change someone’s life. The more the consequences land on people outside your company, the higher the bar.

2. Can the person contest it? A decision is much safer to automate when the affected person can see that it happened, ask for a human, and get a real review in reasonable time. Without an appeal path, you are asking the model to be right with no safety net.

3. Can you explain it in terms a person can act on? “Declined because stated income was below the minimum” is something a person can respond to. A model’s generated account of why it declined is another output, not a record of its reasoning. Where explanations are owed, base the decision on factors you can state.

4. Will you ever find out whether it was right? Watch for what researchers call the selective labels problem. If the system automatically rejects applicants, you never observe whether those applicants would have succeeded, so the evidence needed to check the model stops arriving. Keep a feedback path, such as reviewing a random sample of rejections.

5. Is it a prediction or a value trade-off? Predictions can be measured. Trade-offs need an owner who can defend them.

6. Does it beat the real alternative, including across different groups? Compare against how the work is done today, not against perfection and not against a strawman. Look at error rates by customer segment, because an average can hide a group the system consistently fails. Compare the shape of the errors too. People are noisy: their mistakes are scattered and inconsistent. A model is consistent, including when it is consistently wrong, so the same mistake repeats for every similar case, at machine speed.

7. Who is accountable when it’s wrong? Name a person or a role. Not “the AI team,” and not “the vendor,” but someone who answers to the customer, the regulator, and the board.

If any answer is shaky, move down a rung.

Oversight that actually oversees

Putting a person in the loop accomplishes nothing if the person can’t do the job. Real oversight needs four things:

• Authority. The reviewer can overrule the system without penalty, and isn’t measured in a way that makes disagreement costly.

• Information. They see the evidence behind the recommendation, such as source passages and the fields used, not just the answer.

• Time. A reviewer clearing two hundred cases an hour isn’t reviewing. Set volumes to match what careful review takes.

• Competence. Reviewers must still be able to do the work. If people only ever approve, their skills fade and oversight degrades without anyone noticing. Keep some cases handled manually and rotate who does them.

A few techniques separate oversight that works from the ceremonial kind. For high-stakes decisions, have the reviewer record their own judgment before seeing the model’s recommendation, since seeing it first anchors people. Seed the queue with test cases whose answers are known, including some deliberately wrong recommendations, and check whether they get caught. And watch the override rate. Near zero means either a very good model or a rubber stamp, and you need to find out which. Very high means the system isn’t helping.

There is also a trap for the reviewer. Researchers call it a moral crumple zone: the human nominally in charge of an automated system absorbs the blame for failures they had little real power to prevent. A person who can’t realistically override the system, but is held responsible for it, is not oversight. It is a transfer of liability. Design the role so the authority is real.

Earn autonomy the way a new hire does

A sensible manager doesn’t give a new employee unlimited authority on day one. They start with supervised work, watch the results, and widen the remit as the track record builds. The same approach works for AI, with explicit gates:

1. Shadow. The system produces decisions that nobody sees or acts on. Compare them with what people actually did.

2. Recommend. People decide with the recommendation in view. Measure agreement, overrides, and outcomes.

3. Act on a narrow slice. Let it decide a well-defined, low-risk subset, such as the clearest cases in one category, with sampled audits.

4. Expand by evidence. Widen the slice only when measured performance on real cases meets criteria written down in advance.

5. Demote on signal. Define triggers that pull authority back automatically, such as an error rate above a threshold or a shift in the kinds of input arriving. These are the circuit breakers from the last article, applied to authority.

The important part is that the promotion and demotion rules exist before the results do. Deciding afterward what counts as good enough is how a mediocre pilot becomes permanent.

“The AI decided” is not an answer

When a customer, a regulator, or a court asks why something happened, “the system decided” does not work as a response. As the last article noted, people are entitled to treat what your system does as your decision.

Regulation is moving in the same direction. GDPR restricts decisions based solely on automated processing that significantly affect people, with safeguards that include a route to human intervention. The EU’s AI Act treats uses such as hiring, credit, and access to essential services as high-risk and expects human oversight to be designed in. US credit rules require lenders to give applicants specific reasons for a denial, and regulators have said that using a complex model doesn’t excuse that. The details differ by jurisdiction and are still changing, so check what applies to your use case with a lawyer. We aren’t lawyers. The direction, though, is consistent, and it matches good engineering: a named owner, a human route, and reasons you can state.

Where we’d draw the lines

These are starting positions, not rules. The seven questions above can move any of them.

Reasonable to automate, with sampled audits and an escape hatch:

• Tagging and routing support tickets

• Flagging invoices for review (flagging, not rejecting)

• Approving small, reversible, rule-bounded refunds

• Filtering spam and obvious abuse, with an appeal path

Recommend, with a person deciding:

• Prioritizing sales leads or support escalations

• Drafting customer replies

• Suggesting a medical code, a contract clause, or a risk rating that someone signs off on

• Summarizing a case file for a reviewer who then decides

Keep with people, with AI at most assisting:

• Hiring, firing, and discipline

• Denying credit, insurance, benefits, or care

• Anything involving legal or health consequences that the affected person can’t easily contest

• Decisions that depend on weighing values the organization hasn’t made explicit

Notice what the first list has in common: the decisions are narrow, reversible, contestable, and measurable. The last list is the opposite.

Start with who answers for it

At PraneoTech, before we build anything, we ask who will answer for the decision when it is wrong, and what they will need in order to do that well. If the answer is vague, the design isn’t ready.

The question isn’t whether the AI is smart enough to decide. It’s whether someone is in a position to answer for the decision, and has the power to change it.

In the final article of this series, we’ll step back from individual features to the bigger picture: why the future isn’t AI replacing software, but AI becoming part of it.