The Approval Button You Call Human Oversight Probably Does Not Work

The sentence I hear most often when AI governance comes up is this: "There's human approval, the decision isn't left to the machine."

When I hear it, I have one question. What's your approval rate?

There's usually no answer, because nobody measures it. Where it is measured, the figure is typically above ninety-eight per cent. The reviewer overrides the model roughly once in fifty decisions. Calling that human oversight is formally correct and functionally wrong. The system is making the decision; the human is only signing it.

Why the approval button stops working

The cause isn't bad faith. It's design.

Show a person three hundred recommendations a day and ask them to approve each one, and they will automate themselves within weeks. If the model's accuracy is high, this happens faster, because the reviewer is usually right on the few occasions they do reject, and they stop trusting their own judgement.

This is a documented phenomenon called automation bias: the better a system performs, the weaker human oversight of it becomes. Any oversight design has to take that into account from the start.

Three conditions for oversight to be real

For oversight to be functional, three things have to hold together.

The reviewer has to be able to see the basis for the recommendation. If the inputs the model relied on aren't shown, the person can only judge the outcome, and judging the outcome properly means deciding the case from scratch. That makes oversight impossible in practice.

Rejecting has to cost nothing extra. A user who has to fill in a form, write a justification and get a manager's sign-off every time they reject a recommendation will stop rejecting. The rejection path has to be at least as easy as the approval path.

A rejection has to lead somewhere. If rejected recommendations are logged and nobody looks at them, the reviewer notices and disengages over time. Feeding rejected cases back into improving the model keeps oversight alive.

There is no oversight you don't measure

Even once those three conditions are in place, measurement is still needed.

At minimum, three indicators should be tracked: the override rate, the average time spent per decision, and how often an overridden recommendation later turns out to have been correct. If time per decision has dropped to a matter of seconds, oversight isn't happening. The same conclusion follows if the override rate is close to zero.

Very few organisations can answer these questions when asked in an audit. Yet they aren't hard to measure — they come out of data the system already produces.

The right to challenge shapes the architecture

The relevant provision of Article 11 of Turkish Law No. 6698 gives a person the right to object where a result adverse to them arises solely from analysis by automated systems.

The technical consequence is direct. When a challenge comes in, you have to be able to show how the decision was reached. If you can't show it, you can't defend it.

The architectural rule that follows is this: a model that produces no rationale cannot be used in a flow that produces a result about a person. That has to be written in as a constraint on model choice from the outset. Bolting on explainability afterwards is not possible in most architectures.

Keeping a log isn't enough — you have to log the right thing

An audit trail is usually understood as keeping a log, but what the log contains is what decides its value.

For a decision to be reviewable later, five things need to be recorded together: the data that went into the system, the model's version, the output produced, the rationale the output rests on, and the human's final decision. Kept in separate systems, these five can't be matched up afterwards. If the model version isn't recorded, there is no way to answer, months later, "what was the model saying at that point in time?"

Failing to record the model version is the single gap I see most often in practice. The model gets updated, and old decisions get explained by reference to the new one — and it doesn't hold up.

The difference between confident and uncertain has to be shown

One last point. A system should show, in a way the user can tell apart, when it is producing output with low confidence versus high confidence.

Without that distinction, the user trusts everything equally and gets no warning exactly where an error is most likely. Yet a model's own uncertainty is usually measurable. Surfacing that measurement in the interface is the cheapest, most effective improvement you can make to oversight.

If I had to pick one thing to do on an AI project today, this is what I'd pick.