An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

The Analyst and What Comes Nextintermediate

Human and machine agency

Q: A screening model is right 94% of the time. On the same kind of case, a reviewer deciding alone, without seeing the model output, is right 85% of the time. A manager proposes routing every case to a reviewer, whose decision then stands. What is wrong with the proposal?

It replaces the more accurate decision-maker with the less accurate one on cases the model already handles well, so overall accuracy falls. Referring a case swaps one accuracy for the other. Sending everything to reviewers who are right 85% of the time replaces a 94% decision with an 85% one across the board. Referral earns its keep only when it is aimed at the cases the model is worst at, so the routing rule matters far more than the volume of review. The comparison works here only because the reviewer decides unaided. Once a person sees the model output first and may override it, joint accuracy depends on the override rule and on whether the two sets of errors overlap, which no single accuracy figure tells you.

Q: An insurer's automated system declines a claim and the customer complains. The insurer replies that the decision was made by the model. Assess that reply.

Unsound, because responsibility sits with the organisation that chose to deploy the system, set its thresholds and decided not to review its output. Someone chose the model, chose the threshold, chose to run it without review and chose what to tell the customer. Those are human decisions and they carry the accountability, whether the model was built in house or bought. The reply also fails a practical test: it gives the customer nothing to contest, which is usually how a complaint becomes a regulatory matter.

Agency is the capacity to choose and to be answerable for the choice. When a model or an agent takes part in a decision, agency gets divided, and the division has to be deliberate. Krakowski (2025) treats agency as distributed by design between people and generative systems. On the reading taken here, the human contribution moves towards setting direction and judging output rather than producing it. Two failure modes bracket the problem. Over-reliance is accepting machine output that should have been questioned. Under-use is ignoring a system that is measurably better than the person ignoring it. Both are settled by how the work is designed, not by how people are told to feel about the tool.

Why it matters

Autopilot did not make pilots redundant. It changed what they do, and it created a new job that turns out to be hard for humans: watching something that is almost always right, staying alert, and taking over in the rare minute when it is not. Analytics is arriving at the same place. The work moves from producing the answer to deciding what to ask, checking what came back, and owning what happens next.

Before you read on — recall

A screening model is right 94% of the time. On the same kind of case, a reviewer deciding alone, without seeing the model output, is right 85% of the time. A manager proposes routing every case to a reviewer, whose decision then stands. What is wrong with the proposal?

Formulas

Accuracy of a human-machine pair

a_{\text{system}} = (1 - r)\,a_{\text{model}} + r\,a_{\text{human}}

Let

r

be the share of cases referred to a person, and

a

the accuracy achieved on the cases each side actually decides. The expression assumes each side settles its own cases, so the two accuracies are measured on different, non-overlapping sets. Referring cases only helps when the human is better on exactly those referred cases. If a model is right 92% of the time and a rushed reviewer is right 80% on what reaches them, sending more cases to the reviewer lowers system accuracy rather than raising it. The value is in the routing rule, not the volume of review. Where a person instead checks the model output and may override it, this expression no longer applies, because joint accuracy then turns on the override rule and on how far the two sets of errors overlap.

Worked examples

Scenario

A bank's fraud model flags 500 transactions a day. Two analysts can review about 60 of them between them. Management describes the process as human in the loop.

Solution

Count first. The humans see 12% of the flags, so 88% are actioned by the model alone, and the label describes an aspiration rather than the process. Three honest moves are available: raise the threshold so the model flags nearer 60 cases and accept the misses that creates, triage the queue so analysts see the cases the model is least certain about rather than the top of a list, or fund more reviewers. Doing none of the three while keeping the label is the option that ends badly in a regulatory review.

Scenario

A demand planner overrides the forecasting model on roughly a third of product lines. A review finds the overrides make the forecast worse on average, but the planner is right on the small number of lines affected by promotions.

Solution

This is a routing problem, not a discipline problem. Blanket overriding destroys value, and banning overrides throws away the planner's genuine private knowledge of promotions that never reached the model's data. Narrow the right of override to cases where the human holds information the model cannot see, log every override with its stated reason, and measure the overridden and non-overridden groups separately from then on. Divided agency works when each side gets the decisions it is actually better at.

Common mistakes

✗Keeping a human in the loop makes a system safe. It makes a system safe only if that human has the time, the information and the standing to disagree. A reviewer processing one case a minute against an unfamiliar model score is a formality, not a control.
✗People who over-trust automation are being careless. Over-reliance is a predictable response to a system that is right almost every time. Sustained vigilance against rare failures is something humans are known to be poor at, so the answer is to design the checkpoint rather than to lecture the operator.
✗If the model is more accurate than the human, the human should always defer. Average accuracy hides where each side is strong. The gain comes from routing each case to whichever side is better on that kind of case, and from the human's access to information the model never had.
✗Accountability follows the automation. Responsibility for a decision stays with the organisation and the people who chose to deploy the system and set its thresholds. A model cannot be a defendant, and calling an outcome the model's choice is a description of a governance gap.

Revision bullets

•Agency = the capacity to choose plus answerability for the choice
•Over-reliance and under-use are the two symmetric failure modes
•Human in the loop is only real when review capacity matches alert volume
•Referral helps only when the human is better on the referred cases
•Automation shifts human work from producing answers to judging them
•Accountability stays with the deployer, never with the model

Quick check

An insurer's automated system declines a claim and the customer complains. The insurer replies that the decision was made by the model. Assess that reply.

Connected topics

What Is a Decision Decision Support Algorithmic Bias Ethics by Design Agentic AI Changing Analyst Role

More in The Analyst and What Comes Next

Analytics as a Service Analytics Roles

Sources

Krakowski (2025)
Krakowski, S. "Human-AI agency in the age of generative AI." Information and Organization, 35(1), 100560, 2025.
Treats agency as something distributed by design between people and generative systems. The direction-setting and judgement framing used in this node is the atlas's reading of that argument, not a quoted claim.
Parasuraman & Riley (1997)
Parasuraman, R., & Riley, V. "Humans and Automation: Use, Misuse, Disuse, Abuse." Human Factors, 39(2), 230-253, 1997.
The source of the misuse (over-reliance) and disuse (under-use) framing used in this node.
Bainbridge (1983)
Bainbridge, L. "Ironies of Automation." Automatica, 19(6), 775-779, 1983.
Shows why monitoring an almost-always-correct system is a hard job, and why the operator left with the exceptions has the harder task.
NIST AI RMF 1.0 (2023)
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023.
A public framework that treats human oversight and clear accountability as design requirements rather than afterthoughts.

How to cite this page

Dr. Phil's Quant Lab. (2026). Human and machine agency. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-human-ai-agency