An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Ethics and Governanceintermediate

Algorithmic bias

Q: A lender removes gender from its credit model and confirms the coefficient is gone. Approval rates for women remain 9 percentage points below men's. What is the most likely explanation?

Other variables in the model correlate with gender and carry the same information. Redundant encoding is the normal case in a rich feature set: occupation, income history, part-time patterns and spending categories all carry gender signal. The suggestion that the remaining gap simply reflects genuine risk might turn out to be true, but it cannot be concluded from deleting a column, and that is the point, because you can no longer measure what you deleted.

Q: A recidivism tool is equally well calibrated for two groups, so a score of 7 implies the same reoffending probability for anyone, and the two groups reoffend at different underlying rates. The tool is accurate but not perfect. Critics show it produces a higher false positive rate for one group. Who is right?

Both, because when base rates differ and the classifier is imperfect a calibrated score cannot also equalise the false positive and false negative rates, so the real disagreement is about which criterion should govern. This is the impossibility result Chouldechova and Kleinberg and colleagues proved. Both measurements can be correct at the same time, because calibration and equality on both error rates cannot all hold once base rates differ and the classifier makes mistakes. The question underneath is a normative one about which errors matter more and to whom, and no additional metric settles it.

A model is biased when the errors it makes, the outcomes it allocates or the burdens it imposes differ systematically across groups of people. The cause is rarely a column labelled with a protected attribute. Bias enters through the label you chose to predict, which is usually a convenient proxy for the thing you actually care about; through the sample, meaning who is in the training data and who is missing; and through historical outcomes, because the data records what the organisation used to do and a model that fits it well reproduces it faithfully. Fairness has several formal definitions that contradict one another, so "make it fair" is not a single instruction.

Why it matters

A model learns to repeat the pattern in the data it was handed. If that data records what an organisation used to do, the model learns to do the same thing faster, more cheaply and with a much straighter face. It is not prejudiced. It is obedient. That is why deleting the gender column changes almost nothing when twenty other columns carry the same signal between them.

Before you read on — recall

A lender removes gender from its credit model and confirms the coefficient is gone. Approval rates for women remain 9 percentage points below men's. What is the most likely explanation?

Formulas

Demographic parity

P(\hat{Y}=1 \mid A=a) \;=\; P(\hat{Y}=1 \mid A=b)

Equal selection rate across groups

a

and

b

. It ignores whether the groups differ in the outcome being predicted, so enforcing it can mean selecting people the model scores lower.

Equal opportunity (equal true positive rate)

P(\hat{Y}=1 \mid Y=1,\, A=a) \;=\; P(\hat{Y}=1 \mid Y=1,\, A=b)

Among the people who genuinely have the outcome, the same share is identified in each group. This is usually the criterion that matters when the cost of a miss falls on the individual rather than the organisation.

Predictive parity (equal precision)

P(Y=1 \mid \hat{Y}=1,\, A=a) \;=\; P(Y=1 \mid \hat{Y}=1,\, A=b)

A positive flag means the same thing whichever group it lands on. When base rates differ across groups and the classifier is imperfect, this cannot hold at the same time as equal false positive and false negative rates. That incompatibility was proved independently by Chouldechova and by Kleinberg, Mullainathan and Raghavan.

Worked examples

Scenario

A screening model shortlists graduate applicants. The team checks it for bias, finds equal precision in both groups, and declares it fair.

Solution

Run the arithmetic. Group A has 1,000 applicants of whom 300 would succeed in the role. The model shortlists 200 and 180 of those would succeed. Group B has 1,000 applicants of whom 250 would succeed. The model shortlists 100 and 90 of those would succeed. Precision is 180/200 and 90/100, so 90 per cent in both groups, and the team's test passes. But the true positive rate is 180/300, or 60 per cent, for A and 90/250, or 36 per cent, for B. The model is equally trustworthy when it says yes, and it never sees most of the capable people in B.

Scenario

A health system ranks patients for a care-management programme using predicted future health spending, because spending data is complete, timely and easy to obtain.

Solution

Spending measures how much care a person received, not how sick they are. Where a group has historically received less care at the same level of illness, the model reads that group as healthier and ranks them lower, which withholds exactly the extra help that would have closed the gap. Obermeyer and colleagues documented this in a widely deployed US algorithm in 2019, and switching the label from cost to a direct measure of illness sharply increased the number of Black patients flagged for additional care. Nothing was wrong with the code. The bug was the choice of target variable.

Common mistakes

✗Removing protected attributes makes a model fair. Postcode, school, name, occupation, purchase history and browsing behaviour all correlate with protected attributes, so the model reconstructs them from what is left. Deleting the column removes your ability to audit, not the model's ability to discriminate.
✗Bias is a data problem, so cleaner data fixes it. Some of it does live in the sample, and coverage gaps are real. The most damaging kind lives in the label: you chose to predict spending, arrests or clicks because they were measurable, and the model is faithful to that choice. No amount of cleaning fixes the wrong target.
✗A fair model is one that satisfies the fairness metric. There are several incompatible definitions, and when group base rates differ, equal precision and equal error rates cannot both hold. Choosing among them is a decision about which harm you are willing to accept, and it belongs to the business, not to the modeller alone.
✗If the model beats the humans it replaces, it is an improvement. A model applies one rule to everyone at once, so a modest systematic error becomes a consistent, scaled and hard-to-contest one. Human decisions are noisier but their inconsistency also leaves more room for a person to be heard.

Revision bullets

•Bias enters via the label, the sample and historical outcomes, not a protected column
•Dropping a protected attribute leaves the proxies intact and removes the audit
•Demographic parity, equal opportunity and predictive parity are different tests
•Different base rates plus an imperfect model means those tests cannot all pass
•Obermeyer et al. (2019): cost as a proxy for need read sicker patients as healthier
•Choosing the fairness criterion is a decision about which harm is acceptable

Quick check

A lender removes gender from its credit model and confirms the coefficient is gone. Approval rates for women remain 9 percentage points below men's. What is the most likely explanation?

A recidivism tool is equally well calibrated for two groups, so a score of 7 implies the same reoffending probability for anyone, and the two groups reoffend at different underlying rates. The tool is accurate but not perfect. Critics show it produces a higher false positive rate for one group. Who is right?

Connected topics

Correlation vs Cause Sampling Bias Supervised vs Unsup.NLP Sentiment Analysis Why Ethics Matters Filter Bubbles Ethics by Design Agentic AI Human-AI Agency

More in Ethics and Governance

PAPA Framework GDPR Personalisation Data Governance

Sources

Obermeyer, Powers, Vogeli & Mullainathan (2019)
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. "Dissecting racial bias in an algorithm used to manage the health of populations." Science, 366(6464), 447-453, 2019.
The clearest published case of label bias: health cost used as a proxy for health need, and what changed when the label was replaced.
Buolamwini & Gebru (2018)
Buolamwini, J., & Gebru, T. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." Proceedings of Machine Learning Research, 81, 77-91, 2018.
Audited three commercial classifiers and found error rates under one per cent for lighter-skinned men and up to roughly a third for darker-skinned women, traced to who appeared in the training and benchmark images.
Chouldechova (2017)
Chouldechova, A. "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments." Big Data, 5(2), 153-163, 2017.
Shows that when base rates differ, a predictor cannot be both calibrated and equal in false positive and false negative rates.
Kleinberg, Mullainathan & Raghavan (2017)
Kleinberg, J., Mullainathan, S., & Raghavan, M. "Inherent Trade-Offs in the Fair Determination of Risk Scores." Innovations in Theoretical Computer Science (ITCS), 2017.
The independent formal statement of the same incompatibility, framed as three conditions that cannot hold together except in degenerate cases.

How to cite this page

Dr. Phil's Quant Lab. (2026). Algorithmic bias. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-algorithmic-bias