The Fairness You Can't Have A courtroom algorithm, a fight over what "fair" means, and the theorem that says you can't have it every way at once

In 2016 a newsroom and a software company had a public fight about whether an algorithm was racist. Both sides brought numbers. Both sides were right. What almost nobody noticed at first is that their numbers could not both be made to go away — not because the engineers were careless, but because of a theorem. This essay is about that fight, the machinery underneath it, and the uncomfortable discovery that "fair" is not one thing you can build but several things you must choose between.

When a person is arrested in much of the United States, a piece of software may quietly help decide whether they go home before trial or wait in a cell, how long they are sentenced, and whether they make parole. One widely used tool is COMPAS, built by a company then called Northpointe. It reads answers to a long questionnaire — never the defendant's race — and returns a risk score from 1 to 10: how likely is this person to be arrested again?1 Judges see the number. The number shapes lives.

In May 2016, the investigative newsroom ProPublica published an analysis of COMPAS scores for more than seven thousand people arrested in Broward County, Florida, checked against who was actually re-arrested over the next two years.1,2 Their headline finding was stark. Among defendants who did not go on to reoffend, Black defendants were labeled high-risk almost twice as often as white ones — a false-positive rate of 44.9% versus 23.5%.2 The machine's mistakes, they argued, fell heaviest on Black defendants. The article was called "Machine Bias," and it became one of the most-cited pieces of technology journalism of the decade.

Northpointe fired back with numbers of its own, and here the story stops being simple. Their rebuttal pointed out that COMPAS was calibrated: among defendants the tool flagged as high-risk, the fraction who actually reoffended was about the same for both groups — roughly 63% for Black defendants and 59% for white.3 A high-risk label meant the same thing, statistically, regardless of race. By that standard — the standard the company had been building toward — the tool was not biased at all. It was, they said, working exactly as a fair instrument should.

So which was it? ProPublica measured the algorithm's errors and found them racially skewed. Northpointe measured the algorithm's scores and found them racially even. For a while the debate ran as these debates usually do, with each side convinced the other was either innumerate or acting in bad faith. Then a small group of computer scientists and statisticians proved something that reframed the entire argument: both sets of numbers are correct, and no algorithm on Earth could have made both of them fair at once.

That is the surprise this essay unpacks. The fight over COMPAS looked like a fight about one algorithm's flaws. It was really a fight about which definition of fairness everyone had silently assumed — and about a trade-off so fundamental that it applies to every risk score, every fraud detector, every hiring filter, every model that sorts people into groups where the groups differ. To see why, we have to open the machine.

IWhat a risk score actually is

Strip away the questionnaire and the branding, and a risk tool does one humble thing: it looks at a person and outputs a number meant to track a probability — how likely is the thing we're worried about? That number is not yet a decision. A decision requires a second step almost nobody talks about: a threshold. Score above the line, we act (deny bail, flag the transaction, reject the résumé); below it, we don't. Everything interesting about fairness lives in that line and what it does to the four ways a prediction can turn out.

Those four ways are the confusion matrix. A person either reoffends or doesn't; you either flagged them or didn't. Flag someone who reoffends — a correct catch. Clear someone who stays clean — a correct pass. But flag someone who wouldn't have reoffended (a false positive, a person wrongly burdened) or clear someone who does (a false negative, a risk wrongly released) — those are the errors, and they are not interchangeable. Move the threshold and you trade one for the other. You cannot minimize both.

IITwo groups, one algorithm, both sides right

Here is the fact that makes COMPAS more than a story about one company: in the Broward data, the two groups did not reoffend at the same rate. About 51% of Black defendants were re-arrested within two years, against about 39% of white defendants.2,3 Hold that difference in mind — it is the hinge the whole paradox turns on. Now take a single calibrated algorithm, apply it to both groups at the same threshold, and look at what each side chose to measure.

Sit with how strange this is. Every number both camps reported is correct, computed from the same rows of the same spreadsheet. ProPublica did not exaggerate; Northpointe did not lie. They were answering two different questions — do the scores mean the same thing? and do the mistakes fall equally? — and had simply assumed, as almost everyone does, that a fair algorithm would answer yes to both. The next section is about why that assumption is impossible to satisfy.

IIIThe theorem that ended the argument

In late 2016, two independent results — one by Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, another by Alexandra Chouldechova — proved that the tension in the COMPAS numbers was not a bug anyone could fix.4,5 They formalized several natural definitions of fairness and showed that, whenever two groups have different base rates and the predictor is less than perfect, you cannot simultaneously achieve all of them. Pick calibration and you must accept unequal error rates. Force the error rates equal and calibration breaks. There is no third option, no cleverer model, no better training data that escapes it. It is arithmetic.4

The intuition is almost graspable by hand. If a score is calibrated — a "7" means the same probability for everyone — and one group genuinely reoffends more often, then that group simply contains more high-risk people. Any threshold will flag more of them, including more who happen not to reoffend. The wrongly-flagged rate must rise. Calibration and equal false-positive rates are pulling in opposite directions, and the base-rate gap sets how hard. Don't take the argument's word for it. Try to beat it.

Notice what the theorem does not say. It does not say fairness is meaningless, or that all algorithms are equally biased, or that you may as well give up. It says something more precise and more demanding: fairness is plural. There is a menu of reasonable, rigorous definitions, they genuinely conflict, and choosing among them is not a technical act. It is a value judgment about which mistakes are worse — a judgment that belongs to courts, legislatures, and the public, not to a model's loss function. Which is exactly why handing the decision to a black box feels like a category error. But there is a deeper problem still, one that unsettles both camps.

IVThe ground truth is not the ground

Everything so far — the base rates, the calibration, the error rates, the theorem — takes for granted that we know who "reoffended." We don't. What the data actually records is who was re-arrested, and arrest is not the same as crime. It is crime plus policing. If two neighborhoods commit offenses at the same rate but one is patrolled twice as heavily, the surveilled one will show more arrests, a higher apparent base rate, and — through no fault of the algorithm — will inherit every downstream consequence the theorem describes.1

This is the part that should unsettle everyone. The calibration camp and the error-rate camp were fighting over how to be fair with respect to a target — recidivism — that is itself measured through a biased lens. And there is a further indignity: for all its 137 questions, COMPAS turned out to be no more accurate than random people recruited online given a few facts, and matchable by a model with just two variables.9 A tool can be sophisticated, proprietary, and mathematically calibrated, and still be predicting the wrong thing, badly, in a way that launders enforcement patterns into the language of objective risk.

VSo what does "fair" mean now?

It means you have to choose, and say so. The impossibility theorem is not a counsel of despair; it is a demand for honesty. Because you cannot have every fairness at once, deploying a model on groups that differ forces a decision about which fairness you are buying and which you are giving up — and that decision should be visible, argued, and owned, not buried in a threshold.

Different settings will, and should, choose differently. A cancer screen that flags people for a cheap follow-up test can tolerate a high false-positive rate to drive false negatives near zero — missing a tumor is far worse than a scare. A tool that pushes people toward pretrial detention should weigh false positives enormously, because the cost is a person's liberty before any conviction. The "right" fairness criterion is downstream of a moral question — whose error, at what cost? — and no amount of engineering answers a moral question. It only implements whichever answer you picked, whether or not you noticed picking it.

What the mathematics gives us, then, is not a fair algorithm but an honest vocabulary. It lets us say precisely: this tool equalizes predictive value across groups but not error rates; here is the base-rate gap driving the difference; here is our reason to believe that gap is real and not an artifact of the label; and here is the value judgment behind treating this error as worse than that one. That sentence is long and unsatisfying and full of admissions. It is also the only kind of sentence about algorithmic fairness that is fully true.

VIWhat to carry into your next model review

First: "we removed bias" is not a claim, it's a question. The moment someone says a model is fair or unbiased, ask by which definition. Calibrated? Equal false-positive rates? Equal false-negative rates? Because of the theorem, a model touching groups with different base rates satisfies at most a subset, and "fair" without a named criterion is marketing, not measurement.

Second: audit the errors by group, not just the accuracy. A model can be equally accurate for everyone and still make opposite mistakes for different groups, exactly as COMPAS did. Break out false-positive and false-negative rates and predictive value separately, for each group. The disparities live in the breakdown, never in the top-line number.

Third: interrogate the label before the algorithm. Ask what your outcome variable actually measures versus what it claims to. Re-arrest for crime, cost for illness, "clicked" for "wanted," "was promoted" for "deserved promotion." A biased target contaminates every fairness metric computed on top of it, and no fairness constraint can fix a broken definition of the thing being predicted.

Fourth: put the trade-off where humans can see it. Since fairness is a choice among incompatible goods, make the choice explicit and accountable — a documented decision by people who can be questioned, not an emergent property of a threshold nobody set on purpose. The theorem removed the option of having it every way. What it left is the responsibility to choose, out loud.

The COMPAS fight was never really resolved, because it was never the kind of question that resolves. It was two true accounts of the same machine, colliding with a theorem that says the machine could not have satisfied both. That is not a failure of the algorithm. It is a permanent feature of predicting a world where groups differ — and the beginning of wisdom about it is to stop asking whether a model is fair, and start asking which fairness it chose, at whose expense, and whether anyone decided that on purpose.