Why the Data Won't Tell You Why Judea Pearl, the numbers that can't speak for themselves, and the assumptions hiding inside every decision

Every important question a manager asks is secretly a question about cause. Will this promotion lift sales, or just relabel the sales we would have gotten anyway? Does the training program make employees better, or do better employees pick the program? For a century, statistics answered such questions with a warning — correlation is not causation — and then fell silent. This essay is about the people who refused to stay silent, the machinery they built to get causal answers out of ordinary data, and the uncomfortable catch that machinery can never escape.

There is a sentence every student of statistics learns to recite, usually with a knowing nod: correlation does not imply causation. Ice-cream sales and drownings rise together, but neither causes the other; summer causes both. The mantra is true, it is important, and — taken as a stopping point rather than a starting point — it is one of the most quietly damaging ideas in the working world.1

Because the questions we actually need answered are causal ones. A correlation between a loyalty program and higher spending is not, by itself, a reason to expand the program. Maybe the program causes people to spend more. Or maybe the people who were already going to spend more are exactly the ones who sign up. Those two worlds produce identical spreadsheets and demand opposite decisions. "Correlation is not causation" correctly tells you the spreadsheet can't settle it — and then leaves you exactly where you started, with a decision to make and a warning label where an answer should be.

The modern science of cause and effect grew out of a refusal to accept that dead end. Its most public champion is the computer scientist Judea Pearl, whose 2018 book with Dana Mackenzie, The Book of Why, argued that causation is not some metaphysical luxury beyond the reach of data — it is a kind of information, with its own algebra, that we can reason about rigorously.1 Pearl's provocation was blunt: statistics spent a hundred years building elaborate tools to describe associations while treating the word cause as unscientific. In his telling, the data by itself is — his phrase — "profoundly dumb." It can tell you that people who took the medicine recovered faster; it cannot tell you whether the medicine helped or whether the people who could afford medicine were going to recover anyway.

That is the debate this essay lays out. On one side, a hard-won statistical caution: without an experiment, keep your causal claims to yourself. On the other, Pearl's causal revolution: you can reason from data to cause — provided you first do something the data cannot do for you. What that something is, why it works, and why it can never be fully verified is the arc of what follows. Along the way we'll open up the machinery so you can turn the cranks yourself.

IThe ladder you're standing on

Pearl's central image is a ladder of causation with three rungs, corresponding to three things a mind — or a machine — can do.1

The bottom rung is seeing: association. What does a customer who bought coffee also tend to buy? This is the rung of correlations, of most classical statistics, and of nearly all of machine learning. A model on this rung can predict beautifully. Shown enough data, it will tell you that umbrella sales and rain go together, and it will forecast tomorrow's umbrellas from tomorrow's clouds. What it cannot do is answer the next question.

The middle rung is doing: intervention. If I hand out umbrellas, will it rain less? Obviously not — but nothing on the first rung knows that, because the correlation between umbrellas and rain looks the same whether you're watching the world or reaching into it. To climb to the second rung you have to distinguish seeing an umbrella from placing one, and that distinction, Pearl insists, is not in the data. It has to be supplied by a model of how the world is wired.1

This is why the umbrella example feels trivial but the loyalty-program example does not. In both cases the data sits on rung one. In both cases the decision lives on rung two. We only feel the gap when our common sense doesn't already supply the causal model for free. The whole project of causal inference is a set of tools for climbing that gap deliberately, in cases where intuition can't carry us.

And the reason the climb is hard — the reason it took a century and a Turing Award to formalize — is that the data can actively mislead you about which way the arrows point. Before we build anything, look at how badly.

IIWhen the numbers lie to your face

In the fall of 1973, the University of California, Berkeley admitted about 44% of its male graduate applicants and about 35% of its female applicants.2 The gap was large and statistically unmistakable, and it looked exactly like what it appeared to be: bias against women. The university was sued.

Then three statisticians — Bickel, Hammel, and O'Connell — broke the numbers down department by department. Inside almost every individual department, women were admitted at rates equal to or higher than men.2 The aggregate said one thing; every piece of the aggregate said the opposite. Both were arithmetically correct. What resolved the contradiction was a fact about the world, not about the data: women were applying in larger numbers to the most competitive departments — the ones that rejected nearly everyone, of either sex. The "bias" in the pooled numbers was a shadow cast by where people chose to apply.

This sign-flip has a name: Simpson's paradox, after a 1951 paper by E. H. Simpson, though it long predates him.3 It is not a rare curiosity. It appears in kidney-stone treatment trials, where a therapy looks worse overall but better for both small stones and large stones separately;4 it appears in salaries, batting averages, and, routinely, in the dashboards businesses use to decide things. The interactive below builds the effect from scratch so you can watch the trend reverse.

Notice what the paradox does and does not show. It does not show that the split view is always the right one — sometimes pooling is correct and splitting is the mistake. It shows something stranger and more important: the data is consistent with both stories, and nothing inside the numbers picks a winner. To choose, you need to say how you think the world is arranged — which variable causes which. That claim comes from you, not from the spreadsheet. Which raises the obvious question: how do you say it precisely enough to compute with?

IIIYou have to draw the picture first

Pearl's answer, and the engine of modern causal inference, is the causal diagram: a graph in which an arrow from X to Y means "X is a direct cause of Y." Drawing it forces you to commit, on the record, to a theory of how your variables are wired. And once the arrows are on paper, a small set of rules tells you exactly which variables you must adjust for — and, crucially, which you must not.

That second part is the surprise. The instinct of most analysts, faced with a messy comparison, is to "control for everything" — throw every available variable into the regression and hope the confounding washes out. Causal diagrams reveal this to be not just crude but actively dangerous. There are three basic ways three variables can be wired, and they behave completely differently when you adjust for the middle one.

The collider case is the one that ends careers, because it is the most counterintuitive. Condition on a common effect of two independent causes and you conjure a correlation between them that was never there — the statistical version of noticing that, among people you've dated, the attractive ones tend to be unkind, not because looks cause cruelty but because you selected on a third thing (you agreed to date them).8 Real analyses trip this wire constantly: adjusting for a variable that sits downstream of the treatment, or studying only the customers who survived long enough to appear in your data. The tidy phrase for the general error is the Table 2 fallacy — reading every coefficient in a regression as if it were a clean causal effect, when most of them are contaminated.11

So the diagram tells you which variables to adjust for to strip out confounding. Do that correctly and the association that survives is — the theory promises — the causal effect. But there is a cleaner, more brutal way to get a causal effect, one that needs no diagram at all. It is worth seeing why it works, because its logic explains everything the diagram is trying to imitate.

IVSeeing versus doing

Pearl marks the difference between the first and second rung with a piece of notation: P(Y | X) versus P(Y | do(X)). The first is seeing — the distribution of outcomes among people who happen to have X. The second is doing — the distribution if you reached in and set X yourself, for everyone. The umbrella and the rain have the same P(rain | umbrella) as always, but P(rain | do(umbrella)) is just the base rate of rain: intervening severs the umbrella from whatever used to predict it.

There is one procedure that physically performs do(X) in the real world, and it is the most important invention in the history of empirical science: the randomized experiment, formalized by R. A. Fisher in the 1930s.5 When you assign treatment by a coin flip, you cut every arrow that used to point into the treatment. Rich and poor, healthy and sick, eager and reluctant — all of them are now equally likely to be treated, so no background trait can confound the comparison. Randomization doesn't measure confounders and adjust for them; it annihilates them, including the ones you never thought to record. Below, watch what confounding does to a naive comparison, and what randomization undoes.

Here, finally, is Pearl's real claim, stated precisely. Given a correct causal diagram, there is a rule — the back-door criterion — that identifies exactly which variables to adjust for so that P(Y | do(X)) can be computed from ordinary observational data.9 In other words: draw the right graph, adjust for the right things, and you can simulate the coin flip you never got to make. The training program's true effect can be recovered from the messy voluntary-signup data — if the diagram is right. That "if" is where the whole edifice becomes honest, or dishonest.

VThe assumptions are doing the work

Return to the back-door recipe and read the fine print. It lets you recover the causal effect provided you have measured and adjusted for every confounder. Miss one — a variable that quietly influences both who gets treated and how they turn out — and your carefully adjusted estimate is biased by an unknown amount in an unknown direction. This assumption has a name, no unmeasured confounding, and it has a devastating property: the data cannot test it. A confounder you didn't measure is, by definition, absent from your dataset. No amount of cleverness with the numbers you have can reveal the influence of a number you don't.

This is where the data purist's old caution earns its keep. Pearl is right that observational data plus a correct diagram yields causation. But "a correct diagram" smuggles in claims about the world that no observation can confirm. Every observational causal estimate is therefore conditional — true if the graph is complete, silent about how big an "if" that is. For a long time, researchers simply asserted the assumption and moved on.

The modern discipline's answer is not to pretend the assumption is safe but to ask, out loud, how fragile the conclusion is if it isn't. This is sensitivity analysis. One clean version, the E-value of VanderWeele and Ding, asks: how strong would an unmeasured confounder have to be — in its pull on both treatment and outcome — to fully explain away the effect you reported?12 A finding with a huge E-value is robust: only a monster confounder could erase it, and you'd probably have noticed a monster. A finding with a tiny E-value is fragile: a whisper of hidden bias undoes it. Same p-value, same confidence interval — wildly different trustworthiness. Turn the dial below and watch a "significant" result evaporate.

So the debate resolves not into a winner but into a division of labor. The causal modeler is right: you can climb from seeing to doing on observational data, and the tools — diagrams, the back-door criterion, the do-operator — are real and rigorous, not hand-waving. The data purist is right too: that climb always rests on a ladder of assumptions the data itself cannot inspect, so a causal claim without a stated sensitivity to those assumptions is a claim wearing a lab coat it hasn't earned. Both cautions are load-bearing. Drop either and you get bad decisions — paralysis on one side, false confidence on the other.

VIWhat this means the next time you open a dashboard

Strip away the notation and a working discipline remains, useful to anyone who decides things from data.

First: name the question's rung. "What's associated with churn?" is a rung-one question and a dashboard can answer it. "Will this retention offer reduce churn?" is rung two, and no amount of staring at the historical correlation between offers and churn will answer it — because in the past, offers went to the customers most likely to churn. Knowing which rung you're on tells you instantly whether your data can even in principle deliver what you're about to ask of it.

Second: draw the graph before you run the regression. The five minutes spent sketching what causes what — is this variable a confounder to adjust for, a mediator to leave alone, a collider to never touch? — will save you from the most common and most invisible analytic error, which is controlling for the wrong things and reporting the bias as a result. "Adjust for everything" is not caution; it is a way to inject collider bias while feeling responsible.

Third: prefer the coin flip when you can afford it. An A/B test is do(X) made real, and its great virtue is that it works even when your causal diagram is wrong, because it needs no diagram. The reason to master observational methods is not that they beat experiments — they don't — but that the most important questions are often the ones you're not allowed to randomize.

Fourth: when you can't experiment, report your fragility. Attach to every observational causal claim the answer to one question: how strong a thing you didn't measure would have to be to overturn this. That number is not a technicality. It is the whole difference between a decision made with open eyes and one made in the dark while insisting the lights are on.

The mantra was never wrong. Correlation is not causation. But the century-old habit of stopping there — of treating the gap between them as a locked door rather than a climb with known handholds — has cost more good decisions than any single statistical error. The data really is, as Pearl said, profoundly dumb. It has no idea why anything happens. That was never its job. The why is yours to supply, to draw, to defend, and — this is the part the mantra left out — to state honestly enough that someone can check how much weight it will bear.