Same Betas, Three Standard Errors

This is the empirical companion to a conceptual note, “Weights get the point estimate right; the design gets the uncertainty right.” That note made three claims. Here we put all three on real data — the National Health and Nutrition Examination Survey (NHANES) — and hold every claim to an explicit numerical gate. The setting is deliberately mundane: the association between family income and depression. The point is not the finding; it is that one number can carry three or four different standard errors depending on which variance machinery you turn on — while the estimate itself never moves.

Concretely, we test that (1) point estimates are invariant to the variance machinery — a weighted mean and a weighted regression coefficient are numerically identical whether computed naively, with fixest and clustering, or with the full survey/srvyr design; (2) fixest with weights + clustering at the PSU recovers most of the design-based standard error; and (3) naive standard errors are materially anti-conservative, by an amount we quantify.

1One dataset, three variance machines

NHANES is a stratified, multistage cluster sample. Respondents are nested in primary sampling units (PSUs, roughly counties), PSUs are nested in strata, and each respondent carries a MEC examination weight that scales the sample up to the U.S. population.¹ Pooling the eight PHQ-9 cycles from 2005–2023 and restricting to adults aged 20–80 with a valid income-to-poverty ratio gives an analytic domain of 42,463 respondents, spread over 129 strata and 263 PSUs (2–3 PSUs per stratum, median 2). The weights are far from equal — their coefficient of variation is 0.90, a Kish weighting design effect of $1+\mathrm{CV}(w)^2 = 1.80$ before a single cluster is considered.

The key move is to declare the design once and read four different standard errors off the same estimate. We line up four variance regimes, from the one that knows nothing about the design to the one that knows everything:

V0 is what you get by bolting a textbook standard error onto a weighted group-by. V1 adds White heteroskedasticity-robust corrections. V2 clusters at the PSU — the econometrician’s rendering of the design. V3 is the full Taylor-linearized survey estimator, which additionally nets out the between-stratum variation and uses the design-correct degrees of freedom.² Two families of estimand run through all four: Family A, weighted prevalences of probable major depression (PHQ-9 ≥ 10), overall and by income and age; and Family B, regressions whose headline coefficient is the Low–High income gap.

2The point estimates are identical

Start with the claim that should be boring and is worth making undeniable. A sampling weight fixes who is in the sample; it is the inverse probability of selection, and it enters every estimator as a simple ratio $\hat{\bar Y} = \sum_i w_i y_i / \sum_i w_i$. Nothing about strata or PSUs appears in that formula. So the same number falls out of a naive weighted mean, a fixest weighted regression, and srvyr’s survey_mean() — to the last decimal.³

Across all 42,463-respondent prevalence cells, the closed-form weighted mean and srvyr’s design-based mean agree to max |V0 − V3| = 0.0e+00. For the regressions, feols/feglm and svyglm return the same coefficient vector to 3.3e-14 (linear probability model), 1.8e-12 (linear score), and 1.3e-12 (logit — different IRLS paths, same maximum). Every estimand in the study, plotted below, sits on the 45° line to within 1.8e-12.

The verbatim gate log:

[PASS] POINT-A: srvyr means == closed-form weighted means -- max|V0-V3| = 0.000e+00 over 22 cells
[PASS] POINT-B1: feols == svyglm (LPM) -- max|dcoef| = 3.331e-14
[PASS] POINT-B3: feols == svyglm (linear) -- max|dcoef| = 1.805e-12
[PASS] POINT-B2: feglm == svyglm (logit) -- max|dcoef| = 1.258e-12
[PASS] SANITY-DEFF: A1 implied DEFF (se_V3/se_V0)^2 in [1.2,6] -- implied = 1.470 ; srvyr deff = 2.409

3What the naive SE misses

Now turn on the variance machines one at a time and watch the confidence interval breathe while the point stays pinned. Figure 2 is the thesis in one frame: a single vertical line for the shared coefficient, four stacked intervals of visibly different width.

For that headline gap, the naive standard error is 0.00380 against a design-based 0.00550. The naive interval is 32% too narrow — the honest interval is 1.46× wider — and the t-statistic on the income gap falls from a swaggering 31.4 to a still-decisive 21.6. Here the association is strong enough to survive; for a marginal subgroup contrast, that same factor is the difference between a star and a shrug.

Two structural details earn the tables their credibility. First, the tolerance the logit demanded: feglm and svyglm travel different IRLS routes, so we tightened both to deviance tolerances near $10^{-11}$ to make the coefficients agree past the eighth decimal. Second, and more consequential, the naive iid SE for the logit is only meaningful once the weights are normalized to mean 1.⁴

4Cluster(PSU) gets you most of the way

The good news for practitioners who live in fixest: you do not need the full survey apparatus to get an honest standard error. Weighting and clustering at the PSU — vcov = ~psu_uid — lands within a few percent of the design. Figure 3 shows every shortcut’s SE as a ratio to the design-based truth.

For the three regression coefficients the cluster-robust SE is 1.05×, 1.11×, and 1.13× the design SE — slightly conservative, which is the safe direction to err. What still separates V2 from V3 decomposes into three pieces. (i) Stratification is a precision gain the cluster-robust estimator leaves on the table: guaranteeing balanced coverage of every region×urbanicity cell removes a whole dimension of sampling noise, so V3 sits just below V2. (ii) Degrees of freedom differ — fixest uses $n_{\text{cluster}}-1 = 262$, the survey estimator uses design df = 134, and the naive regimes pretend they have 38,232. (iii) Random domain size: for subgroup means the survey machinery keeps a component that an intercept-only regression within the subgroup drops.⁵

The same story reads off the plain overall prevalence. Its design effect is a substantial 2.41 against a simple random sample of the same size:

Notice that not every naive SE errs downward. For the highest-income group the design SE is actually smaller than the naive iid SE — there, stratification’s precision gain outweighs clustering’s penalty.⁶ The design does not merely inflate; it corrects in whichever direction the sampling scheme demands.

5Effective sample size across subgroups

Clustering’s deepest practical consequence is that a big survey buys a much smaller effective sample. Positively correlated observations within a PSU carry less information than their headcount suggests, so the design-effective $n$ is $n/\text{DEFF}$. Across the 18 income×age prevalence cells, design effects run from 1.22 to 2.51 (median 1.67).

This is why some income×age contrasts are noisier than their raw counts promise, and why power calculations for subgroup comparisons should use $n/\text{DEFF}$, not $n$. It also produces an honest wrinkle worth reporting rather than hiding: in 8 of the fine age×income cells the cluster-robust SE actually comes in below the naive one — a near-zero or slightly negative within-PSU correlation at that granularity. The design effect is an average tendency, not a law that binds every cell.

6Practical translation

1. Use plain weighted group-bys for point estimates. They are exactly right, fast, and convenient for building tables — the estimate is design-unbiased regardless of the variance machinery.
2. Never attach a naive SE, CI, or significance star to them. Anything inferential must come from the declared design or its replicate weights. Here the naive interval ran 32% too narrow.
3. If you live in fixest, weight and cluster at the PSU. That recovers the SE to within a few percent (here 1.05–1.13× the design) — erring slightly conservative. Cluster on a globally-unique psu_uid, never raw SDMVPSU.
4. Use the full design when stratification or degrees of freedom matter. The between-stratum gain and the design df (134, not 38,232) are exactly what svyglm/survey_mean get right and the shortcuts do not.
5. Always report DEFF and think in effective $n$. A DEFF of 2.41 on the headline prevalence, and up to 2.51 in subgroups, means a “large” survey supports far fewer subgroup comparisons than $n$ suggests.

The one-paragraph version: weights answer “who does each observation represent?” and so fully determine the estimate; standard errors answer “how would this estimate move across repeated draws of the same design?” and depend on features weights do not encode. Declaring the design once fixes the second question without touching the first.