Weights Get the Point Estimate Right;
the Design Gets the Uncertainty Right

1The One-Sentence Version

Sampling weights fix who is in the sample; the sampling design — the strata and clusters the sample was drawn through — determines how much a repeated sample would bounce around. Point estimates depend only on the first; standard errors depend on both. Compute a weighted mean with a pandas group-by and again with the full design declared in svy or R's survey package, and the two estimates are numerically identical. The SEs, CIs, and every downstream p-value are not.

This is not a rounding issue or an approximation. It follows directly from what each object is, and the rest of this note unpacks that in three steps: why the estimate can't see the design (§2), why the variance can't ignore it (§3), and why an econometrician should recognize the whole apparatus as clustered standard errors wearing a different hat (§5).

2Why Weights Alone Give the Correct Point Estimate

A sampling weight \(w_i\) is (to first order) the inverse of unit \(i\)'s probability of ending up in the sample, \(w_i = 1/\pi_i\). It answers exactly one question: how many people in the population does this respondent stand in for?1This is the Horvitz–Thompson (1952) logic: weight each sampled unit by the inverse of its inclusion probability and sums become design-unbiased for population totals — no model required.¹ The canonical estimator of a population mean makes everything a weighted sum:

Notice what is not in that formula: strata, PSUs, replicate weights, design effects. The estimator is a pure function of the pairs \((y_i, w_i)\). So any software that multiplies by the weights and divides by their sum — np.average(y, weights=w), a weighted group_by().agg() in Polars, Stata's [pw=w], R's weighted.mean() — produces exactly the same number as sample.estimation.mean() or svymean().

The same equivalence holds for:

Group-by weighted averages (domain estimates). The weighted mean within a subgroup uses only that subgroup's \((y_i, w_i)\) pairs, so a grouped weighted average matches the design-based domain estimate exactly.

Weighted proportions and totals. A proportion is the mean of an indicator; a total is an unnormalized weighted sum. Same logic.

Weighted regression coefficients. A WLS or pseudo-maximum-likelihood fit with probability weights delivers the same \(\hat\beta\) as svyglm / sample.glm.fit(). The design changes only the variance–covariance matrix, never the coefficients.2This is why Stata prints identical coefficients for reg y x [pw=w] and svy: reg y x — and different standard errors.²

3Why the Standard Errors Are Wrong Without the Design

A standard error answers a counterfactual question: if we redrew the sample using the same procedure, how much would the estimate move? That question cannot be answered without knowing the procedure. Three features of real surveys change the answer, and weight-only software knows about none of them.

3.1 Clustering (PSUs) inflates variance

Surveys don't sample 3,000 random individuals scattered across the country — that would require sending interviewers to 3,000 places. Instead they sample, say, 100 counties (primary sampling units, PSUs), then households within counties. People in the same county share local labor markets, local politics, local prices, and often the same interviewer. Their responses are positively correlated.

Positively correlated observations carry less information than their headcount suggests. Thirty respondents from one county are not thirty independent draws; they might be worth ten. The effective sample size is approximately

where \(\bar m\) is the average cluster size and \(\rho\) is the intra-cluster correlation. The denominator is the design effect (DEFF) from clustering — and it is exactly the Moulton factor from econometrics (§5).3Moulton (1990) showed the identical formula governs regressions where an aggregate regressor is shared by all members of a group — the same disease under a model-based diagnosis.³ With \(\bar m = 30\) and a modest \(\rho = 0.05\), DEFF \(\approx 2.45\): naive SEs are too small by a factor of \(\sqrt{2.45}\approx 1.57\), and your “significant” \(t = 2.5\) is really \(t = 1.6\).

The unit of independence is the PSU, not the person. Design-based variance estimation (Taylor linearization) operationalizes this literally: it collapses observations to weighted PSU totals and computes variance from the between-PSU variation within each stratum. The number of PSUs — not the number of respondents — drives your degrees of freedom.4NHANES has ~10,000 respondents per cycle but roughly 15–17 design degrees of freedom (df = #PSUs − #strata). Subgroup inference there is far more fragile than the headline n suggests.⁴

3.2 Stratification reduces variance — and naive SEs miss the gain

Stratification guarantees the sample covers every region-by-urbanicity cell in fixed proportions. Whole dimensions of sampling noise — what if we'd randomly drawn too few rural Southerners? — are eliminated by construction. Variance comes only from within-stratum variation. Ignoring strata makes SEs conservative on this margin: you'd be leaving precision on the table and understating your own significance.

3.3 Weight variability inflates variance

Unequal weights are a form of noise themselves: an observation with \(w_i = 8\) drags the estimate around eight times as hard as one with \(w_i = 1\). Kish's approximation gives \(\text{DEFF}_{\text{wgt}} \approx 1 + \mathrm{CV}(w)^2\).5Kish (1965). This is also why frequency weights and probability weights give identical means but very different SEs: an fweight of 8 means eight real observations; a pweight of 8 means one observation shouting eight times as loudly.⁵ Software that uses weights for the point estimate but then computes an SE as if it had \(n\) equally-weighted observations gets this margin wrong too.

3.4 The net effect

In practice DEFF for the major surveys — NHANES, DHS, GSS, ANES — runs 1.5–4, occasionally higher for spatially clustered outcomes. Naive SEs are typically too small by 20–100%, and every hypothesis test, confidence interval, and “statistically significant subgroup difference” built on them is anti-conservative.

4A Minimal Demonstration

import numpy as np, polars as pl, svy

df = svy.read_parquet("survey.parquet")

# (a) Naive weighted mean — CORRECT point estimate, WRONG uncertainty
est_naive = np.average(df["income"], weights=df["wt"])

# (b) Full design — SAME point estimate, correct SE / CI / DEFF
design = svy.Design(stratum="stratum", psu="psu", wgt="wt")
sample = svy.Sample(df, design=design)
res    = sample.estimation.mean(y="income", deff=True)

# est_naive == res estimate, to machine precision.
# A naive SE computed from weighted residuals will generally be
# smaller than res's SE by a factor of ~sqrt(DEFF).

# Group-by version: identical logic. Weighted group means match a
# Polars group_by weighted average exactly; only the SEs differ.
by_edu = sample.estimation.mean(y="income", by="education", deff=True)

The same holds in regression: sample.glm.fit(y=..., x=[...]) reproduces the probability-weighted GLM coefficients exactly and replaces only the variance matrix with a design-based one — linearized, clustered on PSUs within strata.

5The Punchline: This Is Clustered Standard Errors

If you do applied micro, you already believe everything in §3 — you just call it something else.

Both machineries answer the same question — what is the true unit of independent information? — and both compute variance from between-group variation of group-level aggregates rather than pretending each row is an independent draw. Concretely, a design-based regression SE is essentially what you get from:

import pyfixest as pf

# An econometrician's rendering of a survey design:
m = pf.feols(
    "y ~ x1 + x2",
    data=df,
    weights="wt",               # pweights → same coefficients as svyglm
    vcov={"CRV1": "psu"},       # cluster at the PSU = unit of sampling
)

This recovers the clustering part of the design variance; the full survey machinery additionally nets out the between-stratum component (a precision gain) and uses the design-correct degrees of freedom. For most PSU-heavy surveys, weights + cluster(psu) gets you 90% of the way; svy / survey / svyset gets you exactly there.6The stratum refinement matters most when strata are strongly predictive of the outcome or when there are few PSUs per stratum, where the df correction bites.⁶

Two familiar econometric lessons transfer directly:

Cluster at the level of sampling variation. In a survey, “assignment to the sample” happened at the PSU level, so the PSU is the mandatory clustering level. Using heteroskedasticity-robust SEs on individual rows is the survey-world equivalent of running a state-level DiD with HC1 errors only — the classic Bertrand–Duflo–Mullainathan mistake.7Bertrand, Duflo & Mullainathan (2004), “How Much Should We Trust Differences-in-Differences Estimates?” — serial and within-group correlation collapses the effective number of experiments to the number of groups.⁷

Few clusters → trouble. Just as fewer than 30–40 clusters calls for wild-cluster bootstrap or CRV3 in pyfixest/fixest, surveys with few PSUs rely on t-distributions with df = #PSUs − #strata, and single-PSU strata must be handled explicitly (sample.singleton.handle(...); options(survey.lonely.psu=...) in R). Same disease, same medicine.

Why the two literatures feel different

Survey statistics starts from design-based inference: the \(y_i\) are fixed population values, randomness comes purely from which units were drawn, and the “cluster correlation” is mechanically induced by drawing whole PSUs at once. Econometrics starts from model-based inference: randomness lives in the error term, and cluster correlation reflects shared shocks. But the variance formulas converge — the CRV1 sandwich estimator and Taylor linearization at the PSU level are algebraically near-identical. One framework's PSU is the other's cluster.

6Practical Rules

1. Use plain weighted group-bys freely for exploration and point estimates. They are exactly right, fast, and convenient for building tables.
2. Never attach a naive SE, CI, or significance star to them. Anything inferential must come from the declared design — svy.Design(stratum=..., psu=..., wgt=...), svyset, as_survey_design — or its replicate weights.
3. Declare the design first, subset later (via where= or filter_records, never by filtering the raw DataFrame). Subsetting changes which PSUs and strata are “live,” and the variance math needs to know.
4. Report DEFF for headline estimates. It is the honest summary of how far your data sits from an SRS of the same size, and referees increasingly expect it.
5. Think in effective sample size. A “big” survey with heavy clustering may support far fewer subgroup comparisons than \(n\) suggests; power calculations should use \(n/\text{DEFF}\), not \(n\).
6. If a survey ships replicate weights (ACS, CPS, many restricted files), use them with the producer-documented method and df. They are the survey world's pre-packaged cluster bootstrap — and they let producers withhold confidential strata/PSU identifiers while you still get correct variances.

7One-Paragraph Summary

Weights answer “who does each observation represent?” and therefore fully determine the point estimate — which is why a weighted average in pandas matches the full-design estimate to the last decimal. Standard errors answer “how would this estimate vary across repeated draws of the same design?” — and that depends on features weights don't encode: whole clusters entered or exited the sample together (variance up), strata guaranteed balanced coverage (variance down), and unequal weights added noise of their own (variance up). Ignoring the design is the survey-statistics twin of running a clustered-treatment regression with robust-only standard errors: the coefficient is fine, the t-statistic is fiction. Declare the design once, and both problems disappear at the same time.