An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Data Foundationsintermediate

Sampling bias

Q: A retailer increases its post-purchase survey from 2,000 to 200,000 responses. What genuinely improves?

The precision of the estimate for the population of people who purchase and choose to respond. A hundredfold increase in responses cuts the standard error by a factor of ten for the population actually sampled, which is purchasers who choose to respond. It does nothing about customers who did not purchase or did not respond, so representativeness is unchanged.

Q: A lender builds a credit model using only the accounts it approved in the past and reports excellent predictive accuracy. What is the main threat to using it on new applicants?

Rejected applicants never appear, so the model has never seen the outcomes it most needs to predict. This is selection bias in its most expensive form. The training data contains only cases that passed the existing filter, so the model learns to rank the already-approved and cannot be validated against the population it will actually be applied to.

Sampling bias is a systematic gap between who is in the data and who the decision is about. It is not the same thing as random sampling error, and the difference is decisive: random error shrinks as the sample grows, while bias does not shrink at all. The common forms are coverage bias, where the list you sample from excludes part of the population, non-response bias, where those who answer differ from those who do not, self-selection, where people opt in for reasons related to the outcome, and survivorship bias, where the cases that failed are missing from the file altogether.

Why it matters

Ask a hundred people leaving a concert whether they enjoyed the band and you get a glowing review. Ask a hundred thousand and you get the same glowing review, measured more precisely. The people who walked out after two songs are in neither sample. Adding observations makes you more confident about the wrong population. So the first question about any dataset is not how big it is. It is who is missing from it and why.

Before you read on — recall

A retailer increases its post-purchase survey from 2,000 to 200,000 responses. What genuinely improves?

Formulas

Random error shrinks with sample size

SE(\hat{p}) = \sqrt{\dfrac{p(1-p)}{n}}

For a proportion, the standard error falls with the square root of

n

. Going from 1,000 to 100,000 responses cuts it by a factor of ten, from about 1.6 percentage points to about 0.16 when

p

sits near one half. Nothing in this expression refers to who was left out, which is precisely the point.

Non-response bias does not shrink

p = w\,p_R + (1-w)\,p_N

The true rate

p

mixes respondents and non-respondents, where

w

is the response rate. If 30 per cent respond and 80 per cent of them are satisfied, while only 40 per cent of non-respondents are, the true rate is about 52 per cent. Reporting the respondent figure of 80 per cent overstates satisfaction by 28 points, whether you surveyed a thousand people or a million.

Worked examples

Scenario

A telecommunications provider measures satisfaction through in-app feedback prompts and reports 4.4 out of 5. Complaints to the industry ombudsman are rising over the same period.

Solution

The prompt appears in the app, so customers who cannot make the app work, or who have already left, never see it. It also fires after a completed task, which selects for interactions that went well. Both are coverage problems and neither is fixed by collecting more responses. A defensible measure needs a sample drawn from the customer list rather than from app sessions, a real effort to reach non-responders, and a comparison of responder characteristics against the customer base so the report can state what the sample over-represents.

Scenario

A magazine polls its readers before an election, collects more than two million replies, and predicts the wrong winner by a wide margin.

Solution

This is the Literary Digest poll of the 1936 United States presidential election. Ballots went to lists built from telephone directories, vehicle registrations and the magazine's own subscribers, and the returns were reported as the forecast. Two problems compounded. The list over-represented people who could afford a car or a telephone during the Depression, which is coverage bias, and only about a quarter of those contacted replied, which is non-response bias. The enormous sample made the estimate precise and left it wrong, which is the clearest demonstration that size and representativeness are separate properties.

Common mistakes

✗A very large sample removes the need to worry about representativeness. Size reduces random error only, so a biased frame produces a precisely wrong answer, and the added precision makes the error more persuasive rather than less.
✗Sampling bias is a survey problem that does not affect operational data. Transaction, app and sensor records are samples of behaviour too, and they systematically exclude people who did not transact, did not install the app, or were never covered by a sensor.
✗Weighting always fixes bias. Weighting can adjust for characteristics you measured and can compare against a known benchmark, it cannot adjust for a difference you never observed, and heavy weights inflate variance.
✗If the sample was drawn randomly then bias cannot occur. Random selection from an incomplete list inherits the gaps in that list, and non-response after selection reintroduces bias no matter how the selection was made.

Revision bullets

•Random error shrinks with the square root of sample size; bias does not shrink at all
•Coverage bias: the frame you sample from excludes part of the population
•Non-response bias: those who answer differ from those who do not
•Self-selection and survivorship remove cases for reasons tied to the outcome
•Operational, app and sensor data are samples of behaviour and carry the same problem
•Weighting adjusts only for differences you actually measured

Quick check

A retailer increases its post-purchase survey from 2,000 to 200,000 responses. What genuinely improves?

A lender builds a credit model using only the accounts it approved in the past and reports excellent predictive accuracy. What is the main threat to using it on new applicants?

Connected topics

Correlation vs Cause What Big Data Means Fitness for Use Data Cleansing A/B Testing Network Study Social Data Sources Algorithmic Bias

More in Data Foundations

Forms of Data Data Warehouse ETL and ELT Metadata Star Schema Measures & Dimensions Cloud for Analytics Quality Dimensions

Sources

Squire (1988)
Squire, P. "Why the 1936 Literary Digest Poll Failed." Public Opinion Quarterly, 52(1), 125-133, 1988.
Separates the two causes of the failure, showing that non-response mattered alongside the well-known coverage problem in the mailing list.
Meng (2018)
Meng, X.-L. "Statistical Paradises and Paradoxes in Big Data (I): Law of Large Populations, Big Data Paradox, and the 2016 US Presidential Election." Annals of Applied Statistics, 12(2), 685-726, 2018.
Shows formally that a small amount of selection bias overwhelms the benefit of a very large sample, and quantifies the effective sample size that remains.
Bradley et al. (2021)
Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S. "Unrepresentative Big Surveys Significantly Overestimated US Vaccine Uptake." Nature, 600, 695-700, 2021.
A worked demonstration that very large non-probability surveys were badly wrong while a much smaller probability-based survey was close to the benchmark.

How to cite this page

Dr. Phil's Quant Lab. (2026). Sampling bias. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-sampling-bias