A sample mean that misses the population mean is carrying two different faults at once. Pick a sampling route, draw a sample, and the gap splits into the bias built into the route and the chance error in this one draw. Raise the sample size and watch which half moves.
Total error of this sample+0.43 pts
Where the gap comes from, in satisfaction points out of 5
Component
Arithmetic
Points
Selection bias
4.00 − 3.60
+0.40
Sampling error
4.03 − 4.00
+0.03
Total error
4.03 − 3.60
+0.43
P population mean, R route expectation, S this sample. The axis is zoomed to 2.4 to 5.0 of the 1 to 5 scale so the three markers separate. Position is being read here, not bar length.
eligible, solid drawn, ringed out of reach, hollow
Population mean 3.60 of 5Route expectation 4.00 of 5Eligible 252 of 400
Coverage bias. Only customers who use the app can ever see the prompt. App use rises with satisfaction here, from 20% of the 1-point band to 92% of the 5-point band.
Sample size n40 customers
Maximum for this route is 252 customers, because that is all it can reach.
Try this
200 repeated draws of 40 from this route
Spread of the 200 draws: 0.14 pts. The triangle marks which of those draws is the one shown alongside. Drawing again highlights a different one and leaves the histogram exactly where it is, because this shape belongs to the route and the size rather than to any single draw.
This draw of 40 reports 4.03 out of 5 against a true population mean of 3.60. Of that +0.43 pts gap, +0.40 pts was settled before anyone was asked anything, the moment this route was chosen. It reaches 252 of the 400. The other +0.03 pts is chance in this draw. Take n higher and the spread of the 200 draws narrows, roughly with the square root of n while n is small next to the 252 this route reaches and faster than that as n closes on the cap, where it reaches zero. The red line does not move at all. More responses buy precision about the wrong population. Read the counts under the dots: this route reaches 4 of the 20 least satisfied customers and 92 of the 100 most satisfied.
One stylised case and one historical case
Stylised. A telecommunications provider measures satisfaction through an in-app prompt and reports 4.4 out of 5 while complaints to the industry ombudsman are rising. The prompt can only reach app users, and it fires after a completed task. Both faults are coverage, and the route above with a 4.40 expectation is that same shape. No named company is being described here.
Historical. The Literary Digest poll of the 1936 United States presidential election collected more than two million replies from lists built on telephone directories, vehicle registrations and its own subscribers, and about a quarter of those contacted replied. The forecast was precise and it named the wrong winner.
The 400 customers are a stylised population written for this widget. Their scores, app-usage rates and reachability rates are invented so the mechanism is visible, and they measure no real organisation. The route expectations are computed exactly from that population, so the bias figure is exact rather than simulated.