a. Sampling and Generalization
The Literary Digest magazine correctly called every U.S. presidential election from 1916 to 1932 — five in a row.
For 1936 they planned the largest poll ever attempted:
Their final headline: “Landon wins, 57% to 43%.”
Election Day was three days away.
Literary Digest cover, Feb. 19, 1921. Public domain, via Wikimedia Commons.
FDR won 61% to 37% — one of the biggest landslides in U.S. history (46 of 48 states). Counting only the votes for Roosevelt and Landon, as the Digest’s 57–43 did, FDR took 62%. The Digest had him at 43%, a miss of nearly 20 points — and they had the winner wrong.
Meanwhile, an upstart statistician named George Gallup — at a year-old polling institute — had called the election correctly using a sample of just ~50,000, chosen to deliberately match the demographics of the U.S. electorate. The Digest folded in 1938; Gallup helped establish modern polling.
How can 2.4 million responses miss by nearly 20 points while just 50,000 — chosen the right way — get it right?
In hypothesis testing and confidence intervals (topics 03 & 04), we:
Key question: When can we trust these inferences to apply to the broader population?
Answer: It depends on how we selected our sample.
Census: examine every case in the population.
Sample survey: examine a subset.
In practice, we almost always work from samples — so how we choose the sample drives what we can claim.
Generalization: applying findings from a sample to the broader population.
Possible only when our sample is representative — its composition mirrors the population’s mix of relevant characteristics.
Representative sample → we can generalize
Unrepresentative sample → we cannot
The critical factor: how we selected our sample.
Two things went wrong with their sample.
Their list tilted toward the well-off. The Digest drew mostly from telephone directories and car registration records — both skewed toward better-off Americans in 1936, so working-class voters were under-represented.
Only 24% bothered to mail their ballot back. And the responders weren’t a random slice — they were the more politically engaged, and more anti-FDR.
Gallup used quota sampling. Interviewers were given quotas to fill — so many men and women, urban and rural, low- and high-income — chosen to match the population’s demographics.
The sample’s 50,000 voters were not a random draw, but they were balanced against the electorate. (True random sampling came later — and 5b digs into it.)
Bigger doesn’t beat representative.
Several campus surveys are running. For each, decide: can the team generalize to all students at the college?
The student newspaper wants to know how students feel about a controversial campus speaker. They post a poll on the paper’s Instagram and tally responses. No. Only students who follow the paper’s account see the poll, and those who feel strongly self-select to respond. Two layers of bias on top of each other.
A psychology professor studies stress by recruiting subjects from her intro-psych course, where students earn extra credit for participating. No. The sample is limited to students in this professor’s section who chose to participate for credit — a doubly self-selected slice. Generalizing even to “intro-psych students” is shaky, let alone to the college as a whole.
Two more — same question.
The Provost’s office emails a course-evaluation survey to all 3,200 students after finals. About 280 respond (~9%). Probably not. The 9% who responded likely differ from the 91% who didn’t — more engaged, more strongly opinionated. Same nonresponse bias the Digest had.
The college’s institutional research office uses a random number generator to pull 200 students from the registrar’s full enrollment list. They follow up persistently and get responses from 180 (~90%). Yes. Every student had an equal chance of being chosen, and almost everyone selected actually responded — so the sample’s mix mirrors the population’s. (We’ll formalize this in 5b.)
