Elementary Statistics—Sampling Examples

For each of the following scenarios, identify the population, the sample, the variable being measured, and any problems with the sampling process; each scenario contains a source of probable bias—state the reason for the bias and the likely direction of the bias.

1. A congressperson is interested in whether his constituents favor a proposed gun control bill. His staff reports that letters on the bill have been received from 361 constituents and that 323 of these oppose the bill. He then assumes that about (323/361)100 = 89% of his constituents oppose the bill, so he votes against it.

2. A flour company wants to know what fraction of Minneapolis households bakes some or all of their own bread. The company selects a sample of 500 residential addresses in Minneapolis and sends interviewers to these addresses. The interviewers work during regular working hours on weekdays and interview only during those hours.

3. The Miami Police Department wants to know how black residents of Miami feel about police service. A sociologist prepares several questions about the police. A sample of 300 mailing addresses in predominantly black neighborhoods is chosen, and a police officer goes to each address to ask the questions of an adult living there.

4. The National Rifle Association is interested in the opinion of adult Americans on the right to bear arms. They randomly select 5,000 US phone numbers and conduct a phone interview with an adult who answers the phone. The main question is:

Which of these best represents your opinion on gun control?

(1) The government should confiscate our guns.

(2) We have the right to keep and bear arms.

Elementary Statistics—Solutions to Sampling Examples

Remember not all sampling/data collection problems necessarily lead to bias. You need to make the case for bias.

2. A flour company wants to know what fraction of Minneapolis households bakes some or all of their own bread. The company selects a sample of 500 residential addresses in Minneapolis and sends interviewers to these addresses. The interviewers work during regular working hours on weekdays and interview only during those hours.

Population – all Minneapolis households

Sample – the households (out of the 500) from which they get responses

Variable – whether or not a household bakes some or all of their own bread

Potential problems – possibly undercoverage (depending on how the 500 households were selected), nonresponse (since many people will be at work during the interview hours)

Probable bias – the nonresponse will most likely create a bias, since the people who aren’t heard from (those at work or out of the house during the day) probably share something in common that affects the response (i.e., they might not have as much time to bake their own bread). Hence, the sample proportion of bread bakers might be biased high.

Important note: the big issue here is one of nonresponse, not undercoverage; recall that undercoverage is a problem with the process of choosing a sample, and nonresponse is a problem with data collection.

3. The Miami Police Department wants to know how black residents of Miami feel about police service. A sociologist prepares several questions about the police. A sample of 300 mailing addresses in predominantly black neighborhoods is chosen, and a police officer goes to each address to ask the questions of an adult living there.

Population – black residents of Miami

Sample – the adults (out of the 300 mailing addresses) from which they get responses

Variables – feelings about police service

Potential problems – nonresponse, undercoverage (since black residents not living in predominantly black neighborhoods are excluded from the process of selecting a sample), response error (since some respondents may not speak truthfully about their feelings to a police officer)

Probable bias – the response error will most likely create the largest bias. Even if a person has negative feelings about the police, he/she might not report these feelings, since a police officer serves as the interviewer. This would bias the sample toward more positive feelings about the police. It’s also possible for the undercoverage to cause a bias, as black residents living in non-black neighborhoods might have different feelings about police (and these feelings are excluded from the sampling process). Finally, the nonresponse might also cause a bias, as some people might not answer the door, simply because a police officer is outside, and these people might share concerns about the police force (e.g., they might fear the police).

4. The National Rifle Association is interested in the opinion of adult Americans on the right to bear arms. They randomly select 5,000 US phone numbers (excluding residents of Hawaii and Alaska) and conduct a phone interview with an adult who answers the phone. The main question is:

Which of these best represents your opinion on gun control?

(3) The government should confiscate our guns.

(4) We have the right to keep and bear arms.

Population – adult Americans

Sample – the adults (out of the 5000 Americans) from which they get responses

Variables – opinions on the right to bear arms

Potential problems – nonresponse, undercoverage (since residents without phones are excluded from the process of selecting a sample), wording of the question (since the question is written to favor answer 2)

Probable bias – it’s possible that nonresponse and undercoverage might cause bias (since the people who don’t respond or the people without phones might share a certain stance on gun control), but the biggest source of bias is probably the wording of the question. Even people who support gun control might select answer 2 (because of the limited options given and the phrasing of the question).