Chapter 10 · Statistics: samples and populations

Comparing Two Populations: Inferences and Error Checks

Use the means and spreads of two random samples to judge whether two populations differ in a meaningful way, and learn to find and fix mistakes in comparisons.

20 min read

Are seventh graders in a school district generally taller than fourth graders? Does one variety of tomato generally weigh more than another? You cannot measure every student or every tomato, so you measure some of them and reason from there.

These terms carry the lesson:

A population is the whole group you want to know about.
A sample is the part of the population you actually measure.
In a random sample, every member of the population has the same chance of being chosen. This makes the sample likely to be representative: its values are much like those of the whole population.
An inference is a conclusion about a population that you draw from a sample. A comparative inference says how two populations compare.

An inference about a population is valid only when it comes from a random, representative sample. If a sample is not random, you cannot rely on it to tell you about the whole population.

Even a random sample has a limit. A number from a sample, such as its mean, is only an estimate of the population's value. A different random sample would usually give a slightly different estimate.

So every conclusion in this lesson stays informal and approximate. You say what the samples suggest, not what they prove.

From each sample you need two numbers.

The mean is a measure of center, one number that stands for a typical value. Add the values and divide by how many there are.
The mean absolute deviation (MAD) is a measure of variability, one number that tells how spread out the values are. It is the average distance of the values from their mean.

The gap between two sample means is the larger mean minus the smaller mean. A gap means little by itself. The same gap can be large for one pair of samples and small for another, depending on how spread out the values are.

So you use the MAD as a measuring unit. You already know how to write the gap between two centers as a multiple of a measure of variability. The multiple answers one question: how many MADs fit in the gap? The table below shows two pairs of samples with the same gap and different MADs.

Same gap between the means, different spread

Samples	Sample means	MAD of each sample	Gap between the means	Gap as a multiple of the MAD
Pair 1	$50$ and $56$	$2$	$56 - 50 = 6$	$6 \div 2 = 3$
Pair 2	$50$ and $56$	$12$	$56 - 50 = 6$	$6 \div 12 = 0.5$

The last column tells you about overlap. A distribution is the way the values of a data set are spread out: where the center is and how much the values vary. Two distributions overlap when their values are mixed together over the same stretch of numbers.

Pair 1: the gap is $3$ MADs. The means are $6$ apart, but a value is, on average, only $2$ away from its own mean. The two sets of values sit mostly apart.
Pair 2: the gap is $0.5$ MAD. The means are $6$ apart, but a value is, on average, $12$ away from its own mean. The two sets of values are usually well mixed.

The multiple is a ratio of the gap to the MAD. Both are measured in the unit of the data, so the unit divides out and the multiple has no unit.

Key concept

Measure the gap between the means in MADs

Use this only with two random, representative samples whose MADs are similar and greater than $0$ .

Subtract the smaller sample mean from the larger, so the gap is not negative. Then divide the gap by the MAD. If the MADs are close but not equal, use the larger one.

As a guideline:

A multiple of $2$ or more means the samples are noticeably separated. They suggest a meaningful difference: values in one population are generally greater than values in the other.
A multiple less than $2$ means the samples overlap a lot. They do not show a meaningful difference.

A multiple close to $2$ is a borderline case. A different pair of samples would usually give slightly different numbers, so every conclusion stays informal and approximate.

\text{multiple} = \dfrac{\text{larger mean} - \text{smaller mean}}{\text{MAD}}

Worked example

A gap that is large compared with the spread

A random sample of $40$ fourth graders and a random sample of $40$ seventh graders are drawn from all the students in those grades in a school district. Each student's height is measured in centimeters (cm). Fourth-grade sample: mean $136$ cm, MAD $6$ cm. Seventh-grade sample: mean $154$ cm, MAD $6$ cm. What can you infer about the heights of fourth and seventh graders in the district?

Check the samples first, because an inference is valid only from random, representative samples. Each sample was drawn at random from all the students in its grade, so it is likely to be representative of that grade.
Check the spreads, because one MAD has to act as the measuring unit for both samples. Both MADs are 6 cm, so the spreads are similar. The MAD is greater than 0, so you can divide by it.
Find the gap between the means. Subtract the smaller mean from the larger so the gap is positive. Both means are in centimeters, so the gap is 18 cm.
$154 - 136 = 18$
Measure the gap in MADs. Dividing the gap by the MAD tells you how many MADs fit in the gap. Centimeters divided by centimeters leaves a number with no unit.
$18 \div 6 = 3$
Compare the multiple with the guideline. A multiple of 3 is more than 2, so the two samples are noticeably separated.
$3 > 2$
State the inference in careful words. Say what the samples suggest about the two grades in general, and keep the size of the difference approximate, because a different pair of random samples would usually give slightly different means.

Answer

The samples suggest a meaningful difference: seventh graders in the district are generally taller than fourth graders, by roughly $18$ cm on average.

Worked example

A gap that is small compared with the spread

A grower has one field of variety R tomatoes and one field of variety S tomatoes. She picks a random sample of $20$ tomatoes from each field and finds the mass of each tomato in grams. Variety R sample: mean $142$ grams, MAD $10$ grams. Variety S sample: mean $148$ grams, MAD $12$ grams. Do the samples show that the variety S tomatoes in her field are generally heavier than the variety R tomatoes?

Check the samples. Each was picked at random from its whole field, so each is likely to be representative of that field. The MADs, 10 grams and 12 grams, are similar and greater than 0, so you can measure the gap in MADs.
Find the gap between the means. Subtract the smaller mean from the larger so the gap is positive. The gap is 6 grams.
$148 - 142 = 6$
Choose the measuring unit. The MADs are close but not equal, so use the larger one, 12 grams. Dividing a positive gap by a larger number gives a smaller multiple, so this choice does not make the difference look bigger than it is.
$6 \div 12 = 0.5$
Test whether the choice matters. With the smaller MAD, 10 grams, the multiple is 0.6. Both multiples are well below 2, so the conclusion does not depend on which MAD you use.
$6 \div 10 = 0.6$
Interpret the multiple. The gap is only about half of the average distance of a tomato's mass from its own sample mean, so the two samples overlap a lot. Sample means also change from sample to sample, so a gap this small is weak evidence of a difference.
$0.5 < 2$

Answer

No. The samples do not show a meaningful difference between the masses of the tomatoes in the two fields. This does not prove that the two varieties weigh the same. It means these samples do not give good evidence that variety S tomatoes are generally heavier.

Checking someone else's comparison is called error analysis: you find the mistake, explain why it is a mistake, and correct it. Look in four places, in this order:

Samples. Is each sample random and representative of its population? Are the two MADs similar? If the samples are not random, no inference about the populations is valid, however good the arithmetic is.
Setup. Are the two sample means compared, and is the gap divided by the MAD? Dividing the MAD by the gap, or subtracting the MADs, answers a different question.
Arithmetic. Are the subtraction and the division correct?
Conclusion. Does it match the multiple, and does it stay informal? Words like suggest, generally and about fit a sample. Words like every, exactly and prove claim more than samples can show, because distributions overlap and estimates change from sample to sample.

Worked example

Find and fix two mistakes

A city draws a random sample of $30$ trips from all of last month's trips on bus Route P, and another random sample of $30$ trips from all of last month's trips on Route Q. Route P sample: mean $34$ minutes, MAD $4$ minutes. Route Q sample: mean $48$ minutes, MAD $4$ minutes. Leo writes: " $4 \div 14 \approx 0.29$ , which is less than $2$ . So trips on both routes take exactly the same time." Find and correct Leo's mistakes.

Check the samples. Each is a random sample from all of last month's trips on its route, so each is likely to be representative of that route. Both MADs are 4 minutes. The samples can support an inference, so the mistake is not here.
Check the setup. The gap between the means is 14 minutes. Leo's arithmetic is correct, but he divided the MAD by the gap. That tells you what fraction of the gap one MAD is, not how many MADs fit in the gap. This is his first mistake.
$48 - 34 = 14$
Correct the setup. Divide the gap by the MAD, so that the MAD is the measuring unit. Check the arithmetic: 4 times 3.5 is 14.
$14 \div 4 = 3.5$
Compare the corrected multiple with the guideline. A multiple of 3.5 is more than 2, so the samples are noticeably separated and suggest a meaningful difference. Leo's setup hid this difference.
$3.5 > 2$
Check the wording. Even a small multiple would not show that the times are exactly the same, because sample means change from sample to sample. This is his second mistake. The corrected conclusion says what the samples suggest.

Answer

Leo divided the wrong way and claimed too much. The correct multiple is $14 \div 4 = 3.5$ , so the samples suggest that trips on Route Q generally took longer than trips on Route P last month, by about $14$ minutes on average.

Three more conclusions to correct. Rows 2 and 3 use the height samples from the first worked example.

What someone wrote	Where it goes wrong	A sounder conclusion
School A's basketball team has a greater mean height than a random sample from School B, so School A is taller.	Samples: a basketball team is not a random sample of its school, so it cannot stand for the school.	No comparison of the two schools is valid until both samples are random.
The gap is $3$ MADs, so every seventh grader is taller than every fourth grader.	Conclusion: some overlap can remain. A tall fourth grader can be taller than a short seventh grader.	Seventh graders are generally taller than fourth graders.
The sample means differ by $18$ cm, so the population means differ by exactly $18$ cm.	Conclusion: sample means change from sample to sample, so $18$ cm is an estimate, not an exact value.	The population means differ by about $18$ cm.

The big idea. To compare two populations, start from random, representative samples with similar MADs. Find the gap between the sample means and measure it in MADs. As a guideline, a gap of $2$ MADs or more suggests a meaningful difference, and a smaller gap means the samples overlap too much to show one.

To check someone else's comparison, test the samples, the setup, the arithmetic and the conclusion, in that order. Keep every conclusion informal and approximate, because a different sample would usually give slightly different numbers.

Up next

Likelihood from 0 to 1 and Sample Spaces

13 min read

Continue