Math 207—Summary of Large-Sample Significance Tests

 

As I mentioned when we discussed confidence intervals, in some sense I want you to become “bored” with the material, as this means you completely understand the bigger picture (with confidence intervals the bigger picture was  estimate  multiplier (standard error of the estimate)).

 

In significance testing, the bigger picture is 1) Determine the type of situation (e.g., one- or two-sample, mean or proportion) and define the appropriate hypotheses; 2) Decide whether a large-sample test is appropriate; 3) Calculate the appropriate test statistic (in the large-sample cases the test statistic is always a z-value); 4) Determine the p-value (remember this depends on the alternative hypothesis); and 5) Define and interpret the p-value in the context of the problem and provide a conclusion (which might depend on a given significance level, ). And it’s important to draw and carefully label a picture (a normal curve) as part of the solution process.

 

Furthermore, there is a direct relationship between confidence intervals and two-sided significance tests, which is included in the summaries below. And there is an important issue called practical significance that should be addressed.

 

Important Note: The textbook describes a “critical-value approach” to significance testing. This method is out-dated and limits your conclusion to a certain significance level. The p-value approach allows you to make conclusions for any significance level (and it provides more useful information). You must know the p-value approach.

 

Practical Significance versus Statistical Significance

It’s possible for test results to be statistically significant, yet not practically significant. For example, you might find a statistically significant difference in means (i.e., you can reject the null hypothesis that the means are the same), yet in the context for the problem the difference might not be practically important. (For example, perhaps you find a significant difference in average decrease in cholesterol for patients taking a drug versus patients taking a placebo, but the magnitude of the difference in only 5 mg/dL. Doctors probably won’t find this practically important—certainly not important enough to put their patients on that drug.)

 

Hence, if you find statistically significant test results, it’s a good idea to accompany your results with a corresponding confidence interval (to assess the practical significance—but realize it’s an expert in the field, not necessarily a statistician, who should assess the practical importance).

 

Large-Sample Significance Test for a Population Mean

Suppose we have a large (), random sample from a population with unknown mean, . Furthermore, suppose we want to test the null hypothesis, . Then we first calculate the test statistic,   (you can estimate the population standard deviation with the sample standard deviation), which tells us the number of standard errors our particular sample average is from the null-hypothesized population mean. Then by the Central Limit Theorem, we can use the standard normal distribution to determine the approximate p-value (recall the p-value depends on the direction of the alternative hypothesis):

 

 

Finally, we define and interpret the p-value in the context of the problem and provide a conclusion (which might depend on a given significance level, ).

 

Relationship between Confidence Interval and Significance Test:

A level , two-sided significance test rejects the hypothesis   exactly when the value  falls outside the   confidence interval for. (Put another way, the significance test does not reject the hypothesis if the value  falls inside the corresponding confidence interval.)

More Explanation on the Relationship between Confidence Interval and Significance Test

For a more mathematical explanation of the previous result, consider a two-sided test using significance level . Then the “acceptance region” (really the “do-not-reject region”) for the test statistic is between -1.96 and 1.96. Then (via simple algebra),

 

which says the value of  is inside the 95% confidence interval.

 

Large-Sample Significance Test for a Population Proportion

The textbook discusses a large-sample test for a population proportion. We do not often use this test in practice, because it’s uncommon to have a situation where there is a precise value of p we want to test. Hence, when doing inference about a single population proportion, we’ll stick with a confidence interval. (That is, you can skip textbook section 9.5, with the exception of practical significance, which you need to know.)

 

Large-Sample Significance Test for a Difference in Population Means

Suppose we have two distinct populations with unknown means,  and . Furthermore, suppose we have large  independent, random samples from each population. Lastly, suppose we want to test the null hypothesis,  (this will always be the default null hypothesis). Then we first calculate the test statistic,

 

  (subtracting 0 clearly does nothing to the numerical value, but I include it to emphasize this is a z-value), which tells us the number of standard errors our particular difference in sample averages is from the null-hypothesized difference in population means. Then by the Central Limit Theorem, we can use the standard normal distribution to determine the p-value (recall the p-value depends on the direction of the alternative hypothesis—in the same way described in detail for the one-sample test).

 

 

Finally, we define and interpret the p-value in the context of the problem and provide a conclusion (which might depend on a given significance level, ).

 

Relationship between Confidence Interval and Significance Test:

A level , two-sided significance test rejects the hypothesis   exactly when the value 0 falls outside the   confidence interval for. (Put another way, the significance test does not reject the hypothesis if the value 0 falls inside the corresponding confidence interval.)

 

Large-Sample Significance Test for a Difference in Population Proportions

Suppose we have two distinct populations with unknown “success” proportions,  and . Furthermore, suppose we have large (), independent, random samples from each population. Lastly, suppose we want to test the null hypothesis,  (this will always be the default null hypothesis).

 

Brief Remark (needed to better understand the test statistic)

Unlike the confidence-interval situation, there is information in the null hypothesis we can use to better estimate the standard error of . Recall the standard error of  is . Since all our calculations are done assuming the null hypothesis is true, for our test statistic we can assume  (let’s call this common success probability, p). Then the standard error reduces to . To estimate the unknown value of p , we can pool our information from both samples:  , where  is the number of successes in the first sample and  is the number of successes in the second sample.

 

Back to the significance test…

Then we first calculate the test statistic,  , where  (subtracting 0 clearly does nothing to the numerical value, but I include it to emphasize this is a z-value), which tells us the number of standard errors our particular difference in sample proportions is from the null-hypothesized difference in population proportions. Then by the Central Limit Theorem, we can use the standard normal distribution to determine the p-value (recall the p-value depends on the direction of the alternative hypothesis—in the same way described in detail for the one-sample test).

 

 

Finally, we define and interpret the p-value in the context of the problem and provide a conclusion (which might depend on a given significance level, ).

 

 

Relationship between Confidence Interval and Significance Test:

When comparing population proportions the confidence interval and test statistic have slightly different forms. Hence, there isn’t an exact relationship between the interval and test, but it’s still a closely approximate relationship: A level , two-sided significance test rejects the hypothesis   approximately when the value 0 falls outside the   confidence interval for. (Put another way, the significance test does not reject the hypothesis if the value 0 falls inside the corresponding confidence interval.)