Elementary Statistics—More Information about Significance Testing

In some sense I want you to become “bored” with the material, as this means you completely understand the bigger picture (with confidence intervals the bigger picture was estimate multiplier (standard error of the estimate)).

In significance testing, the bigger picture is 1) Determine the type of situation (e.g., one- or two-sample or paired-data) and define the appropriate hypotheses; 2) Decide whether a z-test or t-test is appropriate (and check conditions of the appropriate test); 3) Calculate the appropriate test statistic (in all cases this is some sort of standardization); 4) Determine the p-value (remember this depends on the alternative hypothesis); and 5) Define and interpret the p-value in the context of the problem and provide a conclusion (which might depend on a given significance level, ). And it’s important to draw and carefully label a picture (a normal curve or t-distribution curve) as part of the solution process.

Furthermore, there is a direct relationship between confidence intervals and two-sided significance tests, which is included in the summaries below. And there is an important issue called practical significance that should be addressed.

Relationship between Confidence Interval and Significance Test for a Single Population Mean:

A level , two-sided significance test rejects the hypothesis exactly when the value falls outside the confidence interval for. (Put another way, the significance test does not reject the hypothesis if the value falls inside the corresponding confidence interval.) This holds whether the population standard deviation is known (and a z-test is used) or if the population standard deviation is not know (and a t-test is used).

More Explanation of the Relationship between Confidence Interval and Significance Test

For a more mathematical explanation of the previous result, consider a two-sided test using significance level (assuming the population standard deviation is known). Then the “acceptance region” (really the “do-not-reject region”) for the test statistic is between -1.96 and 1.96. Then,

which says the value of is inside the 95% confidence interval.

Relationship between Confidence Interval and Significance Test for a Difference in Population Means:

A level , two-sided significance test rejects the hypothesis exactly when the value 0 falls outside the confidence interval for. (Put another way, the significance test does not reject the hypothesis if the value 0 falls inside the corresponding confidence interval.)

Practical Significance versus Statistical Significance

It’s possible for test results to be statistically significant, yet not practically significant. For example, you might find a statistically significant difference in means (i.e., you can reject the null hypothesis that the means are the same), yet in the context for the problem the difference might not be practically important. (For example, perhaps you find a significant difference in average decrease in cholesterol for patients taking a drug versus patients taking a placebo, but the magnitude of the difference in only 5 mg/dL. Doctors probably won’t find this practically important—certainly not important enough to put their patients on that drug.)

Hence, if you find statistically significant test results, it’s a good idea to accompany your results with a corresponding confidence interval (to assess the practical significance—but realize it’s an expert in the field, not necessarily a statistician, who should assess the practical importance).