Key takeaway: The significance level, written alpha, is the cutoff chosen before a test: if the p-value is at or below alpha, reject the null hypothesis, and alpha is also the probability of a Type I error.
Grade 12 – AP StatisticsAP Statistics Unit 3Statistics and Data
Definition of Significance Level
The significance level, alpha (\(\alpha\)), sets how much evidence is demanded before the null hypothesis is rejected. Common choices are 0.10, 0.05 and 0.01. A smaller alpha is stricter: it requires a more surprising result.
It is chosen in advance, based on the costs of being wrong. When the null hypothesis is actually true, a test run at level \(\alpha\) will wrongly reject it in about \(\alpha\) of all samples, so alpha is the Type I error rate.
Formula and notation
The decision rule compares the p-value with alpha:
\[ \text{p-value}\le\alpha\ \Rightarrow\ \text{reject }H_0,\qquad \text{p-value}\gt\alpha\ \Rightarrow\ \text{fail to reject }H_0. \]
For a two-sided z-test at \(\alpha=0.05\), each tail holds 0.025, so the critical values are \(\pm 1.96\).
Key facts
- Alpha must be set before looking at the data, otherwise the decision can be rigged.
- Lowering alpha reduces Type I errors but makes Type II errors more likely.
- Statistically significant means the p-value is at most alpha; it does not mean important.
- A test at alpha = 0.05 matches a 95% confidence interval for two-sided tests.
Significance Level example
A two-sided test gives a p-value of 0.043. Decide at the 0.05 level and again at the 0.01 level.
-
At \(\alpha=0.05\): \(0.043\le 0.05\), so the result is significant.
-
At \(\alpha=0.01\): \(0.043\gt 0.01\), so the result is not significant.
-
The data are identical; only the strictness of the standard changed.
Reject \(H_0\) at \(\alpha=0.05\), but fail to reject \(H_0\) at \(\alpha=0.01\).
Common mistake
Watch out
After seeing a p-value of 0.043, a researcher picks alpha = 0.05 “because it is just above” and calls the result significant.
Why it is wrong
Alpha belongs to the planning stage. Choosing it after seeing the p-value lets the researcher steer the conclusion. Fix the significance level from the stakes of the decision first, then compare, and report the actual p-value as well.
Related terms and lessons
- P-Value
- Hypothesis Testing
- Type I and Type II Errors
- Null and Alternative Hypotheses
- Confidence Interval
Browse the glossary: A–Z list · AP Statistics vocabulary · Statistics and Data