Type I and Type II errors: the two ways a test can be wrong

A hypothesis test only has two ways to be wrong: claiming an effect exists when it doesn’t (a Type I error, a false alarm), or failing to detect an effect that genuinely does exist (a Type II error, a missed detection). This is exactly the same false-positive/false-negative trade-off covered in classification metrics — a significance test is a binary decision rule, and it inherits the same trade-offs a detector does.

The four possible outcomes

	$H_0$ actually true	$H_0$ actually false
Test rejects $H_0$	Type I error (false positive), rate $\alpha$	Correct detection, rate $1-\beta$ (power)
Test fails to reject $H_0$	Correct, rate $1-\alpha$	Type II error (false negative), rate $\beta$

$Power=1-\beta=P(\text{reject }H_0\mid H_1\text{ true})$

$\alpha$ (typically set to 0.05 by convention, not by any law of nature) is chosen directly — it’s the tolerance for false alarms the analyst commits to before seeing the data. $\beta$, and therefore power, is not directly chosen; it falls out of the sample size, the effect size, and the noise level, exactly as covered in power analysis.

import numpy as np
from scipy import stats

x = np.array([8.2, 8.7, 9.1, 9.4, 9.8])
estimate = np.mean(x)
se = stats.sem(x)
interval = stats.t.interval(0.95, len(x) - 1, loc=estimate, scale=se)
print(f"estimate={estimate:.2f}  95% CI={interval}")

The trade-off can’t be eliminated, only moved

Making a test stricter (lowering $\alpha$, demanding stronger evidence before declaring significance) reduces false positives but, all else equal, increases false negatives — real effects now need to clear a higher bar to be detected. The only way to reduce both error rates simultaneously is to reduce noise or increase the amount of independent evidence (a larger, better-designed sample) — there is no free adjustment of the threshold alone that improves both at once.

Failing to detect isn’t evidence of absence

A non-significant result can mean the effect genuinely isn’t there, or the study didn’t have enough power to detect it (see power analysis) — those look identical from the p-value alone. A study using too few independent cultures can systematically fail to detect a real morphology shift not because the effect is absent, but because $\beta$ was high from the start given the sample size actually used. Reporting the achieved power (or, better, an effect-size confidence interval) alongside a non-significant result is what distinguishes “probably no effect” from “this study couldn’t have found it either way.”

Takeaways

• Type I (false positive) and Type II (false negative) errors trade off against each other — tightening one loosens the other, for a fixed sample size and noise level.
• $\alpha$ is chosen directly by the analyst; $\beta$ (and power) is a consequence of sample size, effect size, and noise, not an independent knob.
• A non-significant result is ambiguous between “no real effect” and “underpowered to detect it” — check power or report an effect-size interval to tell them apart.
• The only way to lower both error rates together is to reduce noise or collect more independent evidence — adjusting the decision threshold alone only shifts which error you’re more likely to make.

Continue through the concepts

Related notes

Bias, variance, and mean-squared error

Decompose estimation error into systematic displacement and sampling variability.

How to interpret confidence intervals

Use confidence intervals as procedure-based ranges of compatible parameter values, not posterior probabilities.

Estimands and estimators

Define the exact scientific quantity of interest before choosing a test or model.

Law of large numbers

Understand why averages stabilise with increasing independent observations and why dependence can defeat the intuition.