Statistical note

P-values, false positives, and multiple testing

What a p-value means, common misinterpretations, and why image-rich experiments need multiplicity-aware analysis.

22 July 20266 min read

A p-value is the probability, under a specified null model, of obtaining a result at least as incompatible with that model as the observed result. It is not the probability that the null hypothesis is true.

False positives accumulating across many tests

What a small p-value does not tell us

It does not measure effect size, biological importance, reproducibility, or the probability that a finding occurred by chance. It is conditional on the model, assumptions, and analysis choices.

Multiplicity grows quickly

Testing many features, time points, channels, regions, or model variants increases the opportunity for false positives. Imaging pipelines can generate thousands of candidate comparisons without making that multiplicity visually obvious.

For $m$ independent true null hypotheses tested at level $\alpha$ , the probability of at least one false rejection is

$1-(1-\alpha)^m.$

import numpy as np
from scipy import stats

p = np.array([0.001, 0.009, 0.021, 0.08, 0.42])
adjusted = stats.false_discovery_control(p, method="bh")
print(np.c_[p, adjusted])

Common controls

Bonferroni-style procedures control the probability of at least one false positive but can be conservative. False-discovery-rate procedures such as Benjamini–Hochberg control the expected proportion of false discoveries among rejected hypotheses under stated conditions.

Better practice

Predefine primary outcomes, distinguish exploratory from confirmatory analyses, report all tested comparisons, use appropriate corrections, and emphasize effect estimates with uncertainty.

Case study: high-content microscopy

If 200 morphology features are tested across treatments, report the complete feature family and control its false-discovery rate. Splitting one experiment into many unreported analysis branches makes the effective multiplicity larger than the final table suggests.