Statistical note

Effect sizes and confidence intervals

Moving from binary significance decisions to estimates of magnitude, direction, precision, and scientific relevance.

26 July 20265 min read

A statistical result should communicate how large an effect may be and how precisely it has been estimated. Effect sizes and confidence intervals provide that information more directly than a significance label.

Point estimates and confidence intervals

Choose a meaningful scale

An absolute mean difference preserves the measurement unit. A ratio can be useful for multiplicative effects. Standardized differences such as Cohen’s d aid comparison across scales but can hide what the change means physically or biologically.

For two groups with pooled standard deviation $s_p$ ,

$d=\frac{\bar{x}_1-\bar{x}_2}{s_p}.$

import numpy as np
from scipy import stats

a = np.array([10, 11, 9, 12, 10], dtype=float)
b = np.array([13, 12, 14, 11, 15], dtype=float)
pooled_sd = np.sqrt(((a.size-1)*a.var(ddof=1) + (b.size-1)*b.var(ddof=1)) / (a.size+b.size-2))
cohens_d = (b.mean() - a.mean()) / pooled_sd
ci = stats.ttest_ind(b, a, equal_var=False).confidence_interval()
print(cohens_d, ci)

Read an interval carefully

A confidence interval gives a range of parameter values compatible with the data and model under repeated-sampling logic. It is not a probability statement that the fixed true value lies inside this particular realized interval.

Precision is not importance

A narrow interval around a negligible effect is precise but may not be scientifically important. A wide interval spanning meaningful benefit and harm signals insufficient information, even when the p-value crosses an arbitrary threshold.

Define practical relevance

Whenever possible, specify a smallest effect of interest before examining the result. Interpretation can then focus on whether the interval excludes trivial effects, meaningful effects, or neither.

Case study: segmentation-induced area bias

A two-pixel mean boundary bias may be statistically detectable across thousands of cells but negligible for a downstream phenotype. Define tolerance in physical units, summarize effects per independent experiment, and compare the interval with that tolerance.