Statistical note

Bootstrap and permutation inference

Resampling distributions, confidence intervals, permutation p-values, exchangeability, and custom image-analysis statistics.

28 June 202610 min read

The bootstrap approximates the sampling distribution of an estimator by repeatedly resampling observed units with replacement. A permutation test approximates a null distribution by rearranging labels in ways justified by exchangeability.

Resamples generating a distribution of estimates

import numpy as np
from scipy import stats

control = np.array([8.1, 8.4, 8.0, 8.7, 8.3])
treated = np.array([9.0, 9.4, 8.8, 9.5, 9.1])

ci = stats.bootstrap((treated,), np.mean, n_resamples=20_000, rng=42)
print(ci.confidence_interval)

def mean_difference(x, y, axis=0):
    return np.mean(x, axis=axis) - np.mean(y, axis=axis)

test = stats.permutation_test(
    (treated, control), mean_difference,
    permutation_type="independent", n_resamples=20_000, rng=42,
)
print(test.statistic, test.pvalue)

Mathematics

The empirical bootstrap distribution $\{\hat\theta^{*(b)}\}_{b=1}^B$ approximates uncertainty in $\hat\theta$ . Monte Carlo error decreases only as $1/\sqrt{B}$ , so reproducible random generators and sufficient resamples matter.

Case study: median cell response

Resample independent experiments or donors, not individual cells pooled across experiments, when biological replicates define sampling uncertainty. Cluster bootstrap methods may be needed for nested images.

Functions: scipy.stats.bootstrap, permutation_test, monte_carlo_test, and power. For bootstrap replicates drawn from the empirical distribution,

$\hat\theta^{*(b)}=T(X_1^{*(b)},\ldots,X_n^{*(b)}).$

Mathematics

Case study: median cell response

Related notes