Statistical note

Non-parametric tests: what they do and do not test

A guide to rank-based methods, permutation tests, distributional assumptions, and interpretation beyond the median myth.

3 August 20266 min read

Non-parametric does not mean assumption-free. These methods avoid specifying a small, fixed distributional family for every aspect of the data, but they still depend on the sampling design, independence, exchangeability, or assumptions about distributional shape.

Values transformed into ranks for comparison

Rank-based tests

The Mann–Whitney test compares the ordering of observations between independent groups. It is not automatically a test of medians: that interpretation requires similarly shaped distributions. The Wilcoxon signed-rank test uses paired differences and assumes their distribution is symmetric.

If $R_x$ is the rank sum for sample $x$ , then

$U_x=R_x-\frac{n_x(n_x+1)}{2}.$

from scipy import stats

control = [4.1, 4.5, 4.9, 5.0, 5.2]
treated = [5.1, 5.4, 5.8, 6.2, 8.9]
print(stats.mannwhitneyu(treated, control, alternative="two-sided"))

before = [12, 14, 11, 15, 13]
after = [10, 13, 9, 12, 12]
print(stats.wilcoxon(after, before))

Permutation tests

A permutation test constructs a reference distribution by rearranging labels in ways justified by the null hypothesis. It can target a mean difference, median difference, correlation, or a custom statistic. Its key requirement is exchangeability under the null.

When they help

Rank and permutation methods are useful when outliers, skew, ordinal outcomes, or small samples make a conventional model questionable. They may sacrifice information by replacing values with ranks, and they do not repair dependence or poor experimental design.

Report the estimand

State exactly what difference the procedure targets, and accompany the test with an effect estimate and uncertainty interval whenever possible.

Case study: skewed cell-motility measurements

When a few cells travel exceptionally far, rank methods reduce their leverage. But cells within one movie remain dependent. Compute replicate-level summaries or use a hierarchical analysis before choosing a rank test.