Bayes factors and Bayesian model comparison

How should prior knowledge and observed evidence combine into a probability distribution over plausible parameter values? This note turns that question into a practical analysis workflow.

Core idea

Compare marginal evidence carefully, recognising sensitivity to prior scale and the distinction from posterior predictive quality. The method is useful only when its target, data-generating assumptions, and unit of analysis match the scientific question.

$BF_{10}=\frac{p(y\mid M_1)}{p(y\mid M_0)}$

The equation is a compact statement of the target; it is not a substitute for checking design and data quality. State what each observation represents, how it entered the sample, and which sources of dependence remain.

Practical workflow

1. Write the scientific question and estimand in one sentence.
2. Identify biological units, technical replicates, nesting, repeated measurements, and exclusions.
3. Visualise raw observations and group structure before fitting the method.
4. Check assumptions using design knowledge, diagnostic plots, and sensitivity analyses.
5. Report the estimate, uncertainty, effect magnitude, sample sizes at every level, and limitations.

import numpy as np
from scipy import stats

# Beta-binomial update: prior Beta(2, 2), then 18 successes in 25 trials.
a_post, b_post = 2 + 18, 2 + 25 - 18
posterior = stats.beta(a_post, b_post)
print(posterior.mean(), posterior.interval(0.95))

Interpretation and failure modes

Do not read a software output as an automatic scientific conclusion. Ask whether dependence was modeled, whether preprocessing used information from validation data, whether missingness or selection is informative, and whether the reported uncertainty covers every level of sampling. Prefer estimates and intervals to threshold-only language.

Microscopy case study

Evidence for a time trend can change materially under an implausibly diffuse slope prior, so sensitivity analysis is essential.

Connections

Use the tags above to continue to notes on design, assumptions, uncertainty, diagnostics, and complementary methods. A robust analysis normally combines several concepts rather than selecting one test in isolation.

Continue through the concepts

Related notes

Bayes theorem for statistical inference

Update prior plausibility with data evidence and distinguish posterior probability from a p-value.

Credible intervals and highest-density intervals

Summarise posterior uncertainty with direct probability statements while preserving skew or multimodality when important.

Posterior predictive checks

Evaluate whether replicated data from a fitted Bayesian model resemble important features of the observations.

Priors and prior-predictive checks

Encode plausible parameter scales and test their implications before observing the study outcome.