Statistical note

Evaluating object detection: matching, not just counting

Why detection and counting metrics live or die on the matching rule between predictions and ground truth — and how duplicate detections and missed objects distort the count.

11 August 20267 min read

Classification metrics ask “was this one label right?” Object detection evaluation asks a harder question first: which predicted object corresponds to which real one? A model can output the right number of objects while none of them line up with the true ones, or find every real object while also hallucinating three extra ones on top. Neither failure shows up if you only compare counts — you have to match predictions to ground truth before precision and recall mean anything.

Conceptual guide to object detection and counting evaluation

The matching rule decides the metric

Before computing anything, you have to define when a predicted object “counts” as the same object as a ground-truth one. Two common choices:

Overlap-based (IoU) — for boxes or masks, a prediction matches a ground-truth object if their intersection-over-union exceeds a threshold (commonly 0.5).
Distance-based — for point-like objects (puncta, particles, phages), a prediction matches if it falls within a chosen radius of a ground-truth point.

Once objects are matched one-to-one, the familiar formulas apply to matched pairs, not raw counts:

$Precision=\frac{TP}{TP+FP},\qquad Recall=\frac{TP}{TP+FN}$

Change the IoU threshold or the matching radius and every downstream number changes with it — a looser tolerance forgives near-misses, a tighter one punishes them. Report the tolerance used, since a precision-recall number without its matching rule cannot be compared to anyone else’s.

Two failure modes counting alone can’t see

Duplicates. If a model predicts two overlapping boxes for one real object, a naive count treats it as a correct detection, but a proper matching rule only allows one prediction to match each ground-truth object — the extra one becomes a false positive.
Density-dependent misses. In a dense field of overlapping puncta or crowded cells, both missed detections and duplicate detections rise together, and they partially cancel in the raw count while precision and recall both quietly get worse. A count that happens to be “close to right” can still hide a detector that’s wrong about which objects it found.

Worked example: matching fluorescent puncta by distance

import numpy as np
from scipy.optimize import linear_sum_assignment

true_points = np.array([[10, 10], [40, 12], [70, 55], [22, 80]])
pred_points = np.array([[11, 9], [41, 60], [69, 54], [90, 90]])
radius = 5

cost = np.linalg.norm(true_points[:, None, :] - pred_points[None, :, :], axis=2)
row, col = linear_sum_assignment(cost)
matched = cost[row, col] <= radius

tp = matched.sum()
fn = len(true_points) - tp          # real puncta with no acceptable match
fp = len(pred_points) - tp          # predictions with no acceptable match
print(f"TP={tp} FN={fn} FP={fp}  precision={tp/(tp+fp):.2f}  recall={tp/(tp+fn):.2f}")

linear_sum_assignment finds the best one-to-one pairing before applying the distance tolerance, which is what stops one bright real punctum from “absorbing” several nearby false positives into false matches.

Takeaways

Decide and report the matching tolerance (IoU threshold or distance radius) before quoting precision/recall — it is part of the metric, not an implementation detail.
Match predictions to ground truth one-to-one before scoring; comparing raw counts alone hides duplicates and misses that cancel each other out.
Density matters: re-check performance separately in sparse and crowded regions of the image rather than trusting one global number.
For downstream biology, propagate the per-image count bias forward — see measurement error propagation — rather than treating the detector’s count as ground truth.

The matching rule decides the metric

Two failure modes counting alone can’t see

Worked example: matching fluorescent puncta by distance

Takeaways

Related notes