StarDist: segmenting nuclei by predicting their shape directly

Segmenting a field of densely packed, touching nuclei with a plain per-pixel classifier has a persistent failure mode: without extra help, adjacent nuclei that touch tend to merge into one connected blob, because the network only ever sees “nucleus” versus “not nucleus,” never a boundary between two nuclei that are pressed against each other. StarDist sidestepped the problem by changing what the network predicts: not a per-pixel class, but the shape of each nucleus directly.

A shape a network can predict in one shot

For every pixel, StarDist’s network predicts two things: the probability that the pixel is a nucleus centre, and, for candidate centres, the distance to the nucleus boundary along a fixed set of radial directions (say, every few degrees around a full circle). Connecting those boundary distances traces out a star-convex polygon — any shape where every point on the boundary is visible from the centre along a straight line, which describes most round or oval nuclei well. Overlapping polygon candidates near the same object are then merged with non-maximum suppression, giving one polygon per nucleus without a separate post-processing step to split touching objects.

Why this beats plain per-pixel segmentation for this specific problem

Because each object is represented as one shape prediction anchored at one centre point, two touching nuclei simply produce two separate polygons from two separate centres — there’s no shared boundary pixel for the network to get confused about, the way there is with plain semantic segmentation. This makes StarDist particularly strong exactly where naive pixel classification is weakest: dense fields of nuclei that touch or nearly touch.

The shape assumption is also the limitation

“Star-convex” is not “any shape.” An elongated, branching, or concave object can’t be represented well as distances from a single centre point along straight radial lines — trying to fit one anyway produces a polygon that clips the true boundary. This is exactly the constraint Cellpose’s flow-field approach, developed shortly after, was designed to remove: predicting a direction field toward a centre works for arbitrary shapes, not just star-convex ones, at the cost of a slightly more involved inference procedure.

Takeaways

• Predicting object shape directly (as a star-convex polygon) rather than a per-pixel mask avoids the touching-object merging problem without extra post-processing.
• The method is well matched to round or oval nuclei specifically — that’s the shape family “star-convex” actually covers, not an approximation for shape in general.
• For elongated, irregular, or branching objects, prefer a shape-agnostic method such as Cellpose, which was built partly to remove this exact constraint.

Source

Schmidt, Weigert, Broaddus & Myers, Cell Detection with Star-Convex Polygons, MICCAI 2018 — paper.

Continue through the concepts

Related notes

Cellpose: segmentation that generalises because it doesn't assume a shape

Why predicting a flow field toward each object's centre, instead of a mask directly, let one trained model segment cell types it had never seen — without retraining.

Van Valen et al.: CNNs learn live-cell tracking from ~100 annotated cells

One of the earliest demonstrations that a small amount of manual annotation was enough to train a CNN that out-tracked classical live-cell segmentation methods — the work that grew into DeepCell.

Mask R-CNN: instance segmentation as detection plus one branch

How adding a single mask-prediction branch onto an existing object detector turned 'find the objects' into 'find and precisely outline every object' — and the alignment fix that made small-object masks accurate.

SAM 2: keeping a mask's identity across a video

How a streaming-memory architecture let promptable segmentation extend from single images to video, tracking an object's identity through occlusion and motion in real time.