Cellpose: segmentation that generalises because it doesn't assume a shape

Most cell-segmentation networks before Cellpose were trained on one cell type, one imaging modality, one lab’s data — swap in a different cell shape and performance often collapsed. Cellpose is a single trained model that segments a wide variety of cell and nucleus shapes with no retraining, and the reason it generalises is a change in what the network is asked to predict: not a mask, but a flow field.

Predicting flow instead of a mask

For every pixel inside a cell, Cellpose predicts a small vector pointing toward that cell’s centre — a horizontal and a vertical flow value, plus a separate probability that the pixel belongs to any cell at all. At inference, every foreground pixel is nudged step by step along its predicted flow direction until it converges near a centre point; pixels that converge to the same point belong to the same cell. Instances fall out of this process automatically, without a separate detection or non-max-suppression step.

The reason this generalises better than predicting cell shape directly: flow fields don’t encode any assumption about what shape a cell should be. A round nucleus and an elongated, irregular cell both just need pixels that flow toward their own centre — the network never has to commit to “cells are roughly circular” or “cells are star-convex,” assumptions that break down the moment a new cell type doesn’t fit them.

Why the training data mattered as much as the method

Cellpose was trained on a deliberately diverse dataset — tens of thousands of manually segmented objects spanning many cell types, microscopy modalities, and even some non-biological “cell-like” images (added specifically to force the network to learn general boundary and shape cues rather than memorising one dataset’s appearance). That diversity is why the pretrained model works as a strong baseline on new data straight away, rather than needing to be retrained per experiment the way earlier specialist networks did.

Takeaways

• Predicting a flow field toward each object’s centre sidesteps having to assume a cell shape, which is why Cellpose generalises across cell types that plain mask prediction struggles with.
• Instances emerge from the flow-following step itself — no separate detection stage is needed.
• Training-set diversity, not just architecture, is a large part of why the released pretrained model works as an out-of-the-box baseline rather than a starting point that still needs retraining.
• Reach for Cellpose as the default first attempt before building anything custom — see the Evolution timeline entry for where it sits relative to StarDist and the later SAM-based segmentation models it predates.

Source

Stringer, Wang, Michaelos & Pachitariu, Cellpose: a generalist algorithm for cellular segmentation, Nature Methods, January 2021 — paper.

Continue through the concepts

Related notes

Van Valen et al.: CNNs learn live-cell tracking from ~100 annotated cells

One of the earliest demonstrations that a small amount of manual annotation was enough to train a CNN that out-tracked classical live-cell segmentation methods — the work that grew into DeepCell.

Mask R-CNN: instance segmentation as detection plus one branch

How adding a single mask-prediction branch onto an existing object detector turned 'find the objects' into 'find and precisely outline every object' — and the alignment fix that made small-object masks accurate.

SAM 2: keeping a mask's identity across a video

How a streaming-memory architecture let promptable segmentation extend from single images to video, tracking an object's identity through occlusion and motion in real time.

SAM 3: segmenting by concept, not by click

How moving from single-instance prompts to open-vocabulary concept prompts let one phrase or example image segment every matching object in a scene at once.