Van Valen et al.: CNNs learn live-cell tracking from ~100 annotated cells

By 2016, deep learning had already reshaped natural-image classification, but live-cell microscopy segmentation was still largely handled by classical methods — watershed, thresholding, hand-tuned pipelines that broke whenever imaging conditions shifted. Van Valen et al. showed a CNN could segment and track individual cells in live-cell time-lapse movies more accurately than those classical methods, trained from as few as roughly 100 manually annotated cells — a small enough annotation budget that it was actually achievable in a normal lab, not just at internet scale.

Why “~100 cells” was the notable part, not just the accuracy

Contemporary deep learning results mostly relied on the assumption that more data was always available if you were willing to collect it — ImageNet had over a million labelled images. Biology labs don’t have that luxury; manually tracing cell boundaries frame by frame is slow, skilled work. Demonstrating strong performance from a training set two orders of magnitude smaller was the part that made this practical for individual labs, not just large, well-resourced ones.

Tracking, not just single-frame segmentation

The task wasn’t only “find the cells in this frame” but “keep each cell’s identity consistent across a live-cell time-lapse” — the same cell has to remain the same tracked object as it moves, divides, and changes shape across hundreds of frames. That combination of segmentation and identity-preserving tracking, learned end-to-end rather than glued together from separate detection and tracking stages, is what the paper demonstrated was learnable directly from a small annotated set.

What it grew into

This work became the foundation for DeepCell, a continuing platform for deep-learning-based cell segmentation and tracking that expanded well beyond the original demonstration. It’s an early, concrete example of a pattern that recurs throughout this timeline: a research result that starts as “here’s what’s possible with limited data” becomes, over the following years, packaged infrastructure that other labs can actually run.

Takeaways

• The headline result wasn’t just accuracy — it was that a CNN trained on roughly 100 manually annotated cells could outperform classical live-cell tracking methods, a realistic annotation budget for an individual lab.
• Segmentation and tracking were learned together, preserving cell identity across frames rather than treating each frame as an independent segmentation problem stitched together afterward.
• This is a direct ancestor, in spirit and method, of density-map and instance-tracking approaches used elsewhere in bacteriophage and particle counting work: small annotation, deep networks, quantitative per-object output.

Source

Van Valen et al., Deep Learning Automates the Quantitative Analysis of Individual Cells in Live-Cell Imaging, PLOS Computational Biology, 4 November 2016 — paper.

Continue through the concepts

Related notes

Mask R-CNN: instance segmentation as detection plus one branch

How adding a single mask-prediction branch onto an existing object detector turned 'find the objects' into 'find and precisely outline every object' — and the alignment fix that made small-object masks accurate.

U-Net: the architecture built for too little data

Why the encoder-decoder-with-skip-connections shape became the default starting point for biomedical segmentation, and how it was designed specifically around having very few annotated images.

AlexNet: the result that ended the debate about deep learning

How a convolutional network trained on two GPUs cut ImageNet error by a margin large enough to reset computer vision, and why every microscopy segmentation model on this site descends from it.

Cellpose: segmentation that generalises because it doesn't assume a shape

Why predicting a flow field toward each object's centre, instead of a mask directly, let one trained model segment cell types it had never seen — without retraining.