Field note
SAM 2: keeping a mask's identity across a video
How a streaming-memory architecture let promptable segmentation extend from single images to video, tracking an object's identity through occlusion and motion in real time.
The original SAM segments one image at a time — prompt it, get a mask, done. Time-lapse and video data need something more: the same mask, following the same object, as it moves, deforms, or briefly disappears behind something else across hundreds of frames. SAM 2 extends promptable segmentation from single images to video by adding memory: prompt an object once, and the model propagates its mask forward through the rest of the sequence on its own.
A memory bank, not a frame-by-frame re-detection
Running image-SAM independently on every frame wouldn’t preserve object identity — it would find a mask each time, with no guarantee it’s the same object as the frame before, and no way to handle a brief occlusion. SAM 2 instead keeps a streaming memory of recent frames’ features. As each new frame arrives, the model attends back into that memory to figure out where the previously prompted object is now, carrying identity forward even through motion, partial occlusion, and temporary disappearance from view, rather than treating each frame as a fresh, unrelated segmentation problem.
One model for both images and video
SAM 2 doesn’t run as two separate systems — a single-image mask is just the video case with a sequence length of one. That unification is what let SAM 2 also outperform the original SAM on plain image segmentation while adding video support, rather than trading one capability for the other.
Why this matters for time-lapse microscopy
Keeping identity across frames despite occlusion and motion is precisely the requirement in live time-lapse imaging: a cell or particle has to keep being “the same object” as it moves through the field, temporarily overlaps a neighbour, or drifts out of focus. A promptable, memory-based tracker removes the need to train a bespoke tracking model per experiment — the same trade the original SAM made for single-image segmentation, extended to the temporal dimension.
Takeaways
- The addition over image-SAM is memory: attending back into recently seen frames to keep the same object identified as it moves, rather than re-detecting independently on every frame.
- Occlusion handling comes from the same mechanism — a briefly hidden object’s identity persists in memory rather than being lost and re-detected as new.
- Images and video are handled by one unified model, not two separate systems bolted together.
Source
Ravi et al., SAM 2: Segment Anything in Images and Videos, Meta AI, 29 July 2024 — announcement.
Continue through the concepts
Related notes
SAM 3: segmenting by concept, not by clickHow moving from single-instance prompts to open-vocabulary concept prompts let one phrase or example image segment every matching object in a scene at once.
µSAM: closing the gap between SAM and real microscopy imagesSAM was trained on natural photographs and performs noticeably worse on fluorescence and electron microscopy out of the box. µSAM fine-tunes it for exactly those modalities and ships it as a usable napari tool.
Segment Anything (SAM): segmentation you can prompt instead of retrainHow SAM turned segmentation into a promptable task, why its billion-mask training set exists, and why that idea mattered enough to spawn a whole branch of microscopy-specific models.
AlexNet: the result that ended the debate about deep learningHow a convolutional network trained on two GPUs cut ImageNet error by a margin large enough to reset computer vision, and why every microscopy segmentation model on this site descends from it.