Segment Anything (SAM): segmentation you can prompt instead of retrain

Training a segmentation model has traditionally meant collecting labelled examples of exactly the objects you want segmented and training (or at least fine-tuning) a network on them. Segment Anything (SAM) reframed the task itself: instead of a model that only knows the categories it was trained on, SAM takes a prompt — a click, a box, or text — and returns a mask for whatever the prompt indicates, on objects and image types it has never seen before, with no retraining. That single idea is why it appears twice more, evolved, later on this site’s Evolution timeline: µSAM and CellSAM both exist because researchers asked what happens when this idea is pointed at microscopy specifically.

Three ideas that make it work

• A promptable task definition. Given an image and a prompt (point, box, rough mask, or text), the model must return a valid segmentation mask for whatever the prompt refers to. This is a different task from “classify this pixel into one of N known categories” — the categories are never fixed in advance.
• A data engine, not just a dataset. SAM was trained on SA-1B: 1.1 billion masks across 11 million images. That scale didn’t come from manual annotation alone — an early version of the model helped annotators label faster, the improving model was used to auto-generate more masks, and human review kept correcting the loop. The dataset and the model were built together, each making the other better.
• A split architecture built for interactivity. A heavyweight image encoder runs once per image; a lightweight prompt encoder and mask decoder then turn any new prompt into a mask in milliseconds. This is what makes clicking around an image and getting instant masks back actually feel real-time, rather than re-running a full network per click.

Ambiguity is a first-class output, not an edge case

A single point on an image is often genuinely ambiguous: a click on a shirt could mean “the shirt,” “the person wearing it,” or “the whole scene.” Rather than forcing one answer, SAM outputs several valid candidate masks with confidence scores for an ambiguous prompt, and leaves the choice to whoever (or whatever) issued the prompt. For dense, cluttered scientific images — overlapping cells, aggregated particles — this matters more than it might for typical photographs.

What it does not solve, on its own

SAM was trained on natural photographs, not fluorescence, phase-contrast, or electron microscopy images. Its zero-shot performance on those modalities is real but noticeably weaker than on natural images — cell boundaries, faint fluorescence signal, and phase-imaging contrast don’t look like anything in SA-1B. That gap is exactly what the later, microscopy-specific entries on this timeline (µSAM, CellSAM) exist to close, by fine-tuning or re-architecting around this same promptable idea for scientific imaging data.

Takeaways

• The shift is from “train a detector for your object” to “prompt a general model for whatever you point at” — retraining a detector for every new experiment stops being the default path.
• The billion-mask training set was produced by a model-in-the-loop data engine, not manual labelling at that scale — the data and the model improved together.
• Multiple candidate masks per ambiguous prompt is a deliberate design choice, not model uncertainty leaking through by accident.
• Out-of-the-box performance on microscopy images is weaker than on natural photos; that gap, not a flaw to work around quietly, is the direct motivation for µSAM and CellSAM later in the same timeline.

Source

Kirillov et al., Segment Anything, Meta AI, April 2023 — announcement. See this entry in context on the Evolution timeline.

Continue through the concepts

Related notes

SAM 2: keeping a mask's identity across a video

How a streaming-memory architecture let promptable segmentation extend from single images to video, tracking an object's identity through occlusion and motion in real time.

SAM 3: segmenting by concept, not by click

How moving from single-instance prompts to open-vocabulary concept prompts let one phrase or example image segment every matching object in a scene at once.

µSAM: closing the gap between SAM and real microscopy images

SAM was trained on natural photographs and performs noticeably worse on fluorescence and electron microscopy out of the box. µSAM fine-tunes it for exactly those modalities and ships it as a usable napari tool.

AlexNet: the result that ended the debate about deep learning

How a convolutional network trained on two GPUs cut ImageNet error by a margin large enough to reset computer vision, and why every microscopy segmentation model on this site descends from it.