CLIP: learning to see from what images are captioned, not labelled

A conventional image classifier is trained against a fixed list of categories decided in advance — add a new category later, and the model needs retraining. CLIP (contrastive language-image pretraining) trained on roughly 400 million naturally occurring image-caption pairs instead of a fixed label set, and the result was a model that classifies images into categories it was never explicitly trained on, just by comparing them against a text description at inference time.

Matching, not labelling

CLIP trains two encoders together — one for images, one for text — so that the embedding of an image and the embedding of its matching caption end up close together in a shared space, while embeddings of mismatched image-text pairs end up far apart. This is a contrastive objective: for every batch of image-caption pairs, the model is pushed to correctly match each image to its real caption among all the others in the batch, using nothing but the massive supply of images already paired with descriptive text on the open web.

Classification becomes a similarity search

Once trained, “classifying” an unseen image doesn’t require retraining or even a classification head: embed the image, embed a handful of candidate text prompts (“a photo of a {category}”), and pick whichever text embedding the image is closest to. New categories are just new text prompts — nothing about the model itself changes. That’s what “zero-shot” means here: no examples of the new category were ever shown during training, only its description in natural language at inference time.

Why this is the conceptual ancestor of open-vocabulary tools

The appeal that makes CLIP-style models interesting for scientific imaging is the same appeal driving foundation models generally: categories in real experiments keep changing (a new phenotype, a new condition, a new structure of interest), and retraining a fixed-category classifier for each one is impractical. A model that can be redirected with a text prompt instead of a labelled retraining set is a direct answer to that constraint — the same open-vocabulary logic that SAM later applied to segmentation rather than classification.

Takeaways

• The training signal is matching image-caption pairs from the open web, not a fixed set of category labels — scale came from data that already existed, not new annotation effort.
• Classification becomes a similarity comparison between an image embedding and text-prompt embeddings, so new categories require a new prompt, not retraining.
• This zero-shot-via-text-prompt idea is the direct conceptual predecessor of later open-vocabulary segmentation tools such as SAM 3.

Source

Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), OpenAI, January 2021 — announcement.

Continue through the concepts

Related notes

Transformers: the architecture that made attention the whole model

How replacing recurrence with self-attention unlocked full parallelism during training, and why nearly every model later on this timeline is built on this one mechanism.

AlexNet: the result that ended the debate about deep learning

How a convolutional network trained on two GPUs cut ImageNet error by a margin large enough to reset computer vision, and why every microscopy segmentation model on this site descends from it.

DINOv2: features learned with no labels at all

How a self-supervised training recipe produced general-purpose visual features that match supervised pretraining — the exact constraint most scientific imaging labs work under: plenty of images, almost no annotation.

DINOv3: fixing the feature drift that blocked longer self-supervised training

Why dense patch-level features used to degrade over very long self-supervised training runs, and how Gram anchoring fixed it well enough to be the first SSL model to beat weakly-supervised alternatives broadly.