Transformers: the architecture that made attention the whole model

Before 2017, the standard way to process a sequence — a sentence, a time series — was a recurrent network that read it one element at a time, carrying a hidden state forward. That sequential dependency was also the bottleneck: step 500 couldn’t be computed until step 499 finished, so training couldn’t be parallelised across the sequence length. Transformers removed recurrence entirely and replaced it with self-attention: every position looks directly at every other position in one parallel computation. It was introduced for machine translation, but the mechanism is now the shared foundation under ViT, DINO, CLIP, and SAM — every vision model later on this timeline.

What self-attention actually computes

For each position in the input, self-attention produces three vectors — a query, a key, and a value. A position’s new representation is a weighted sum of every position’s value vector, where the weight comes from comparing that position’s query against every other position’s key. Concretely:

$\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$

The practical consequence: information can flow directly between any two positions in a single layer, regardless of how far apart they are in the sequence — an RNN has to relay information step by step across that same distance, losing signal along the way. “Multi-head” attention just runs several of these comparisons in parallel with different learned projections, so different heads can specialise in different kinds of relationships.

Why removing recurrence mattered more than the attention mechanism itself

Attention as an add-on to RNNs already existed before this paper. What changed was removing the RNN and keeping only attention (plus position information injected separately, since attention alone has no notion of order). Without a sequential dependency, every position in a training batch can be processed simultaneously on a GPU. That parallelism is why transformer-based models could be scaled up so much faster than RNNs of comparable size — training that would take weeks with recurrence took days, which is a large part of why the next decade of “just add more data and compute” scaling became practical at all.

Why this appears on a microscopy timeline

Vision Transformers, DINO, DINOv2/v3, CLIP, and SAM are all built from this same self-attention block, applied to image patches instead of words. None of the ideas that made those models work — patch tokens, self-supervised pretraining at scale, promptable segmentation — required inventing a new core computation. They required figuring out how to feed images into this architecture and what to train it on.

Takeaways

• The core move was subtraction, not addition: dropping recurrence entirely, not bolting attention onto it, is what unlocked full parallel training.
• Self-attention lets any two positions interact directly in one layer — an RNN needs as many steps as the distance between them.
• Every model referenced elsewhere on this timeline from ViT onward inherits this exact mechanism, adapted to a different input (image patches instead of tokens) and a different training objective.

Source

Vaswani et al., Attention Is All You Need, NeurIPS 2017 — paper. See this entry in context on the Evolution timeline.

Continue through the concepts

Related notes

CLIP: learning to see from what images are captioned, not labelled

How pairing images with their naturally occurring text turned classification into a similarity search, and why that's the conceptual ancestor of every open-vocabulary vision tool since.

AlexNet: the result that ended the debate about deep learning

How a convolutional network trained on two GPUs cut ImageNet error by a margin large enough to reset computer vision, and why every microscopy segmentation model on this site descends from it.

DINOv2: features learned with no labels at all

How a self-supervised training recipe produced general-purpose visual features that match supervised pretraining — the exact constraint most scientific imaging labs work under: plenty of images, almost no annotation.

DINOv3: fixing the feature drift that blocked longer self-supervised training

Why dense patch-level features used to degrade over very long self-supervised training runs, and how Gram anchoring fixed it well enough to be the first SSL model to beat weakly-supervised alternatives broadly.