Stable Diffusion: generative image modelling on one consumer GPU

Text-to-image generative models existed before Stable Diffusion, but running or training one generally needed hardware and access well beyond a single consumer GPU, and the strongest models weren’t openly released. Stable Diffusion was the first widely-used text-to-image diffusion model small enough to run on a single consumer GPU, with weights released openly rather than kept behind an API — the combination that made large-scale generative image modelling broadly accessible for the first time.

The efficiency trick: diffusion in a compressed space, not pixel space

A diffusion model generates an image by starting from noise and iteratively removing it, guided by a text prompt. Running that iterative denoising process directly on full-resolution pixels is expensive — Stable Diffusion instead performs the diffusion process in a much smaller latent space, a compressed representation produced by a separate autoencoder, and only decodes back to full-resolution pixels at the very end. Denoising a small latent grid instead of a full-size image is what cut the compute requirement enough to fit on a single consumer GPU, without abandoning the diffusion approach itself.

Open weights changed who could build on it

Releasing the trained weights publicly, rather than only offering access through a hosted API, meant researchers and engineers outside the original team could inspect, fine-tune, and build directly on the model — the difference between “a capability that exists somewhere” and “a capability anyone can run and adapt locally.” That openness is a large part of why the ecosystem of tools and downstream applications built on Stable Diffusion grew as quickly as it did.

Where this connects to microscopy

Diffusion models trained this way later found their way into microscopy image analysis as denoising and restoration tools (a generative, learned alternative to the supervised restoration approach in CARE), for super-resolution, and for synthetic-data augmentation in label-scarce imaging datasets — generating additional plausible training examples when annotated real data is limited. The relevance isn’t the natural-image outputs themselves; it’s the underlying latent-diffusion machinery being repurposed for scientific image restoration and augmentation tasks.

Takeaways

• The efficiency breakthrough was running diffusion in a compressed latent space rather than directly on full-resolution pixels, cutting compute enough to fit a single consumer GPU.
• Open weights, not just capability, is what let a broad ecosystem of downstream tools and fine-tunes grow around this specific model rather than a closed, API-only alternative.
• The same latent-diffusion mechanism was later adapted for microscopy denoising, restoration, and synthetic-data augmentation — a generative alternative to supervised restoration methods like CARE.

Source

Stability AI, Stable Diffusion Public Release, 22 August 2022 — announcement.

Continue through the concepts

Related notes

AlexNet: the result that ended the debate about deep learning

How a convolutional network trained on two GPUs cut ImageNet error by a margin large enough to reset computer vision, and why every microscopy segmentation model on this site descends from it.

CLIP: learning to see from what images are captioned, not labelled

How pairing images with their naturally occurring text turned classification into a similarity search, and why that's the conceptual ancestor of every open-vocabulary vision tool since.

Van Valen et al.: CNNs learn live-cell tracking from ~100 annotated cells

One of the earliest demonstrations that a small amount of manual annotation was enough to train a CNN that out-tracked classical live-cell segmentation methods — the work that grew into DeepCell.

DINOv2: features learned with no labels at all

How a self-supervised training recipe produced general-purpose visual features that match supervised pretraining — the exact constraint most scientific imaging labs work under: plenty of images, almost no annotation.