Microscopy + computer vision
How microscopy and computer vision learned to see more.
A curated journey from optical sectioning and computational restoration to quantitative phase imaging, deep learning, and foundation models. Milestones appear newest first and connect directly to their scientific sources.
Image credit: Original explanatory graphic based on Miao et al. (CC BY preprint)
Miao et al. · bioRxiv
Independent benchmarking of foundation cell-segmentation models
Cellpose cyto3, Cellpose-SAM, μSAM and CellSAM are compared on phase-contrast and fluorescence cell-culture images, with Mesmer and InstanSeg added for multiplexed CODEX tissue imaging.
No single model is universally best: segmentation must be selected for the imaging modality, data quality and downstream biological question.
Image credit: Israel et al., Nature Methods (Springer Nature)
Israel et al. · Nature Methods
CellSAM: a foundation model for cell segmentation
CellSAM reframes cell segmentation as a foundation-model problem spanning imaging domains.
It shifts the emphasis from collections of specialist networks toward reusable cell representations.
Image credit: Meta AI
Meta AI
SAM 3: segment anything with concepts
Moved beyond single-instance prompts to open-vocabulary "concept" prompts. A short phrase or an example image now segments every matching instance at once, with SAM 3D adding single-image 3D reconstruction.
Concept-level, multi-instance prompting is close to how a microscopist actually wants to work: "segment every phage," not "click every phage."
Read the field note
Image credit: BioImage.IO official project asset
Field-wide development
Microscopy foundation-model ecosystem
SAM variants, CellSAM, DINO-style embeddings, multimodal systems and domain-specific foundation models proliferate.
The paradigm is shifting toward reusable representations and promptable models instead of one network per dataset.
Image credit: Meta AI
Meta AI
DINOv3: self-supervised learning at unprecedented scale
Trained on 1.7 billion images with up to 7 billion parameters, DINOv3 introduced Gram anchoring to keep dense, patch-level features stable through very long training runs, the first SSL model to outperform weakly-supervised alternatives across a broad task suite.
I explore DINOv3-style patch features as a transfer-learning backbone for microscopy segmentation. See the pipeline demo on the homepage.
Read the field note
Image credit: Nature Methods (Springer Nature)
Nature Methods
Cell simulation as cell segmentation
A simulation-driven approach is published for cell segmentation in spatial data.
It shows how segmentation is expanding beyond conventional pixel-classification architectures.
Image credit: Archit et al., Nature Methods (Springer Nature)
Archit et al., Nature Methods
Segment Anything for Microscopy (µSAM)
Submitted in August 2023, accepted in November 2024 and published in February 2025, μSAM fine-tuned SAM for light and electron microscopy and packaged the models with annotation, instance-segmentation and tracking workflows in napari.
This is the exact translation I care about: a foundation model adapted, validated, and shipped as a usable tool for working microscopists.
Read the field note
Image credit: SAMCell preprint, Figure 6 (CC BY)
bioRxiv
SAMCell
SAMCell is introduced for generalised, label-free biological-cell segmentation using a SAM-derived approach.
It is another sign that the SAM ecosystem is branching into domain-specific cell models.
Image credit: Meta AI
Meta AI
SAM 2: segmentation for images and video
Announced on 29 July 2024, with the full paper released on 1 August, SAM 2 extended promptable segmentation to images and video through a streaming-memory architecture that tracks objects across frames in real time.
Multi-object tracking that survives occlusion and motion maps directly onto time-lapse microscopy, where the same cell or particle must keep its identity across hundreds of frames.
Read the field noteImage credit: Archit et al., Nature Methods (Springer Nature)
Bioimage-analysis community
SAM enters biomedical and microscopy research
Researchers begin testing SAM across medical, microscopy and other scientific-image domains.
The work reveals strong transfer potential but also the limits of natural-image pretraining for specialised modalities.
Image credit: Meta AI
Meta AI
DINOv2: self-supervised vision backbones
A self-supervised training recipe producing general-purpose visual features that match or exceed supervised pretraining, without needing any labelled data.
Self-supervision is exactly the constraint most scientific imaging labs work under: plenty of images, very little annotation.
Read the field note
Image credit: Meta AI
Meta AI
Segment Anything (SAM)
A promptable segmentation foundation model trained on 1.1 billion masks, able to segment unfamiliar objects from a point, box, or text prompt without task-specific fine-tuning.
Prompted, zero-shot segmentation is directly relevant to cell and particle segmentation in fluorescence and TIRF microscopy, where retraining a detector for every new experiment is impractical, and directly led to µSAM below.
Read the field note
Image credit: Pachitariu & Stringer, Nature Methods (Springer Nature)
Pachitariu & Stringer · Nature Methods
Cellpose 2.0
Cellpose adds human-in-the-loop retraining for rapid model specialisation.
Researchers can adapt a generalist model to their images instead of starting from scratch.
Image credit: Stability AI
Stability AI
Stable Diffusion public release
The first widely-used open-weights text-to-image diffusion model, small enough to run on a single consumer GPU. It made large-scale generative image modelling broadly accessible for the first time.
Diffusion models later found their way into microscopy as denoising, super-resolution, and synthetic-data-augmentation tools for label-scarce imaging datasets.
Read the field note
Image credit: Greenwald et al., Nature Biotechnology (Springer Nature)
Greenwald et al. · Nature Biotechnology
Mesmer and TissueNet
Large-scale tissue-cell data and deep learning enable near-human whole-cell segmentation.
Cell segmentation expands from cultured-cell imagery into multiplexed tissue imaging.
Image credit: BioImage.IO official project asset
BioImage.IO community
BioImage.IO and the BioImage Model Zoo
Microscopy models begin to ship with standardised descriptions and interoperable inference specifications.
The trained model itself becomes a portable and reusable scientific artefact.
Image credit: Edlund et al., Nature Methods (Springer Nature)
Edlund et al. · Nature Methods
LIVECell dataset
More than one million manually annotated live cells across several cell types are released.
It provides a major generalisation benchmark for label-free cell segmentation.
Image credit: Caron et al., arXiv:2104.14294 (Meta AI)
Caron et al. · Meta AI
DINO self-supervised vision
Self-supervised vision transformers learn useful visual representations without manual labels.
Large, unlabelled microscopy archives can become training data rather than dormant image collections.
Image credit: OpenAI
OpenAI
CLIP: connecting text and images
Contrastive language-image pretraining showed that a single model could learn transferable visual representations from natural-language supervision, with strong zero-shot transfer to new visual categories.
The conceptual ancestor of today’s open-vocabulary, zero-shot vision tools, the same appeal that makes foundation-model backbones interesting for microscopy, where labelled data is scarce and categories keep changing.
Read the field note
Image credit: Stringer et al., Nature Methods (Springer Nature)
Stringer, Wang, Michaelos & Pachitariu, Nature Methods
Cellpose: a generalist algorithm for cellular segmentation
A single trained model that segments a huge variety of cell and nucleus types with no retraining or parameter tuning, trained on a new dataset of over 70,000 annotated objects. It became one of the most widely used tools in bio-image analysis.
Cellpose is the tool I reach for as a strong baseline before trying anything custom, a good example of "generalist beats bespoke" for everyday cell segmentation.
Read the field note
Image credit: Dosovitskiy et al., arXiv:2010.11929 (Google Research)
Dosovitskiy et al. · Google Research
Vision Transformer (ViT)
Images are processed as sequences of patch tokens by a transformer.
ViT forms the architectural bridge from convolutional vision to modern visual foundation models.
Image credit: napari official project repository (BSD-3-Clause)
napari community
napari
A Python-native viewer for multidimensional images grows into an extensible microscopy-analysis platform.
It connects scientific Python, AI models and interactive microscopy workflows.
Image credit: Wang, Rivenson et al., Nature Methods (Springer Nature)
Wang, Rivenson et al. (Ozcan Lab, UCLA), Nature Methods
Deep learning enables cross-modality super-resolution in fluorescence microscopy
A GAN trained to transform diffraction-limited images into super-resolved ones, confocal images made to match STED, and TIRF images made to match TIRF-SIM, without modelling the optics or measuring a point-spread function.
This is the "AI super-resolution" moment for fluorescence microscopy: trading acquisition complexity for a trained model, one of the clearest viral demonstrations that deep learning could push past the physical resolution of the instrument itself.
Image credit: Weigert et al., Nature Methods (Springer Nature)
Weigert et al., Nature Methods
CARE: content-aware image restoration
Trained networks to restore clean images from noisy, low-light acquisitions, recovering usable data from up to 60-fold fewer photons, near-isotropic resolution from undersampled volumes, and higher frame rates than direct acquisition allowed.
CARE reframed noise and undersampling as a learnable, reversible problem rather than a hard physical limit, a mindset that shows up everywhere in modern computational microscopy, including phase imaging.
Read the field note
Image credit: Krull, Buchholz & Jug, arXiv:1811.10980
Krull, Buchholz & Jug
Noise2Void and self-supervised denoising
Networks learn to denoise directly from noisy images without matched clean targets.
This removes one of microscopy restoration’s hardest requirements: clean ground truth.
Image credit: Schmidt et al., arXiv:1806.03535
Schmidt, Weigert, Broaddus & Myers, MICCAI
StarDist: cell detection with star-convex polygons
Represented nuclei as star-convex polygons predicted directly by a CNN, giving accurate instance segmentation of round, densely packed nuclei without the shape-refinement steps earlier detectors needed.
Along with Cellpose, StarDist (via its Fiji/napari plugins) is one of the tools that brought deep-learning segmentation to microscopists who don’t train models themselves, infrastructure work that matters as much as the architecture.
Read the field note
Image credit: Ouyang et al., Nature Biotechnology (Springer Nature)
Ouyang et al. · Nature Biotechnology
ANNA-PALM
A neural network reconstructs PALM-like images from far fewer acquisition frames.
It demonstrates that computation can be traded for acquisition time.
Image credit: Nehme et al., arXiv:1801.09631
Nehme et al. · Optica
Deep-STORM
Deep networks reconstruct localisation microscopy from dense emitter images.
AI begins replacing expensive iterative super-resolution reconstruction.
Image credit: Vaswani et al., arXiv:1706.03762
Vaswani et al. · NeurIPS
Transformers: Attention Is All You Need
The transformer architecture replaces recurrence with attention.
It becomes the architectural basis for ViT, CLIP, DINO, SAM and modern vision foundation models.
Read the field note
Image credit: ImageJ/Fiji Trainable Weka Segmentation documentation
Arganda-Carreras et al. · Bioinformatics
Trainable Weka Segmentation
The Fiji/ImageJ machine-learning pixel classifier is formally published.
Microscopists gain practical access to random forests and engineered image features without writing ML code.
Image credit: He et al., arXiv:1703.06870 (Facebook AI Research)
He, Gkioxari, Dollár & Girshick, Facebook AI Research
Mask R-CNN
Extended Faster R-CNN with a mask-prediction branch, giving a simple, general framework for instance segmentation, detecting and precisely outlining every object instance in an image.
Long before SAM, Mask R-CNN-based pipelines were adapted throughout bio-image analysis for instance segmentation of cells, organelles, and particles in dense fields of view.
Read the field note
Image credit: Van Valen et al., PLOS Computational Biology (CC BY)
Van Valen et al., PLOS Computational Biology
Deep learning automates quantitative analysis of individual cells in live-cell imaging
One of the earliest demonstrations that a CNN could segment and track individual cells in live-cell time-lapse movies more accurately than classical methods, from as few as ~100 manually annotated cells, the work that grew into DeepCell.
A direct ancestor of the density-map and tracking work I did during my PhD: small amounts of annotation, deep networks, quantitative single-object output.
Read the field note
Image credit: Ronneberger et al., arXiv:1505.04597
Ronneberger, Fischer & Brox, MICCAI
U-Net: convolutional networks for biomedical image segmentation
The encoder–decoder architecture with skip connections that became the default starting point for biomedical image segmentation, born from winning the ISBI cell tracking challenge on transmitted-light microscopy images with very little training data.
The architecture almost every microscopy segmentation model since, including tools I have used and adapted, is a descendant of, in some form.
Read the field note
Image credit: Goodfellow et al., arXiv:1406.2661
Goodfellow et al. · NeurIPS
Generative adversarial networks
Generative adversarial networks introduce competition between a generator and discriminator.
GANs later enable virtual staining, domain translation, restoration, synthesis and super-resolution in microscopy.
Image credit: Wikimedia Commons
Krizhevsky, Sutskever & Hinton, NeurIPS
AlexNet and the deep learning break
A deep convolutional network trained on GPUs cut the ImageNet top-5 error rate by over 10 points against the previous best, a margin large enough to end the debate about whether deep learning worked, the starting gun for everything on this page.
Every segmentation and restoration method microscopy now depends on is downstream of this result, the origin point of the field this whole timeline traces.
Read the field note
Image credit: ilastik official website
ilastik team
ilastik interactive machine learning
Users label a small number of pixels or objects and a machine-learning classifier infers the remainder.
It marks the transition from hand-written threshold rules to interactive learned segmentation.
Image credit: OpenSlide official project asset (LGPL)
Digital pathology
Whole-slide imaging becomes practical
Automated scanners combine acquisition, focusing, registration, stitching and pyramidal storage to digitise pathology slides.
Histology shifts from viewing glass to navigating gigapixel digital specimens, enabling computational pathology.
Image credit: ImageJ BigStitcher official documentation
Preibisch, Saalfeld & Tomancak · Bioinformatics
Globally optimal microscopy stitching
Pairwise overlap estimates and global optimisation reconstruct tiled 2D and 3D acquisitions without accumulating local registration error.
This becomes a key computational foundation for large-field microscopy mosaics.
Image credit: Schindelin et al., Nature Methods (Springer Nature)
Fiji / ImageJ community
Fiji ecosystem develops
Fiji develops as an integrated ImageJ distribution and plugin ecosystem; its landmark paper appears in 2012.
Processing, registration, segmentation, tracking and scripting become part of one reproducible ecosystem.
Image credit: Rust, Bates & Zhuang, Nature Methods (Springer Nature)
Rust, Bates & Zhuang · Nature Methods
STORM
Stochastic optical reconstruction microscopy demonstrates localisation-based super-resolution.
Single-molecule localisation microscopy becomes a major experimental paradigm.
Image credit: Carpenter et al., Genome Biology (BioMed Central, CC BY)
Carpenter et al.
CellProfiler
CellProfiler introduces automated, high-throughput cell-image analysis.
Phenotypic screening becomes computational and scalable.
Image credit: Correlative PALM figure, PMC3871616
Betzig et al.
PALM
Photoactivated localisation microscopy localises sparse single emitters.
It establishes a landmark route to nanometre-scale fluorescence localisation.
Image credit: Light-sheet microscopy review, PMC2685720
Huisken et al. · Science
SPIM and modern light-sheet microscopy
Selective plane illumination microscopy is demonstrated for biological imaging.
Large living specimens can be imaged rapidly in 3D with substantially less phototoxicity.
Image credit: Phasics official QWLSI technology page
Phasics · SID4 technology
QWLSI wavefront sensing and Phasics technology
Quadriwave lateral shearing interferometry uses a two-dimensional diffractive grating to encode phase gradients, reconstructing wavefront and optical path difference in a compact, single-shot measurement. Phasics introduced its patented QWLSI technology commercially in 2004; foundational QWLSI papers followed in 2005.
High-resolution, achromatic wavefront sensing without a separate reference arm enabled robust quantitative measurements for optics metrology and, subsequently, plug-and-play quantitative phase imaging with systems such as SID4 Bio.
Image credit: Open Microscopy Environment official asset
OME consortium
Open Microscopy Environment
OME establishes open infrastructure for microscopy data and metadata.
It begins addressing interoperability between microscope manufacturers and analysis software.
Image credit: EPFL official publication repository asset
Thévenaz, Ruttimann & Unser
Pyramid-based subpixel image registration
A coarse-to-fine, spline-based method makes rigid and affine registration accurate at subpixel scale.
Robust registration supports repeated acquisitions, stacks, multimodal data and mosaics.
Image credit: ImageJ official project asset
Wayne Rasband · NIH
ImageJ and the NIH Image lineage
ImageJ emerges from Wayne Rasband’s NIH Image work as an extensible image-processing platform.
Quantitative microscopy becomes accessible to everyday biologists and plugin developers.
Image credit: Phasics official QWLSI technology page
Digital interferometry and computational microscopy
Quantitative phase imaging emerges
Advances in digital interferometry, phase-shifting methods, digital holographic microscopy and transport-of-intensity reconstruction make quantitative phase imaging increasingly practical. Rather than fluorescence intensity, QPI measures the optical path difference introduced by transparent specimens.
QPI establishes label-free quantitative microscopy for studying properties such as cell morphology, dry mass, growth and refractive-index variation without staining or fluorescent labels.
Image credit: Tomographic STED microscopy, PMC7316010 (CC BY)
Hell & Wichmann
STED microscopy proposed
Stimulated-emission depletion microscopy is proposed.
It shows conceptually that Abbe’s diffraction limit is not an absolute barrier.
Image credit: Two-photon microscopy review, PMC2718834
Denk, Strickler & Webb · Science
Two-photon microscopy
Two-photon laser-scanning fluorescence microscopy is demonstrated.
It enables deeper live-tissue imaging with localised excitation and reduced phototoxicity.
Image credit: Richardson–Lucy deconvolution study, PMC3986040 (CC BY)
L. B. Lucy
Lucy iterative deconvolution
Lucy independently develops a maximum-likelihood iterative image-restoration method.
Together with Richardson’s work, it forms the Richardson–Lucy algorithm used for PSF deconvolution.
Image credit: Richardson–Lucy deconvolution study, PMC3986040 (CC BY)
William H. Richardson
Richardson iterative deconvolution
A Bayesian iterative method reconstructs an object from a blurred observation and a known point-spread function.
It establishes one of microscopy’s fundamental computational deconvolution approaches.
Image credit: Confocal microscopy principles review, PMC6961134
Marvin Minsky
Confocal microscopy
Marvin Minsky patents the confocal scanning microscope concept.
Rejecting out-of-focus light enables optical sectioning and lays a foundation for modern 3D optical microscopy.
This is a curated history rather than an exhaustive chronology. Titles link to primary papers, project pages, patents, or official announcements wherever available. Dates shown as ranges mark developments that emerged over time. Visuals are paper figures, official project assets, relevant open-access illustrations, or clearly identified original explanatory graphics. Continue with myresearch and projects, or read theblog.