What Is a Vision Transformer (ViT)?

A Vision Transformer, or ViT, applies the transformer architecture — originally built for language — to images, by slicing an image into a grid of fixed-size patches and treating each patch like a token in a sentence. Instead of the sliding convolutional filters that dominated computer vision for over a decade, a ViT lets every patch attend to every other patch through self-attention, learning which parts of an image matter to each other directly from data.

From pixels to tokens

A convolutional neural network (CNN) processes an image through local filters that slide across pixels, building up features from edges to textures to objects layer by layer. A ViT skips that local-to-global buildup. It instead:

1. Splits the image into patches — commonly 16×16 pixel squares, so a 224×224 image becomes a 14×14 grid of 196 patches.
2. Flattens and linearly projects each patch into a vector, the same way a word gets turned into an embedding in a language model.
3. Adds positional encoding to each patch embedding, since — just as in text — the transformer has no inherent sense of where a patch sits in the image without it. See positional encoding in transformers for how that works.
4. Prepends a special classification token, borrowed from BERT-style language models, whose final representation is used for whatever the model is predicting.
5. Feeds the whole sequence through standard transformer encoder layers, where self-attention lets every patch weigh its relationship to every other patch, regardless of distance.

The result is a model architecture nearly identical to a text transformer, with the patch-embedding step as the only real vision-specific piece.

Why attention instead of convolution

A convolutional filter only sees a small local neighborhood at each layer; a network has to stack many layers before a neuron’s receptive field covers the whole image. Self-attention has no such restriction — from the very first layer, a patch in one corner of the image can directly attend to a patch in the opposite corner. That gives ViTs an easier time modeling long-range relationships between distant parts of an image, like recognizing that a shadow on one side relates to an object on the other.

The tradeoff is that convolutions come with useful assumptions — translation invariance and locality — baked into the architecture itself. A ViT has to learn those patterns from data instead of getting them for free, which is why vision transformers historically needed far more training data than CNNs to match or beat them. Trained on a small dataset, a ViT tends to underperform a comparably sized CNN; trained on a very large one, it tends to pull ahead.

ViT vs. CNN

	CNN	Vision Transformer
Core operation	Convolutional filters	Self-attention over patches
Inductive bias	Strong (locality, translation invariance)	Weak — learned from data
Data efficiency	Good on smaller datasets	Needs large-scale pretraining to shine
Receptive field	Grows gradually with depth	Global from the first layer
Compute scaling	Scales locally with resolution	Attention cost grows quadratically with patch count

Neither fully replaced the other. Many production vision systems today use hybrid designs — convolutional stems feeding into transformer blocks — to get CNN-style data efficiency with transformer-style long-range modeling.

Where ViTs show up

Vision transformers underpin a lot of modern multimodal work. A vision-language model typically uses a ViT, or a close variant, as its image encoder, converting an image into a sequence of embeddings that get fed alongside text tokens into a language model, letting the system answer questions about an image or generate a caption for it. Image classification, object detection, and segmentation systems have all adopted transformer-based backbones, often pretrained on large unlabeled image sets before fine-tuning on a specific task.

ViTs are also common on the generation side, though generative image models more often lean on diffusion architectures than pure autoregressive transformers; see how GANs work for the earlier generative approach that transformers and diffusion models have largely displaced for state-of-the-art image synthesis.

Practical considerations

The quadratic cost of self-attention with respect to sequence length matters more for vision than for text, because a moderately sized image easily produces hundreds of patches, and higher resolution multiplies that count further. This has pushed research toward hierarchical and windowed attention variants that limit how many patches attend to each other directly, trading some of the global-context advantage for lower compute cost — a similar tradeoff space to the one long-context language models navigate when balancing context length against inference cost.

The takeaway

A Vision Transformer treats an image as a sequence of patches and processes them with the same self-attention mechanism that powers language transformers, trading the built-in locality assumptions of convolutions for a global, learned view of the image. It needs more data to reach its potential than a CNN does, but it scales well and has become the default image encoder inside modern multimodal systems.