What Is a Transformer? The Architecture Behind Modern AI

The transformer is a neural-network architecture introduced in the 2017 paper “Attention Is All You Need.” It is the foundation of GPT, Claude, Gemini, and virtually every other large language model — and increasingly of AI systems for images, audio, and code as well. Before the transformer, sequence models processed text one word at a time, which made them slow to train and poor at tracking meaning across long passages. The transformer replaced that sequential approach with a mechanism called attention, and the field has not looked back.

Tokens and embeddings

Processing starts by breaking text into tokens — roughly, word fragments. The tokenizer turns “transformer” into one token, “unbelievable” into two or three. Each token is mapped to an embedding: a vector of hundreds or thousands of floating-point numbers that positions the token in a high-dimensional meaning space. Tokens with similar meanings land near each other.

To give the model a sense of where each token falls in a sequence, positional encodings are added to the embeddings. Without them, the model would treat “the dog bit the man” and “the man bit the dog” identically.

Self-attention

The core mechanism is self-attention. For every token in the sequence, the model asks: which other tokens are most relevant to understanding this one? It computes three vectors for each token — a query, a key, and a value — and then matches each query against all keys in the sequence. High-scoring matches produce a weighted average of the corresponding values. The result is a new representation of each token that incorporates context from across the whole input.

This happens in parallel across all tokens simultaneously, which is why transformers train so much faster than the recurrent networks they replaced. RNNs had to process tokens one at a time; transformers process the entire sequence in a single pass.

Multi-head attention runs this process several times in parallel (the “heads”), each with different learned projections. Different heads can track different types of relationships — syntax in one head, coreference in another, domain-specific associations in a third — and their outputs are concatenated and projected back to the model’s working dimension.

The full pipeline

A transformer block stacks two sub-layers:

1. Multi-head attention — the context-gathering step described above.
2. Feed-forward network — a two-layer MLP applied independently to each token’s representation, adding non-linearity and capacity.

Each sub-layer has a residual connection (the input is added back to the output) and layer normalization. These techniques stabilize training and allow gradients to flow cleanly through dozens or hundreds of stacked blocks.

A full transformer model is many such blocks stacked end-to-end — GPT-3 has 96 layers; larger models have more. Information flows upward through the stack, each layer refining the representations produced by the one below.

Encoder, decoder, and decoder-only

The original paper introduced an encoder-decoder architecture for translation: the encoder reads the source sentence and builds rich contextual representations; the decoder generates the output token by token, attending to both its own generated text and the encoder’s output. This design still powers sequence-to-sequence tasks like machine translation and summarization.

Modern LLMs use a decoder-only architecture (the GPT line, Claude, Llama, Mistral). There is no separate encoder; the model attends to all preceding tokens and generates one next token at a time. This maps naturally to open-ended generation: you prompt the model and it continues the text. Vector embeddings extracted from transformer models are also useful for semantic search and retrieval.

Why it won

Two properties explain the transformer’s dominance:

• Parallelism. Because attention operates on the full sequence at once, transformers can exploit the thousands of cores in a GPU far more efficiently than sequential models ever could. This made it possible to train on internet-scale data.
• Scalability. Transformer performance improves predictably as you add more parameters, data, and compute — what the field calls scaling laws. No prior architecture showed this property as cleanly.

Together these unlocked the scaling era. The co-authors of the original paper scattered across the industry; one of them, Noam Shazeer, founded Character.AI and recently joined OpenAI. Their 2017 insight turned out to describe not just a better translation model but the engine behind nearly all of modern AI.

The newest models extend the architecture further: reasoning models add explicit chain-of-thought steps before producing an answer, and large language models built on transformers have grown from millions to hundreds of billions of parameters in under a decade.

The takeaway

The transformer replaced sequential processing with parallel self-attention, making it possible to capture long-range context across an entire sequence in a single pass. That parallelism made it trainable at scale; that scalability made it the foundation of modern AI. Understanding tokens, embeddings, attention heads, and stacked layers is understanding what every major AI model is, at its core.