What Is a Diffusion Model? How AI Makes Images

A diffusion model is a type of generative AI that creates images — and increasingly audio and video — by learning to reverse a gradual noising process. Given a text prompt, the model starts from pure random noise and iteratively refines it, step by step, into a coherent image. It’s the technology behind Stable Diffusion, DALL·E 3, Midjourney, and Sora.

The core idea: destroy, then rebuild

Diffusion models are trained in two phases:

The forward process takes a real image and adds small amounts of random noise across many steps — hundreds or thousands — until the image is completely destroyed, reduced to pure Gaussian noise. This is simple mathematics, not a neural network.

The reverse process is where learning happens. A neural network is trained to look at a slightly noisy image and predict what noise was added at that step, so it can remove it. Repeat that operation hundreds of times starting from pure noise and you reconstruct a coherent image. The model never sees the final image directly; it just learns to undo one step of noise at a time.

At inference, you skip the forward process entirely. You start with random noise, feed it through the learned reverse process repeatedly, and an image emerges.

The neural network inside

The network doing the denoising is typically a U-Net — an architecture with a contracting path that captures context and an expanding path that reconstructs detail, connected by skip connections. More recent models replace the U-Net with a diffusion transformer (DiT), which applies the same transformer architecture used in language models. Transformers scale better and handle longer-range dependencies, which is why they dominate newer image and video models.

Text conditioning

A diffusion model without a text prompt would just generate random images. Text conditioning links language to visual generation. A text encoder — often a model like CLIP — converts your prompt into a dense numerical representation (embeddings). Those embeddings are injected into the denoising network at each step via cross-attention, steering the image toward whatever the prompt describes. The more specific the prompt, the more precisely it guides the output.

Latent diffusion: why Stable Diffusion is fast

Running thousands of denoising steps on full-resolution pixel data is computationally brutal. Latent diffusion solves this by operating in a compressed latent space instead of pixel space. A separate encoder (a variational autoencoder, or VAE) compresses the image into a much smaller representation; the diffusion model works in that compressed space; and the VAE decoder expands the result back to a full image at the end.

This is why Stable Diffusion can run on a consumer GPU. The denoising network handles a small tensor, not millions of pixels — the heavy lifting is done in a compact abstract space. This design is sometimes called an LDM (latent diffusion model) and became the dominant architecture after Stable Diffusion popularized it.

What they power

Diffusion models are now behind most state-of-the-art image generation tools:

• Stable Diffusion — open-weight latent diffusion model, runs locally, community-driven ecosystem
• DALL·E 3 — OpenAI’s image generator, integrated into ChatGPT
• Midjourney — proprietary, known for stylized aesthetic output
• Imagen and Flux — diffusion models from Google and Black Forest Labs respectively
• Sora and other video generators — extend diffusion to temporal sequences of frames

The GPU requirements for running or training diffusion models are steep; high-VRAM cards and optimized sampling algorithms (like DDIM or DPM-Solver) are needed for practical speeds.

How diffusion models compare to alternatives

Before diffusion models dominated, GANs (generative adversarial networks) were the leading approach. A GAN pits a generator against a discriminator in an adversarial loop, which can produce sharp results but is notoriously unstable to train and prone to “mode collapse” (generating limited variety). Diffusion models are more stable, produce more diverse outputs, and handle text conditioning naturally.

Autoregressive image models (like the early versions of DALL·E) treat images as sequences of tokens, predicting one patch at a time — similar to how an LLM predicts text tokens. They’re flexible but slow for large images. Diffusion models in latent space are currently faster and produce higher-quality outputs.

The takeaway

Diffusion models work by learning to undo noise — a simple principle that, scaled up with good text conditioning and a compressed latent space, produces remarkably capable image generators. They’ve largely displaced GANs for photorealistic and artistic generation, and the same architecture is now driving video and audio generation as well. Understanding forward and reverse diffusion, the role of embeddings, and the latent space trick covers most of what you need to know about how today’s AI image tools actually work.