How Diffusion Models Generate Images and Video From Noise

Type a sentence and, seconds later, an image you have never seen appears. Describe a scene to Sora or feed a prompt to Midjourney, and the system hands back something that looks lifted straight from your imagination. Yet what actually happens inside that black box is rarely explained, even to people who use these tools every day.

This guide is about how diffusion models generate images and video—the engine behind nearly all of it. The idea sounds exotic but is surprisingly intuitive: add noise to a picture until it is pure static, then teach a network to run that destruction in reverse. We will walk from that single intuition through latent diffusion, text conditioning, and Diffusion Transformers, all the way to video. No equations required.

What Diffusion Models Are: Restoring a Picture From Noise

At their core, diffusion models are generative networks that deliberately corrupt data and then learn to reverse the corruption. The mechanism splits into two directions. A forward process adds noise to a clean image step by step until nothing recognizable is left, and a reverse process starts from that noise and peels the noise away to recover an image.

A useful analogy is a sculptor revealing a figure inside a block of marble. Here the “marble” is random noise and the “statue” is whatever the prompt describes. MIT Technology Review frames it the same way: the model “will start off with a random mess of pixels and step by step turn that mess into an image,” like static resolving on an old TV set.

Almost every image and video tool you have heard of belongs to this family. Midjourney and Stable Diffusion on the image side, Sora and Veo on the video side—all of them rest on this denoising mechanism. In other words, when people say “AI draws a picture,” the thing under the hood is almost always this noise-removal loop. The branding differs; the engine is essentially one idea.

The Forward Process: Breaking a Picture Into Static

Start with the forward process. The model takes a clean image and adds Gaussian noise (random jitter drawn from a normal distribution) a little at a time. It does not dump all the noise in at once; it layers it across many steps until, at the end, the original is buried under near-pure static.

How much noise enters at each step is set by a noise schedule, usually written as a sequence of beta (β) values that controls how fast the picture degrades. The crucial point is that this direction has nothing to learn. It is a fixed, mechanical procedure—add noise according to the schedule, full stop.

So why destroy a perfectly good image on purpose? Because these noised images are the training material. The model studies clean pictures paired with their corrupted versions and practices identifying exactly what noise was added. The forward process, then, is a factory stamping out an endless supply of practice problems and answer keys.

The Reverse Process: The Model Predicts the Noise Itself

Now the part that does the real work. Generation means starting from pure noise and stripping it away one step at a time until a picture emerges. Here is the counterintuitive twist: the network is not trained to predict the clean image directly. It is trained to predict the noise that was added to the current image.

Why bother with that framing? Because if you can pin down exactly which noise is present, subtracting it gives you a slightly cleaner image. Repeat that predict-and-subtract loop hundreds of times and random static turns into a sharp picture. From the model’s point of view, each step answers one modest question—”what noise is mixed into this blurry frame?”—and the stack of those answers is a finished image.

The model is also told which step it is on. That timestep is fed in as an embedding (a number converted into a vector the network can work with) built from sine and cosine functions. The traditional denoising backbone is the U-Net: downsampling layers that compress information, upsampling layers that rebuild it, and skip connections bridging the two halves in a “U” shape.

Why It Got Fast: Latent Diffusion

Early diffusion systems had a crippling flaw. Wrangling millions of raw pixel values across hundreds of steps was enormously expensive. The fix that broke the logjam is latent diffusion.

The idea is simple. Instead of manipulating high-resolution pixels directly, a VAE (variational autoencoder—a network that compresses an image into a small code and back) encodes the picture into a far smaller latent space, and the noising and denoising happen there. OpusClip notes that this latent space can be up to 64 times smaller than the original. Once denoising finishes, a decoder expands the result back to a full-size image.

The payoff was decisive. Cleaning a compact code instead of millions of pixels cut the compute bill enough that high-resolution generation became feasible on ordinary GPUs—part of the same broader push toward cheaper AI inference. Stable Diffusion, the most widely used open-source image model, popularized exactly this latent setup.

How a Line of Text Steers the Picture

Everything so far describes making some image out of noise. So how does a prompt like “a dog on a beach at sunset” pull the result in that direction? The answer is text conditioning.

The prompt is first turned into a meaning-rich embedding by a model such as CLIP. That embedding then intervenes at every denoising step, nudging the output toward images that match the text. As MIT Technology Review puts it, the language model “guides each step of the cleanup process, pushing the diffusion model toward images that the large language model considers a good match to the prompt.” The language model interpreting your prompt is on its own trajectory, much like reasoning models are.

The limits are just as real, though. The model learns the distribution of pixels but does not understand letters as symbols. So even though you drive generation with text, the signage and document text inside the output often come out smeared or turn into gibberish. Newer models keep chipping away at this, but it remains a signature weakness.

How Diffusion Differs From GANs

Before diffusion took the crown, image generation belonged to GANs (generative adversarial networks). A GAN pits a generator that fabricates convincing fakes against a discriminator that tries to tell real from fake, and the two improve by competing. The trouble was that this rivalry was often unstable.

GANs were prone to diverging during training and frequently fell into mode collapse (where outputs cluster into a few patterns and lose variety). Diffusion, by contrast, trains on the simple, stable objective of adding and removing noise, and it reproduces the true distribution of real images more faithfully. On both diversity and stability, the consensus is that diffusion holds the edge.

It is not free, of course. Where a GAN produces an image in a single pass, the model must grind through many sampling steps, which makes it slow. In one study, generating 50,000 images with the early DDPM approach took roughly 20 hours, while a GAN did the same in under a minute. To close that gap, accelerators like DDIM and model distillation arrived, cutting hundreds of steps down to dozens or fewer.

From U-Net to the Diffusion Transformer (DiT)

The biggest structural shift in recent diffusion work is that the denoising backbone moved from the U-Net to the transformer. The result is called a Diffusion Transformer (DiT).

The transformer’s strength is its attention mechanism, which captures relationships between distant elements—the long-range dependencies across both space and time. It also scales predictably: pour in more compute and performance improves in a way you can plan around. This is the very same transformer family that underpins large language models and Mixture of Experts (MoE) systems.

As a result, a large share of leading 2026 video models have converged on DiT backbones. Video generation is the most demanding case here. Consistency across a complex scene is paramount, and that is exactly where the transformer’s long-range reach pays off most. Put simply, the diffusion “engine” stayed the same while its internal “parts” were swapped for something more powerful.

Video Generation: Adding the Dimension of Time

Moving from images to video introduces an entirely new axis: time. Each video frame can look fine on its own, yet once you string seconds together, a subject that morphs or an object that blinks in and out instantly breaks the illusion. So video diffusion systems do not build frames in isolation; they process many frames together to lock in temporal consistency.

A common approach is to dice the video into small pieces. Tim Brooks, a Sora researcher, described it this way: cutting video “across both space and time…like if you were to have a stack of all the video frames and you cut little cubes from it.” A transformer then processes these spacetime patches and learns what natural motion looks like across them. Google DeepMind’s Veo 3 went a step further, compressing audio and video into a single piece of data so that sound and picture generate in sync.

There is a price, though. Attention’s cost grows quadratically as the sequence gets longer, so the longer the video, the more steeply compute climbs. Why this matters: the fact that video generation is mostly offered in short clips today is rooted in exactly this structural cost, not just product choice.

Frequently Asked Questions (FAQ)

Q1. What is the biggest difference between diffusion models and GANs?

A GAN trains through adversarial competition, which makes it unstable and prone to mode collapse, where diversity suffers. Diffusion trains on the simple objective of adding and removing noise, so it is stable and reproduces the real image distribution more faithfully. The cost is speed: multi-step sampling makes it slower than a GAN.

Q2. What is a latent diffusion model?

Rather than working on raw pixels, a VAE encoder compresses the image into a much smaller latent space—potentially dozens of times smaller—and the noising and denoising happen there. The lower compute load made high-resolution generation feasible on ordinary GPUs, and Stable Diffusion popularized the design.

Q3. Are Midjourney and Sora diffusion models too?

Yes. Most image tools like Midjourney and Stable Diffusion and most video tools like Sora and Veo belong to this family. Notably, the latest video models adopt a Diffusion Transformer (DiT) backbone—a transformer instead of a U-Net—to improve consistency between frames.

Q4. Why are diffusion models bad at text inside images?

The model learns the distribution of pixels but does not treat letters as symbols. As a result, even when you generate from a text prompt, signs and document text in the output frequently come out smeared or meaningless. Newer models are improving, but it remains a known weak spot.

Q5. How many steps does denoising take?

Early DDPM-style models were slow enough that the reverse process could run up to roughly 1,000 steps. Accelerators like DDIM and distillation later cut that to dozens, and in some cases a handful, of steps. Fewer steps mean faster generation but trade off against quality and diversity.

Q6. How does a Diffusion Transformer (DiT) differ from a U-Net?

A DiT replaces the U-Net denoising backbone with a transformer. Its attention captures long-range dependencies across space and time, and it improves predictably as compute grows, which is why it became mainstream for consistency-critical tasks like video.

References

• MIT Technology Review (2025-09-12) — Forward noising and reverse denoising, latent space, and the language model guiding each step.
• MIT Technology Review — Tim Brooks quote (2025-09-12) — Sora’s spacetime-cube decomposition and Veo 3’s joint audio-video generation.
• Murat Isikdogan, How Diffusion Models Work — The model predicting the noise itself, plus timestep embeddings.
• Murat Isikdogan, How Diffusion Models Work — The U-Net backbone of downsampling, upsampling, and skip connections.
• SuperAnnotate, Introduction to Diffusion Models — Up to ~1,000 reverse steps; DDPM at 50,000 images in ~20 hours vs. a GAN in under a minute (Song et al., 2020).
• SuperAnnotate, Introduction to Diffusion Models — DDIM’s fewer-step acceleration and same-seed reproducibility.
• SuperAnnotate, Introduction to Diffusion Models — Diffusion’s stable distribution matching versus GAN mode collapse.
• OpusClip, Understanding Diffusion Models — Latent space up to 64x compression and the Diffusion Transformer’s long-range dependencies.
• Scale AI, Diffusion Models: A Practical Guide — CLIP-embedding text conditioning and the weakness at rendering text in images.
• Scale AI, Diffusion Models: A Practical Guide — The emergence of Midjourney, Stable Diffusion, and CLIP-filtered training data.

jugger

Jugger | Backend Developer

A backend engineer working in enterprise solutions software since June 2021. The hands-on perspective from system design, API implementation, and production operations directly informs this site’s
AI, IT, and cloud infrastructure coverage.

For topics outside direct professional experience—blockchain and macroeconomics—analysis is built on primary-source citations such as SEC filings, whitepapers, institutional research, and government
statistics. Speculation is clearly separated from verified fact, and unverified claims are not published.

AI tools assist with drafting, but every article is reviewed, revised, and verified before publication. Errors are corrected with visible notes when discovered.

Contact: jjunggulee@gmail.com