Vision and image generation

SeeingCopy Anchor Link

Before 2012, image recognition used hand-designed features. AlexNet, a convolutional neural network (a network that slides small filters across an image to detect edges, then shapes, then objects), cut the ImageNet error rate sharply. Later models like ResNet passed human-level accuracy on that benchmark. Vision models now power photo search, medical imaging support, self-driving perception and document reading.

DrawingCopy Anchor Link

Generative adversarial networks (GANs, 2014) pitted a generator against a critic to produce realistic images. Diffusion models took over from 2020. A diffusion model is trained by adding noise to images step by step and learning to reverse it. To generate, it starts from pure noise and removes it step by step, guided by a text prompt. DALL-E, Midjourney and Stable Diffusion (open weights, 2022) made this mainstream. Video models such as Sora extended the idea to moving images.

Why it mattersCopy Anchor Link

The same pattern of a scalable architecture, lots of data and lots of compute drove both. It also raised real questions about training data rights, deepfakes and provenance, which the field is still working out.

A worked example: how a filter finds a faceCopy Anchor Link

A small filter, say 3 by 3 pixels, slides across the image. Where the pixels look like a vertical edge the filter produces a high number. Many filters produce many maps: edges, then corners, then eyes and noses. Stack layers and the last one says "face". The network learns the filters from examples. This sliding filter is called a convolution.

A worked example: diffusion in four stepsCopy Anchor Link

Start with a clean photo of a dog. Add a little random noise, then more, until it is pure static. Train a network to guess the noise that was added at each step. To generate, start from static and repeatedly subtract the guessed noise, steered by your text prompt "a dog wearing a hat". After about 20 to 50 steps an image appears.

How the filters get their valuesCopy Anchor Link

A filter is a small grid of numbers. At the start those numbers are random. The network sees a labelled photo, makes a guess and measures how wrong it was (the loss). Gradient descent then nudges every filter number a little in the direction that shrinks the loss. After millions of photos, the early filters turn into edge detectors, because edges help with almost every recognition task. Nobody types them in. A 3 by 3 filter holds 9 numbers, so a layer of 32 such filters on a black-and-white image holds 9 × 32 = 288 numbers.

Why drawing takes many small stepsCopy Anchor Link

Removing all the noise in one jump gives a blurry average of everything the model has seen. Small steps let it commit to details gradually. Early steps tend to settle the rough layout and colors, and later steps add texture and sharp edges. A guidance setting controls how strongly your text pulls on each step. Too low and the image ignores your words. Too high and images often look harsh and overdone.

When the result is wrongCopy Anchor Link

The symptom is a hand with six fingers, a sign with scrambled letters, or three apples when you asked for four. The cause is that the model learned what these things usually look like, not the rules behind them. A recognition model that only saw bright daytime photos can also fail at night, because the pixels look unlike its training data. To check whether a flaw is a model limit or bad luck, repeat the prompt with ten different random seeds (the seed fixes the starting noise). If most of the ten share the flaw, it is the model.

Try it yourself: count the applesCopy Anchor Link

Open any free image generator. Ask for "a hand holding four red apples" five times. Count the fingers and the apples in each image. Then add detail ("on a white table, daylight, photo") and repeat. Counting errors are common, especially above three or four objects, and often persist while the style changes. Treat that as a sign the model copies appearance more than it counts.

Common mistakeCopy Anchor Link

Believing the image model understands what it draws. It captures how things usually look. That is why hands, text and counting objects were hard for early versions.