Neurons and networks
A neuron is a weighted sum plus a bend. A network is many neurons in layers. Stacking them lets the model represent curved, complex functions.
One neuron: y = ReLU(w₁x₁ + w₂x₂ + b)
output for x₁ (→) and x₂ (↑)
0.5×1 + -1×2 + 0.5 = -1.00 → ReLU → 0.00
The sum is negative, so ReLU clips it to 0: the neuron stays silent.
Words on this page (9)
- neuron
- : One unit in a network: it multiplies inputs by weights, adds a bias and applies an activation function.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- activation function
- : A simple curve such as ReLU applied after a neuron's sum. It lets stacked layers represent complex functions.
- ReLU
- : An activation function that outputs the input if positive, otherwise 0.
- matrix
- : A grid of numbers. Multiplying a vector by a matrix transforms it. Model layers are mostly matrices.
- transformer
- : The neural network architecture behind modern language models. It uses attention to relate every token to the others.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- learning rate
- : How big each gradient descent step is.
- batch
- : A group of examples processed together in one step.
The idea
A takes inputs x, multiplies each by a w, adds a bias b, and passes the sum through an f: y = f(w · x + b). is the common choice: f(z) = max(0, z). Without the bend, stacking layers would still be one straight-line function. With it, layers compose into something far more expressive.
Example
Inputs x = [1, 2], weights w = [0.5, -1], bias b = 0.5. The weighted sum is 0.5×1 + (-1)×2 + 0.5 = -1. ReLU(-1) = 0. The neuron stays silent. Change the bias to 3 and the sum becomes 1.5, ReLU gives 1.5, and the neuron fires.
Layers
A layer is many neurons reading the same inputs, which is a multiplication followed by an activation. A network passes the output of one layer to the next. A modern language model is the same pieces at huge scale, arranged as blocks, with billions of weights.
Try it
Change the weights and bias in the demo and watch the output and the decision line change.
A second example: why one neuron is not enough
Take the XOR task: output 1 when exactly one of two inputs is 1. The four cases are (0,0) → 0, (0,1) → 1, (1,0) → 1 and (1,1) → 0. A single neuron draws one straight line to separate its answers, and no line can do this. Two layers can. Let h1 = ReLU(x1 + x2) and h2 = ReLU(x1 + x2 - 1), then output h1 - 2×h2. Case (1,1): h1 = 2, h2 = 1, output 2 - 2 = 0. Case (1,0): h1 = 1, h2 = 0, output 1. Case (0,0): both 0, output 0. All four cases come out right, using 9 learned numbers (6 weights and 3 biases).
Why the bend is required
Remove the activation and a second layer does nothing new. If layer one multiplies by 3 and layer two multiplies by 2, together they multiply by 6, which one layer could do. With ReLU in between, the layers cannot merge, and each adds new shapes. That is the mechanism.
What goes wrong: dead neurons
If a neuron's sum is negative for every input, ReLU outputs 0 and its slope is 0 too, so cannot repair it. This is a dead neuron. Typical cause: a so large that one update pushed the bias far below zero. Symptom: many hidden values that are exactly 0 for every example. To confirm, run a through and count how often each neuron is above 0. A neuron stuck at 0% is dead.
Check it by hand
Use x = [2, -1], w = [1, 3] and bias b = 1. The sum is 2 - 3 + 1 = 0, so the output is 0 and the neuron is silent. Which bias makes the output 2? Solve 2 - 3 + b = 2 to get b = 3. Check: 2 - 3 + 3 = 2, and ReLU keeps it.
Common mistake
Imagining that each neuron has a clear human meaning. In big networks most neurons mean nothing on their own. The meaning is spread across many.
Think of a neuron as a bouncer at a club with a checklist. Each fact about a guest is multiplied by how much the bouncer cares about it, the total is added up, and if the total clears a bar the guest gets in. A network is thousands of such bouncers in rows, each row deciding based on what the previous row let through.
Check yourself
Why does a network need an activation function?
Not quite. The bend is what lets deep networks represent complex functions.
x = [1, 2], w = [1, 1], b = -4 and a ReLU. What is the output?
Not quite. The sum is 1 + 2 - 4 = -1, and ReLU clips negatives to 0.
A neuron fires for cats using whiskers (+2), ears (+1) and barking (-3), with bias -1. A barking animal with whiskers and ears gives what?
Not quite. 2 + 1 - 3 - 1 = -1, and ReLU turns a negative sum into 0.
You got 0 of 3 right.
Go deeper
- Video
3Blue1Brown · The standard visual explanation.
- Tool
Google · Train a small network in the browser and watch each neuron.
- Course
Michael Nielsen · A free online book that builds the idea from scratch.
Spotted a mistake or something unclear? Report it on GitHub.