Matrices and tensors

The ideaCopy Anchor Link

A matrix with 2 rows and 3 columns turns a 3-number vector into a 2-number vector. Each output entry is a dot product of one row with the input. Multiplying a whole batch of vectors by one matrix does many dot products at once, which is what GPUs are built for. A tensor is an array with any number of axes: a vector has 1, a matrix has 2, a batch of images has 4 (batch, channels, height, width).

ExampleCopy Anchor Link

Let M = [[1, 0, 2], [0, 1, 1]] and x = [3, 4, 5]. Row 1 · x = 3 + 0 + 10 = 13. Row 2 · x = 0 + 4 + 5 = 9. So M x = [13, 9]. Shapes must match: (2 by 3) times (3) gives (2). When shapes differ in a compatible way, libraries use broadcasting, which stretches the smaller array to fit: adding a bias vector of length 2 to each row of a batch happens without copying it.

Why it mattersCopy Anchor Link

A layer of a network is one matrix multiply plus an activation. A language model with billions of parameters is mostly large matrices. Getting shapes right is the most common source of bugs when writing model code.

How matrix times matrix worksCopy Anchor Link

Multiplying two matrices is many matrix-times-vector steps, one for each column of the second matrix. Take A = [[1, 0, 2], [0, 1, 1]] (2 by 3) and B = [[1, 1], [2, 0], [0, 3]] (3 by 2). The result is 2 by 2, and entry (row i, column j) is row i of A dotted with column j of B.

1. Entry (1,1): 1×1 + 0×2 + 2×0 = 1.
2. Entry (1,2): 1×1 + 0×0 + 2×3 = 7.
3. Entry (2,1): 0×1 + 1×2 + 1×0 = 2.
4. Entry (2,2): 0×1 + 1×0 + 1×3 = 3. So A B = [[1, 7], [2, 3]]. Order matters: B A is 3 by 3, a different shape and different numbers. The inner sizes (3 and 3) must match, and the outer sizes (2 and 2) become the result.

Counting the workCopy Anchor Link

Invented sizes for illustration: a layer that turns 1,000 input numbers into 500 output numbers has a 500 by 1,000 matrix. That is 500 × 1,000 = 500,000 weights, plus 500 biases (one per output), so 500,500 learned numbers. One input needs 500,000 multiplications. A batch of 32 inputs needs 32 × 500,000 = 16,000,000. That is why speed and memory are measured in matrix sizes.

What goes wrong: silent broadcastingCopy Anchor Link

Most shape bugs raise an error, which is the good case. Broadcasting can hide a bug. Add a column of shape (3, 1), [1, 2, 3], to a row of shape (3), [10, 20, 30]. You expected 3 numbers. The library instead stretches both and returns a 3 by 3 grid whose first row is [11, 21, 31] and last row is [13, 23, 33]. Later code may still run and give a wrong loss. The symptom is a result with more entries than you expected, so print shapes after each step.

Check it by handCopy Anchor Link

Let M = [[2, 0], [1, 1], [0, 3]] and x = [4, 5]. M is 3 by 2, so the answer has 3 numbers. Row 1 gives 8, row 2 gives 4 + 5 = 9, and row 3 gives 15. You should get [8, 9, 15]. If you try x = [4, 5, 6], the inner sizes (2 and 3) clash, which is exactly the error you would see in code.

Common mistakesCopy Anchor Link

Shape errors. If the matrix is 2 by 3, the vector must have 3 numbers, and the result has 2. Most beginner bugs in model code are shapes that do not line up. Print the shape of everything while you debug.