Machine learning and the deep learning turn

Learn instead of programCopy Anchor Link

Machine learning flips programming. Instead of writing rules, you provide examples of inputs and desired outputs, and an algorithm adjusts a model until it fits. In 1986 Rumelhart, Hinton and Williams popularized backpropagation, the method that trains networks with many layers. In the 1990s Yann LeCun's convolutional networks read handwritten digits on bank checks.

The 2012 momentCopy Anchor Link

ImageNet was a dataset of over a million labeled images, started by Fei-Fei Li in 2009. In 2012 a convolutional network called AlexNet, trained on two gaming GPUs, cut the top-5 error of the annual ImageNet contest from about 26 percent to about 15 percent. The runner-up methods were hand-engineered feature pipelines. After that, nearly every field moved to deep learning.

What came afterCopy Anchor Link

Word2vec in 2013 showed that words could be turned into vectors where directions carry meaning. Sequence-to-sequence networks with attention improved translation in 2014 and 2015. ResNets made networks hundreds of layers deep trainable in 2015. In 2016 DeepMind's AlphaGo beat Lee Sedol at Go, a game thought to be a decade away. Each success used the same recipe: a flexible model, lots of data, lots of compute.

What "deep" meansCopy Anchor Link

A neural network is layers of simple units. "Deep" just means many layers. In an image model, the first layer learns to spot tiny edges, the next combines edges into corners and curves, the next combines those into eyes and wheels, and the last layer says "cat". Nobody programs those steps. The layers fit themselves to the data by training.

A worked example: why GPUs matteredCopy Anchor Link

A network's main job is multiplying big grids of numbers. A normal processor does a few calculations at a time. A graphics card (GPU) was built to color millions of pixels at once, so it can do thousands of multiplications in parallel. Training that took months on a regular chip could take days. The idea of deep networks was old. GPUs plus the ImageNet dataset made it practical.

How a network learns its layersCopy Anchor Link

Backpropagation answers one question for every weight: if I nudge this weight up a little, how much does the output change? Take a toy chain, ignoring activation functions to keep it simple. Input x = 1 goes through weight w1 = 2 and then weight w2 = 3, so the output is 1 × 2 × 3 = 6. The target is 8.

Raise w2 by 0.1 and the output becomes 2 × 3.1 = 6.2, a change of 0.2. Raise w1 by 0.1 and the output becomes 2.1 × 3 = 6.3, a change of 0.3. So w1 deserves the bigger nudge here. Backpropagation computes these sensitivities for every weight in one backward pass, starting from the output error. Then each weight moves a small step, set by the learning rate, toward lower error.

What goes wrong in trainingCopy Anchor Link

Symptoms show up in the loss, the number that measures error.

1. Loss stays flat. The learning rate may be too small, or there is a bug such as labels not matching inputs.
2. Loss jumps up or becomes NaN (a broken "not a number" value). The learning rate is usually too large, so steps overshoot.
3. Training loss falls but validation loss rises. The network is memorizing.

A useful check is to train on just 10 examples. A healthy setup should push the loss close to zero, because memorizing 10 items is easy. If it cannot, suspect a bug before blaming the data.

A second example: when deep learning is the wrong choiceCopy Anchor Link

Say you have 2,000 rows describing customers, and you want to predict who cancels. A network with millions of parameters has far more freedom than the data can justify, so it overfits. A gradient-boosted tree or a simple linear model is cheaper to train, faster to check and easier to explain to the person who must act on its output.

Deep learning pays off when the input is raw and messy, such as pixels or audio, and there is a lot of data. For small tables, start with the simple model and use it as the baseline any network must beat.

Common mistakeCopy Anchor Link

Thinking deep learning replaced everything. It dominates images, speech and language, while simpler methods still win on many tables of business data.