Neural Network Optimizers: From Baby Steps to Intelligent Learning

"SGD walks. Momentum runs. Adam runs intelligently." When Four Examples Become Sixty Thousand In my last post , I showed you how backpropagation could learn the weights for XOR automatically. No more hand-crafting. No more trial and error. Just set a learning rate, run the algorithm, and watch the loss curve drop. It felt like magic. Almost too easy. But here's what I glossed over: XOR has just 4 training examples. With 4 examples, you compute the gradient using all of them at once. Every weight update sees the complete picture. But XOR is a toy problem. Let me tell you about a real dataset. MNIST: The "Hello World" of Deep Learning MNIST is a collection of 70,000 handwritten digit images—60,000 for training, 10,000 for testing. Each image is 28×28 grayscale pixels. The task: look at an image and predict which digit (0-9) it represents. Trivial for humans. Genuinely hard for 1990s computers. It became the standard benchmark for machine learning. Each image has 784 pixels (28×28). To cl