Why Neural Networks Need Activation Functions

Remove one tiny function from a 10-layer neural network, and it drops from 100% to 47%: no better than a coin flip. We build up from a single artificial neuron (it's exactly logistic regression from episode four), the bias, Rosenblatt's perceptron and the XOR problem it can't solve, two layers that solve it, activation functions (sigmoid, tanh, ReLU), the forward pass of a real network on handwritten digits, why stacked layers collapse into one line without activations, universal approximation, and dead ReLUs. Every number on screen is real output from code you can run. Episode 9 of "From a Straight Line to ChatGPT", a full series on AI, machine learning and deep learning for developers and interview prep. ⏱ Chapters 0:00 Intro 0:47 The building block 1:21 You've built one already 1:56 What the bias is for 2:25 The perceptron 3:06 XOR: one line isn't enough 3:47 Two layers solve XOR 4:23 Activation functions 5:03 Why ReLU? 5:43 Layers and the forward pass 6:32 What can go wrong: the collapse 7:12 Universal approximation 8:01 What can go wrong: dead ReLUs 8:44 Try it yourself 9:16 Interview corner 10:25 Recap and next stop 🧪 Try it yourself (pip install numpy) import numpy as np X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) # the four possible inputs y = np.array([0, 1, 1, 0]) # XOR: 1 when exactly one input is 1 W1 = np.array([[1, 1], [1, 1]]) # hidden layer: 2 neurons b1 = np.array([0, -1]) # one bias per hidden neuron W2 = np.array([1, -2]) # output neuron b2 = 0 def relu(z): return np.maximum(0, z) # negatives become 0 Z = X @ W1 + b1 # weighted sum plus bias H = relu(Z) # ...then the activation out = H @ W2 + b2 # the output neuron for x, z, h, o in zip(X, Z, H, out): print(f"input {x} sums {z} hidden {h} output {o}") print("XOR solved:", np.array_equal(out, y)) Now remove the activation: the two layers collapse into one neuron. print("without ReLU:", Z @ W2 + b2) W, b = W1 @ W2, b1 @ W2 + b2 print("one neuron with weights", W, "and bias", b, "gives", X @ W + b) What we got: the hidden values come out [0 0], [1 0], [1 0], [2 1] and the outputs 0, 1, 1, 0: XOR solved. Without ReLU, the outputs become 2, 1, 1, 0, and the two layers collapse into one neuron with weights [-1 -1] and bias 2. 🧠 Cheat sheet Neuron = weighted sum of inputs + bias, then an activation function One neuron with a sigmoid = logistic regression The bias sets how much evidence it takes to tip over One layer (a perceptron) can't solve XOR; two layers can Sigmoid: 0 to 1, slope at most 0.25 · tanh: -1 to 1 · ReLU: max(0, z) Without activations, any stack of layers collapses into one linear function Universal approximation: enough neurons can approximate any continuous function (training isn't guaranteed to find them) Dead ReLU: always negative, slope 0, stops learning; fixes: sensible learning rates, leaky ReLU 📚 Sources McCulloch & Pitts, a logical calculus of nervous activity (1943): https://doi.org/10.1007/BF02478259 Rosenblatt, the perceptron (1958): https://doi.org/10.1037/h0042519 Minsky & Papert, Perceptrons (1969), MIT Press Cybenko, approximation by superpositions of a sigmoid (1989): https://doi.org/10.1007/BF02551274 Hornik, Stinchcombe & White, universal approximators (1989): https://doi.org/10.1016/0893-6080(89)... Nair & Hinton, rectified linear units (2010): https://mlanthology.org/icml/2010/nai... Glorot, Bordes & Bengio, deep sparse rectifier networks (2011): https://proceedings.mlr.press/v15/glo... Maas, Hannun & Ng, leaky ReLU (2013): https://ai.stanford.edu/~amaas/papers... Hendrycks & Gimpel, GELU (2016): https://arxiv.org/abs/1606.08415 Goodfellow, Bengio & Courville, Deep Learning, ch. 6: https://www.deeplearningbook.org/cont... 📚 Series: From a Straight Line to ChatGPT AI07 Decision Trees, Random Forests & Boosting AI08 Learning Without Labels: Clustering, PCA & Nearest Neighbors AI09 Neurons & Neural Networks ← you are here AI10 Backpropagation: How Networks Learn … full playlist:    • Artificial Intelligence (AI)   Just Known: from unknown to known. A channel by Just Unknown. Note: the spam messages and handwritten digits are real data; the XOR points, the two rings, the wiggly curve and the 100 random layers are made-up data. Pictures marked "illustration" are not data. #ai #deeplearning #neuralnetworks #python #computerscience #codinginterview