Introduction

An activation function is a mathematical function applied to a neuron's output, determining whether and how strongly that neuron "fires" before passing its signal to the next layer. Without activation functions, a neural network — no matter how many layers it had — would behave exactly like a single linear model, since stacking purely linear operations still only produces a linear result.

Activation functions introduce the non-linearity that allows neural networks to learn complex, real-world patterns, making them one of the most fundamental design choices in building any deep learning model.

Why are Activation Functions Important?

Activation functions help to:

• Introduce non-linearity, enabling networks to learn complex patterns
• Determine which neurons should "activate" and pass information forward
• Control the range and behavior of values flowing through the network
• Affect how efficiently gradients flow during backpropagation
• Enable networks to approximate virtually any complex function
• Shape a network's training stability and convergence speed

Where Activation Functions Fit

Whiteboard

Loading diagram...

Why Non-Linearity Matters

Without activation functions:
Layer 1: y = w1x + b1
Layer 2: z = w2y + b2  →  z = w2(w1x + b1) + b2  →  still just a linear function of x

No matter how many layers are stacked, the result remains linear —
activation functions are what allow the network to model curves,
not just straight lines.

Common Activation Functions

1. Sigmoid

Squashes input values into a range between 0 and 1, historically popular but now less common in hidden layers.

sigmoid(x) = 1 / (1 + e^-x)

2. Tanh (Hyperbolic Tangent)

Similar to sigmoid but outputs values between -1 and 1, centered around zero.

tanh(x) = (e^x - e^-x) / (e^x + e^-x)

3. ReLU (Rectified Linear Unit)

Outputs the input directly if positive, otherwise outputs zero — the most widely used activation function in modern deep learning.

ReLU(x) = max(0, x)

4. Leaky ReLU

A variation of ReLU that allows a small, non-zero gradient when the input is negative, helping avoid "dead" neurons.

Leaky ReLU(x) = x if x > 0, else 0.01x

5. Softmax

Converts a vector of raw scores into probabilities that sum to 1, commonly used in the output layer for multi-class classification.

softmax(xi) = e^xi / Σ(e^xj)

Choosing an Activation Function by Layer Type

The Vanishing Gradient Problem

Sigmoid and tanh functions can produce very small gradients for large or small input values, causing earlier layers in a deep network to learn extremely slowly — a key reason ReLU became the standard choice for hidden layers.

As sigmoid/tanh inputs grow large (positive or negative),
their gradient approaches zero, slowing or stalling learning
in the earlier layers of a deep network.

Sigmoid vs ReLU

Key Properties of Activation Functions

• Activation functions introduce the non-linearity essential for learning complex patterns.
• ReLU is the most common choice for hidden layers due to its simplicity and efficiency.
• Sigmoid and softmax are typically reserved for output layers, producing probability-like values.
• Poor activation function choice can contribute to problems like vanishing or exploding gradients.
• Different activation functions affect both training speed and final model performance.

Where are Activation Functions Used?

Advantages

• Enables neural networks to model complex, non-linear relationships in data
• Different functions can be chosen to suit specific layer roles and problem types
• Modern functions like ReLU are computationally efficient and easy to implement
• Helps control the flow and stability of gradients during training
• Softmax provides an intuitive probability interpretation for classification outputs

Limitations

• Sigmoid and tanh are prone to the vanishing gradient problem in deep networks
• Standard ReLU can suffer from "dying ReLU," where neurons get stuck outputting zero permanently
• Choosing the wrong activation function can slow or destabilize training
• Some newer activation functions (e.g., GELU, Swish) add computational complexity for marginal gains
• No single activation function works best for every architecture or problem type

Real-World Examples

Best Practices

• Default to ReLU for hidden layers unless there's a specific reason to choose otherwise.
• Use sigmoid for binary classification output layers and softmax for multi-class output layers.
• Consider Leaky ReLU or similar variants if dying ReLU becomes a noticeable training issue.
• Avoid sigmoid/tanh in deep hidden layers due to vanishing gradient risk.
• Research the activation functions used in modern architectures (like GELU in transformers) when working with cutting-edge models.

Interview Tip

A common interview question is:

"Why are activation functions necessary in neural networks, and why is ReLU commonly preferred over sigmoid?"

A strong answer is:

Activation functions are necessary because, without them, stacking multiple layers in a neural network would still only produce a linear function, no matter how many layers were added — non-linearity is what allows networks to model complex, real-world patterns. ReLU is commonly preferred over sigmoid for hidden layers because it's computationally simpler, and it largely avoids the vanishing gradient problem that sigmoid suffers from, where gradients become extremely small for large or small input values, slowing down learning in deep networks.

Explaining why linear layers alone aren't enough makes your answer stronger.

Conclusion

Activation functions are what give neural networks the ability to learn complex, non-linear patterns, transforming simple weighted sums into a rich, expressive modeling system. Understanding common choices like ReLU, sigmoid, and softmax — along with issues like vanishing gradients — provides essential context before exploring loss functions, which work alongside activation functions to guide how a network learns.