Introduction
The Leaky ReLU (Leaky Rectified Linear Unit) is an improved version of the ReLU activation function.
It was introduced to solve one of ReLU's major problems called the Dying ReLU Problem, where neurons become inactive and stop learning.
Leaky ReLU allows a small amount of information to pass even when the input is negative.
What is Leaky ReLU?
The Leaky ReLU Activation Function keeps positive values unchanged and allows a small negative output for negative inputs.
In simple terms:
Leaky ReLU keeps positive values and leaks a small amount of negative values instead of making them zero.
Formula
f(x) = x, if x > 0f(x) = αx, if x ≤ 0where:
- α (alpha) is a small constant.
- Common value: α = 0.01
Output Range
(-∞, ∞) Graph of Leaky ReLU
Unlike ReLU, the negative side has a small slope instead of becoming zero.
How Does Leaky ReLU Work?
| Input (x) | Output f(x) |
|---|---|
| -5 | -0.05 |
| -2 | -0.02 |
| 0 | 0 |
| 3 | 3 |
| 8 | 8 |
(assuming α = 0.01)
Example
Input:
x = -4
Output:
f(x) = 0.01 × (-4) = -0.04
Input:
x = 5
Output:
f(x) = 5
Why is Leaky ReLU Important?
Leaky ReLU:
- Prevents neurons from dying.
- Allows gradients for negative inputs.
- Improves learning in deep networks.
- Provides better convergence than ReLU in some cases.
Derivative of Leaky ReLU
f'(x) = 1, if x > 0f'(x) = α, if x ≤ 0Working of Leaky ReLU
Input↓
Apply Leaky ReLU
↓
Positive Input → Same Value
Negative Input → Small Negative Value
Why is Leaky ReLU Better than ReLU?
ReLU:
Negative Input → 0Leaky ReLU:
Negative Input → Small Negative ValueTherefore, neurons continue learning even for negative inputs.
Advantages of Leaky ReLU
- Solves the Dying ReLU problem.
- Faster training.
- Computationally efficient.
- Works well in deep networks.
- Better gradient flow.
Limitations of Leaky ReLU
- The value of α must be chosen carefully.
- Not always better than ReLU.
- Outputs are not zero-centered.
Applications of Leaky ReLU
| Application | Usage |
|---|---|
| CNNs | Hidden Layers |
| Image Classification | Feature Learning |
| Deep Neural Networks | Hidden Layers |
| Computer Vision | Object Detection |
| NLP Models | Feature Extraction |
Real-World Examples
- Image Recognition
- Face Recognition
- Object Detection
- Medical Image Analysis
- Recommendation Systems
ReLU vs Leaky ReLU
| Feature | ReLU | Leaky ReLU |
|---|---|---|
| Negative Inputs | 0 | Small Negative Value |
| Dying ReLU Problem | Yes | Reduced |
| Gradient for x < 0 | 0 | α |
| Deep Networks | Good | Better in some cases |
Leaky ReLU vs Sigmoid
| Feature | Leaky ReLU | Sigmoid |
|---|---|---|
| Output Range | (-∞,∞) | (0,1) |
| Vanishing Gradient | Less | High |
| Training Speed | Fast | Slow |
When Should You Use Leaky ReLU?
Use Leaky ReLU:
- In deep neural networks.
- When many ReLU neurons become inactive.
- In CNN architectures.
- When training becomes unstable with ReLU.
Best Practices
- Start with α = 0.01.
- Monitor inactive neurons.
- Compare performance with ReLU and GELU.
- Use in hidden layers only.
Interview Tip
A common interview question is:
"Why was Leaky ReLU introduced?"
A strong answer is:
Leaky ReLU was introduced to solve the Dying ReLU Problem by allowing a small gradient for negative inputs, ensuring that neurons continue learning during training.
Conclusion
The Leaky ReLU Activation Function is an improved version of ReLU that addresses the Dying ReLU Problem. By allowing a small negative output, it improves gradient flow and helps deep neural networks train more effectively.