Introduction
The Tanh (Hyperbolic Tangent) Activation Function is a non-linear activation function widely used in neural networks. It converts input values into outputs ranging between -1 and 1.
Compared to the Sigmoid function, Tanh is often preferred because its outputs are zero-centered, which helps neural networks train more efficiently.
What is the Tanh Activation Function?
The Tanh Function maps any real number to a value between -1 and 1.
In simple terms:
Tanh compresses large positive and negative values into a limited range of -1 to 1.
Formula
tanh(x) = (eˣ − e⁻ˣ)/(eˣ + e⁻ˣ)where:
- x = input
- e = Euler's constant
Output Range
-1 < tanh(x) < 1Graph of Tanh Function
The graph forms an S-shaped curve centered around zero.
How Does Tanh Work?
| Input (x) | Output tanh(x) |
|---|---|
| -3 | -0.995 |
| -1 | -0.761 |
| 0 | 0 |
| 1 | 0.761 |
| 3 | 0.995 |
Example
Input:
x = 0
Output:
tanh(0) = 0
Input:
x = 2
Output:
tanh(2) ≈ 0.964
Input:
x = -2
Output:
tanh(-2) ≈ -0.964
Why is Tanh Important?
The Tanh function:
- Introduces non-linearity.
- Produces zero-centered outputs.
- Helps gradient updates become more balanced.
- Performs better than Sigmoid in many hidden layers.
Derivative of Tanh
tanh'(x) = 1 − tanh²(x)This derivative is used during backpropagation.
Working of Tanh Activation
Input↓
Tanh Function
↓
Output (-1 to 1)
↓
Prediction
Advantages of Tanh Function
- Non-linear activation.
- Zero-centered output.
- Faster convergence than Sigmoid.
- Suitable for hidden layers.
- Smooth and differentiable.
Limitations of Tanh Function
- Suffers from the Vanishing Gradient Problem.
- Computationally expensive.
- Less popular than ReLU in modern deep networks.
Vanishing Gradient Problem
For very large positive or negative inputs:
- The gradient becomes very small.
- Weight updates become tiny.
- Training slows down.
This is why deep networks often prefer ReLU and GELU.
Applications of Tanh Function
| Application | Usage |
|---|---|
| Hidden Layers | Neural Networks |
| RNNs | Sequence Modeling |
| LSTMs | Memory Cells |
| NLP Models | Text Processing |
| Time-Series Prediction | Forecasting |
Real-World Examples
- Language Translation
- Speech Recognition
- Text Generation
- Sentiment Analysis
- Time-Series Forecasting
Sigmoid vs Tanh
| Feature | Sigmoid | Tanh |
|---|---|---|
| Output Range | (0,1) | (-1,1) |
| Zero-Centered | No | Yes |
| Vanishing Gradient | Yes | Yes |
| Training Speed | Slower | Faster |
Tanh vs ReLU
| Feature | Tanh | ReLU |
|---|---|---|
| Output Range | (-1,1) | [0,∞) |
| Zero-Centered | Yes | No |
| Vanishing Gradient | Yes | Less |
| Deep Networks | Less Common | Widely Used |
When Should You Use Tanh?
Use Tanh:
- In hidden layers of smaller networks.
- In RNNs and LSTMs.
- When zero-centered outputs are beneficial.
Avoid using it in very deep networks because of vanishing gradients.
Best Practices
- Normalize input data.
- Use Tanh in sequence models.
- Monitor gradients during training.
- Consider ReLU or GELU for very deep networks.
Interview Tip
A common interview question is:
"Why is Tanh often preferred over Sigmoid?"
A strong answer is:
Tanh produces outputs between -1 and 1 and is zero-centered, which leads to more balanced gradient updates and often faster convergence than the Sigmoid function.
Conclusion
The Tanh Activation Function is a powerful non-linear activation function that improves upon Sigmoid by producing zero-centered outputs. Although it still suffers from the vanishing gradient problem, it remains important in recurrent neural networks and sequence modeling tasks.