Introduction
The Sigmoid Activation Function is one of the most popular activation functions in neural networks. It converts any input value into a probability value between 0 and 1.
Because of its probabilistic output, the Sigmoid function is widely used in binary classification problems.
What is the Sigmoid Activation Function?
The Sigmoid Function is a non-linear activation function that maps any real number to a value between 0 and 1.
In simple terms:
The Sigmoid function converts numbers into probabilities.
Formula
f(x) = 1 / (1 + e⁻ˣ)where:
- x = input
- e = Euler's constant (≈ 2.718)
Output Range
0 < f(x) < 1Graph of Sigmoid Function
The graph forms an S-shaped curve.
How Does the Sigmoid Function Work?
| Input (x) | Output f(x) |
|---|---|
| -5 | 0.0067 |
| -2 | 0.119 |
| 0 | 0.5 |
| 2 | 0.881 |
| 5 | 0.993 |
Example
Input:
x = 0
Output:
f(0) = 0.5
This means the model is 50% confident.
Input:
x = 5
Output:
f(5) ≈ 0.993
This means the model is highly confident.
Why is Sigmoid Important?
The Sigmoid function:
- Introduces non-linearity.
- Produces probability values.
- Helps in binary classification.
- Makes predictions interpretable.
Derivative of Sigmoid Function
f'(x) = f(x)(1 - f(x)) This property makes gradient computation easier during backpropagation.
Working of Sigmoid Activation
Input↓
Sigmoid Function
↓
Probability (0 to 1)
↓
Prediction
Example: Email Spam Detection
Output:
0.95 → Spam
0.10 → Not Spam
Example: Disease Prediction
Output:
0.85 → Disease Present
0.20 → Disease Not Present
Advantages of Sigmoid Function
- Produces probabilities.
- Smooth and differentiable.
- Easy to understand.
- Useful for binary classification.
- Simple mathematical formulation.
Limitations of Sigmoid Function
- Suffers from the Vanishing Gradient Problem.
- Computationally expensive because of exponential calculations.
- Output is not zero-centered.
- Training can become slow in deep networks.
Vanishing Gradient Problem
For very large positive or negative inputs:
- Gradient becomes very small.
- Weight updates become tiny.
- Training slows down.
This is one reason why modern networks prefer ReLU.
Applications of Sigmoid Function
| Application | Usage |
|---|---|
| Spam Detection | Binary Classification |
| Disease Prediction | Binary Classification |
| Fraud Detection | Binary Classification |
| Customer Churn Prediction | Binary Classification |
| Logistic Regression | Output Layer |
Real-World Examples
- Email Spam Detection
- Credit Card Fraud Detection
- Disease Diagnosis
- Customer Churn Prediction
- Loan Approval Systems
Sigmoid vs Linear
| Feature | Linear | Sigmoid |
|---|---|---|
| Output Range | (-∞,∞) | (0,1) |
| Non-Linearity | No | Yes |
| Probability Output | No | Yes |
| Binary Classification | No | Yes |
Sigmoid vs ReLU
| Feature | Sigmoid | ReLU |
|---|---|---|
| Output Range | (0,1) | [0,∞) |
| Vanishing Gradient | Yes | Less |
| Hidden Layers | Rarely Used | Widely Used |
| Binary Classification | Yes | No |
When Should You Use Sigmoid?
Use Sigmoid:
- In the output layer of binary classification models.
- When probabilities are required.
- For Logistic Regression.
Avoid using it in deep hidden layers.
Best Practices
- Use Sigmoid only for binary classification output layers.
- Use ReLU or GELU in hidden layers.
- Monitor for vanishing gradients.
Interview Tip
A common interview question is:
"Why is the Sigmoid Activation Function used in binary classification?"
A strong answer is:
Because the Sigmoid function produces outputs between 0 and 1, which can be interpreted as probabilities for two classes.
Conclusion
The Sigmoid Activation Function is one of the foundational activation functions in Deep Learning. Although it is less commonly used in hidden layers due to the vanishing gradient problem, it remains extremely important for binary classification problems because of its probabilistic output