Introduction
Activation functions are mathematical functions that determine the output of a neuron. One of the simplest activation functions is the Linear Activation Function.
It directly passes the input to the output without introducing any non-linearity.
Although it is simple, the Linear Activation Function plays an important role in regression problems.
What is a Linear Activation Function?
The Linear Activation Function returns the input value directly as the output.
In simple terms:
Output is equal to the input.
Formula
f(x) = xor
f(x) = axwhere:
- x = input
- a = constant (usually 1)
Graph of Linear Activation Function
Input ↑/
/
/
/
Output →
The graph is simply a straight line.
How Does the Linear Activation Function Work?
Suppose:
Input = 5
Then:
Output = 5
If:
Input = -3
Then:
Output = -3
The function does not modify the input.
Example
| Input (x) | Output f(x) |
|---|---|
| -2 | -2 |
| 0 | 0 |
| 3 | 3 |
| 5 | 5 |
Why Do We Need Activation Functions?
Without activation functions, neural networks cannot learn complex relationships.
The Linear Activation Function itself does not introduce non-linearity.
Therefore, deep networks using only linear activations behave like a single-layer model.
Characteristics of Linear Activation Function
- Simple mathematical function.
- Output can take any value.
- Differentiable everywhere.
- No non-linearity.
Derivative of Linear Function
f'(x) = 1The derivative is constant.
Advantages
- Very simple to compute.
- Fast execution.
- Useful for regression problems.
- No vanishing gradient issue.
Limitations
- Cannot learn complex patterns.
- Cannot solve non-linear problems.
- Makes deep networks ineffective.
- Rarely used in hidden layers.
Applications of Linear Activation Function
| Application | Usage |
|---|---|
| House Price Prediction | Output Layer |
| Stock Price Prediction | Output Layer |
| Temperature Prediction | Output Layer |
| Sales Forecasting | Output Layer |
| Regression Models | Output Layer |
Real-World Example
Suppose a neural network predicts house prices.
Output:
Predicted Price = ₹52,00,000
Since the output can be any numerical value, the Linear Activation Function is suitable.
Linear vs ReLU
| Feature | Linear | ReLU |
|---|---|---|
| Non-Linearity | No | Yes |
| Complex Pattern Learning | No | Yes |
| Used in Hidden Layers | Rarely | Frequently |
| Regression Problems | Yes | No |
Linear vs Sigmoid
| Feature | Linear | Sigmoid |
|---|---|---|
| Output Range | (-∞, ∞) | (0,1) |
| Classification | No | Yes |
| Regression | Yes | No |
When Should You Use Linear Activation?
Use Linear Activation when:
- Solving regression problems.
- Predicting continuous values.
- Building output layers for regression models.
Avoid using it in hidden layers.
Best Practices
- Use Linear Activation only in output layers for regression.
- Use ReLU or GELU in hidden layers.
- Avoid building deep networks using only linear activations.
Interview Tip
A common interview question is:
"Why is the Linear Activation Function not used in hidden layers?"
A strong answer is:
Because it does not introduce non-linearity. If all layers use linear activation, the entire neural network behaves like a single-layer model and cannot learn complex patterns.
Conclusion
The Linear Activation Function is one of the simplest activation functions in Deep Learning. Although it is not suitable for hidden layers, it is widely used in regression problems where the output can take any continuous value.