He Initialization, also known as Kaiming Initialization, is one of the most widely used weight initialization techniques in modern Deep Learning.

It was introduced by Kaiming He and his colleagues to improve the training of deep neural networks that use ReLU activation functions.

The technique initializes weights using a carefully calculated variance, helping maintain stable gradients during forward and backward propagation.

Today, He Initialization is the default choice for many CNNs and deep neural networks.

What is He Initialization?

He Initialization is a weight initialization technique specifically designed for neural networks using ReLU and its variants.

Instead of assigning arbitrary random values, it generates weights based on the number of input neurons.

In simple terms:

He Initialization gives ReLU-based neural networks an optimal starting point for learning.

Why Do We Need He Initialization?

ReLU activation sets all negative values to zero.

If weights are initialized too small:

• Many neurons become inactive.
• Learning slows down.

If weights are initialized too large:

• Activations become unstable.
• Gradients may explode.

He Initialization balances these effects by choosing an appropriate weight variance.

How Does He Initialization Work?

 Input Layer      ↓
Calculate Fan-in
      ↓
Generate Scaled Random Weights
      ↓
Stable Forward Pass
      ↓
Stable Backpropagation

Fan-in is the number of input connections to a neuron.

Example:

784 Inputs
     ↓
256 Hidden Neurons 

Fan-in = 784

He Initialization uses only fan-in to determine the variance of the weights.

He Initialization Formula

For a Normal Distribution:

Variance = 2 / fan_in 

For a Uniform Distribution:

W ~ U(-√(6/fan_in), √(6/fan_in))

where:

• fan_in = Number of input neurons

Example

Suppose:

• Fan-in = 256

Then,

 Variance = 2 / 256 = 0.0078125

The weights are randomly initialized using this variance.

This keeps activations and gradients stable during training.

Why Does He Initialization Work?

ReLU removes all negative outputs.

Without proper initialization:

• Some neurons may never activate.
• Gradients may become unstable.

He Initialization compensates for ReLU's behavior by assigning slightly larger initial weights than Xavier Initialization.

This allows information to flow efficiently through deep networks.

Best Activation Functions

He Initialization works best with:

• ReLU
• Leaky ReLU
• ELU
• PReLU
• GELU (commonly used in practice)

It is generally not preferred for Sigmoid or Tanh, where Xavier Initialization performs better.

He Initialization vs Xavier Initialization

He Initialization vs Random Initialization

Advantages

• Designed specifically for ReLU-based networks.
• Reduces vanishing gradients.
• Improves convergence speed.
• Supports very deep neural networks.
• Provides stable gradient flow.
• Widely used in modern Deep Learning frameworks.

Limitations

• Not ideal for Sigmoid or Tanh activations.
• Requires knowing the fan-in value.
• Does not completely eliminate gradient-related problems.

Applications

Real-World Example

Suppose you are training a ResNet-50 model for image classification.

If random initialization is used:

• Training may become unstable.
• Gradients may vanish or explode.
• Convergence may be slow.

With He Initialization:

• Activations remain balanced.
• Gradients flow efficiently.
• The model trains faster and achieves better performance.

This is why most modern computer vision models use He Initialization by default.

Best Practices

• Use He Initialization with ReLU and its variants.
• Combine it with Batch Normalization for improved stability.
• Choose an appropriate learning rate.
• Monitor training loss during early epochs.
• Use Xavier Initialization instead for Sigmoid or Tanh networks.

 Interview Tip

A common interview question is:

"Why is He Initialization preferred for ReLU?"

A strong answer is:

He Initialization uses a larger weight variance based on the number of input neurons, which compensates for the behavior of ReLU activation functions and helps maintain stable activations and gradients during training.

Another common question is:

"What is the difference between He Initialization and Xavier Initialization?"

Answer:

He Initialization is designed for ReLU-based activation functions and uses a variance of 2/fan_in, whereas Xavier Initialization is designed for Sigmoid and Tanh activations and uses a variance of 2/(fan_in + fan_out).

Conclusion

He (Kaiming) Initialization is the standard weight initialization technique for modern Deep Learning models that use ReLU and its variants. By assigning carefully scaled random weights based on the number of input neurons, it improves gradient flow, speeds up convergence, and enables the successful training of very deep neural networks.

It is widely used in CNNs, ResNets, object detection models, and many state-of-the-art computer vision architectures.