The Momentum Optimizer is an improvement over the standard SGD optimizer. It helps neural networks learn faster by remembering the direction of previous updates.

Instead of moving only according to the current gradient, Momentum also considers past gradients, making optimization smoother and faster.

What is the Momentum Optimizer?

The Momentum Optimizer accelerates Gradient Descent by adding a fraction of the previous update to the current update.

In simple terms:

Momentum remembers previous directions and uses them to move faster toward the minimum.

Why Do We Need Momentum?

SGD often suffers from:

• Slow convergence
• Oscillations
• Getting stuck in narrow valleys

Momentum helps solve these problems.

Working of Momentum Optimizer

 Initialize Weights        ↓
 Compute Gradient
        ↓
 Calculate Momentum
        ↓
  Update Weights
        ↓
     Repeat

Velocity update:

 V = βV − η(∂L/∂W)

Weight update:

 W = W + V

where:

• W = weights
• V = velocity
• η = learning rate
• β = momentum coefficient
• L = loss function

What is Velocity?

Velocity stores information about previous gradients.

It helps the optimizer:

• Move faster in the correct direction.
• Avoid unnecessary oscillations.

Common Value of β

Typically:

β = 0.9 

This means:

• 90% previous direction
• 10% current gradient

How Does Momentum Work?

Previous Updates        ↓
Current Gradient
        ↓
Combined Direction
        ↓
Faster Convergence

Suppose:

Current Gradient = 0.3

Previous Velocity = 0.5

Learning Rate = 0.01

Momentum uses both values to determine the next update.

Why is Momentum Faster?

Without Momentum:

 Gradient → Update

With Momentum:

Previous Direction        +
Current Gradient
        ↓
Larger Update

This allows the optimizer to move faster toward the minimum.

Advantages of Momentum Optimizer

• Faster convergence.
• Reduces oscillations.
• Better optimization.
• Works well for deep networks.
• Escapes shallow local minima.

Limitations of Momentum Optimizer

• Requires choosing β carefully.
• Can overshoot minima.
• Slightly more complex than SGD.

Applications of Momentum Optimizer

Real-World Examples

• Image Recognition
• Face Detection
• Object Detection
• Speech Recognition
• Recommendation Systems

SGD vs Momentum

Momentum vs Adam

When Should You Use Momentum?

Use Momentum when:

• SGD converges slowly.
• Training oscillates heavily.
• Deep neural networks are used.
• Faster optimization is required.

Best Practices

• Start with β = 0.9.
• Tune the learning rate carefully.
• Monitor convergence.
• Use with Mini-Batch Gradient Descent.

 Interview Tip

A common interview question is:

"Why is Momentum better than SGD?"

A strong answer is:

Momentum uses information from previous updates to accelerate learning and reduce oscillations, resulting in faster and more stable convergence than standard SGD.

Conclusion

The Momentum Optimizer improves upon SGD by remembering previous updates and using them to accelerate learning. It converges faster and more smoothly, making it a popular optimization technique in Deep Learning.