The AdaDelta Optimizer is an improvement over AdaGrad. It was introduced to solve AdaGrad's major problem of continuously decreasing learning rates.

Instead of storing all previous gradients, AdaDelta keeps track of only recent gradients using an exponentially decaying average.

This allows the optimizer to continue learning even during long training sessions.

What is AdaDelta?

AdaDelta is an adaptive learning rate optimization algorithm that dynamically adjusts learning rates based on recent gradients.

In simple terms:

AdaDelta automatically adapts the learning rate without allowing it to become extremely small.

Why Do We Need AdaDelta?

AdaGrad suffers from:

• Continuously decreasing learning rates.
• Slow convergence after many iterations.
• Training stopping too early.

AdaDelta solves these issues.

Working of AdaDelta

Initialize Weights        ↓
Compute Gradient
        ↓
Store Recent Gradients
        ↓
Compute Adaptive Update
        ↓
Update Weights
        ↓
     Repeat

Accumulated gradient:

Eg² = ρEg² + (1-ρ)g² 

Parameter update:

ΔW = - RMS(ΔW)t-1 / RMS(g)t × g

Weight update:

W = W + ΔW 

where:

• W = weights
• g = gradient
• ρ = decay factor
• RMS = Root Mean Square

How Does AdaDelta Work?

Recent Gradients
        ↓
Adaptive Learning Rate
        ↓
Parameter Update
        ↓
     Repeat 

Why is AdaDelta Better than AdaGrad?

AdaGrad:

Stores All Past Gradients        ↓
Learning Rate → Very Small

AdaDelta:

Stores Recent Gradients Only        ↓
Learning Rate Remains Stable

Example

Suppose the model has been training for thousands of iterations.

AdaGrad:

• Learning rate becomes extremely small.

AdaDelta:

• Continues adapting using recent gradient information.

Advantages of AdaDelta

• No manually chosen learning rate required.
• Solves AdaGrad's decay problem.
• Stable convergence.
• Works well for long training sessions.
• Adaptive parameter updates.

Limitations of AdaDelta

• More computationally expensive than SGD.
• May converge slower than Adam.
• Less commonly used today.

Applications of AdaDelta

Real-World Examples

• Image Recognition
• Text Classification
• Language Modeling
• Recommendation Systems
• Speech Recognition

AdaGrad vs AdaDelta

SGD vs AdaDelta

AdaDelta vs RMSProp

When Should You Use AdaDelta?

Use AdaDelta when:

• Training lasts for many iterations.
• AdaGrad becomes too slow.
• Adaptive learning rates are needed.
• Manual tuning should be minimized.

Best Practices

• Start with default parameters.
• Monitor convergence speed.
• Compare with RMSProp and Adam.
• Use for adaptive optimization tasks.

Interview Tip

A common interview question is:

"Why was AdaDelta introduced?"

A strong answer is:

AdaDelta was introduced to solve AdaGrad's problem of continuously decreasing learning rates by using only recent gradients instead of accumulating all past gradients.

Conclusion

AdaDelta is an important adaptive optimization algorithm that improves upon AdaGrad by preventing learning rates from shrinking indefinitely. Although it is less popular today than Adam and RMSProp, it introduced key ideas that influenced modern optimization algorithms.