Stochastic Gradient Descent (SGD) is one of the most widely used optimization algorithms in Machine Learning and Deep Learning.

Unlike Batch Gradient Descent, which uses the entire dataset, SGD updates the model parameters using only one training sample at a time.

This makes SGD much faster and suitable for large datasets.

What is Stochastic Gradient Descent?

Stochastic Gradient Descent (SGD) is an optimization algorithm that updates model parameters after processing each individual training example.

In simple terms:

SGD uses one training sample to calculate the gradient and immediately updates the weights.

Why Do We Need SGD?

SGD helps to:

• Train large datasets efficiently.
• Reduce computation time.
• Update weights frequently.
• Learn faster than Batch Gradient Descent.

Working of SGD

 Initialize Weights        ↓
Pick One Training Sample
        ↓
 Calculate Gradient
        ↓
  Update Weights
        ↓
 Pick Next Sample
        ↓
     Repeat

Step 1: Initialize Weights

Start with random values.

Step 2: Select One Training Sample

Choose one example from the dataset.

Step 3: Calculate Prediction

Compute the model output.

Step 4: Calculate Error

Find the prediction error.

Step 5: Compute Gradient

Determine how much the weights should change.

Step 6: Update Weights

Update the model parameters immediately.

Step 7: Repeat

Continue for all training samples.

Mathematical Representation

Weight update equation:

W = W − η (∂Lᵢ/∂W)

where:

• W = weights
• η = learning rate
• Lᵢ = loss for one training sample
• ∂Lᵢ/∂W = gradient

Example

Suppose we have:

SGD processes:

1st sample → Update weights

2nd sample → Update weights

3rd sample → Update weights

4th sample → Update weights

Why is it Called "Stochastic"?

The word Stochastic means random.

The algorithm randomly selects training samples and updates the weights continuously.

Visualization

Sample 1 → UpdateSample 2 → Update
Sample 3 → Update
Sample 4 → Update

• Faster training.
• Lower memory requirements.
• Suitable for large datasets.
• Frequent updates help escape local minima.
• Widely used in Deep Learning.

Limitations of SGD

• Training is noisy.
• Loss function fluctuates.
• May take longer to converge.
• Sensitive to learning rate selection.

Applications of SGD

Real-World Example

Training ChatGPT or image classification models with millions of samples using Batch Gradient Descent would be extremely slow.

SGD makes training possible by updating parameters after each sample.

Batch Gradient Descent vs SGD

SGD vs Mini-Batch Gradient Descent

When Should You Use SGD?

Use SGD when:

• Dataset is large.
• Memory is limited.
• Faster updates are required.
• Online learning is needed.

Avoid SGD when extremely stable convergence is required.

Best Practices

• Shuffle the training data.
• Choose an appropriate learning rate.
• Use learning rate scheduling.
• Monitor training loss.
• Consider Mini-Batch GD for better stability.

 Interview Tip

A common interview question is:

"Why is SGD faster than Batch Gradient Descent?"

A strong answer is:

SGD updates the weights after processing each individual training sample instead of the entire dataset, resulting in faster updates and lower memory requirements.

Conclusion

Stochastic Gradient Descent is one of the most important optimization algorithms in Deep Learning. By updating weights after every training example, it enables efficient training on large datasets and forms the foundation for many advanced optimizers used in modern AI systems.