Introduction

A partial derivative measures the rate of change of a multivariable function with respect to one variable, while holding all other variables constant. It extends the concept of an ordinary derivative to functions that depend on more than one input, making it essential for analyzing and optimizing systems with multiple variables.

Partial derivatives are the building blocks of the gradient, which is central to training machine learning models, especially in gradient-based optimization and backpropagation for neural networks.

Why are Partial Derivatives Important?

Partial derivatives help to:

• Measure how a multivariable function changes with respect to one variable at a time
• Form the components of the gradient vector used in optimization
• Enable gradient descent to work on models with many parameters
• Power backpropagation for training neural networks
• Analyze sensitivity of outputs to individual input variables
• Support optimization in engineering, economics, and physics problems

Computing a Partial Derivative

Whiteboard

Partial Derivative Notation

∂f/∂x   — derivative of f with respect to x, holding y constant
∂f/∂y   — derivative of f with respect to y, holding x constant

The symbol ∂ (called "del" or "partial") distinguishes partial derivatives from ordinary derivatives (d/dx).

Step-by-Step Example

Function: f(x, y) = 3x²y + 2y³

Find ∂f/∂x (treat y as a constant):

∂f/∂x = 6xy

Find ∂f/∂y (treat x as a constant):

∂f/∂y = 3x² + 6y²

Evaluate both at the point (x=2, y=1):

∂f/∂x = 6(2)(1) = 12
∂f/∂y = 3(2)² + 6(1)² = 12 + 6 = 18

The Gradient Vector

The gradient collects all partial derivatives of a function into a single vector, pointing in the direction of steepest increase:

∇f(x,y) = [ ∂f/∂x , ∂f/∂y ]

For the example above at (2,1):

∇f(2,1) = [12, 18]

In optimization, moving in the negative gradient direction leads toward the function's minimum — this is the core idea behind gradient descent.

Higher-Order and Mixed Partial Derivatives

Partial derivatives can be taken multiple times, including with respect to different variables:

∂²f/∂x²    — second partial derivative with respect to x
∂²f/∂x∂y   — mixed partial derivative (first with respect to y, then x)

Mixed partial derivatives are used in optimization to analyze curvature (via the Hessian matrix) and to classify critical points.

Ordinary Derivative vs Partial Derivative

Key Properties of Partial Derivatives

• All other variables are treated as constants when differentiating with respect to one variable.
• The gradient vector, made up of all partial derivatives, points in the direction of steepest ascent.
• Setting all partial derivatives to zero helps locate critical points (minima, maxima, saddle points).
• Mixed partial derivatives are often equal when the function is sufficiently smooth (Clairaut's theorem).
• Partial derivatives form the basis of the Jacobian and Hessian matrices used in optimization.

Where are Partial Derivatives Used?

Advantages

• Enables optimization of functions with many variables, essential for machine learning
• Provides detailed insight into how each individual variable affects the output
• Forms the foundation of the gradient, Jacobian, and Hessian used in advanced optimization
• Supports precise sensitivity analysis in scientific and engineering models
• Scales naturally to functions with any number of variables

Limitations

• Computationally more expensive as the number of variables increases
• Requires the function to be differentiable with respect to each variable
• Mixed partial derivatives can add complexity when analyzing curvature
• Interpreting partial derivatives in very high-dimensional spaces can be difficult
• Numerical computation of partial derivatives can introduce approximation errors

Real-World Examples

Best Practices

• Always clearly state which variable you are differentiating with respect to.
• Treat all other variables as constants during each partial differentiation.
• Use the gradient vector to represent all partial derivatives together for optimization.
• Leverage automatic differentiation tools (e.g., PyTorch, TensorFlow) for complex, high-dimensional functions.
• Check mixed partial derivatives when analyzing the curvature of a multivariable function.

Interview Tip

A common interview question is:

"What is a partial derivative, and how is it used in machine learning?"

A strong answer is:

A partial derivative measures the rate of change of a multivariable function with respect to one variable, while holding all other variables constant, using the notation ∂f/∂x. In machine learning, partial derivatives are computed for each model parameter and combined into a gradient vector, which points in the direction of steepest increase of the cost function. Gradient descent then uses the negative of this gradient to update parameters and minimize error, and backpropagation relies on partial derivatives, applied via the chain rule, to update weights throughout a neural network.

Mentioning the gradient vector and its role in gradient descent makes your answer stronger.

Conclusion

Partial derivatives extend the concept of a derivative to functions of multiple variables, measuring how output changes with respect to one variable at a time. As the building blocks of the gradient, they are essential to gradient descent, backpropagation, and optimization across machine learning, physics, economics, and engineering.