Introduction

Modern datasets often contain a large number of features. For example, a house price dataset may contain features such as area, number of bedrooms, age of the property, location score, number of bathrooms, parking spaces, and many more. Similarly, image datasets may contain thousands of pixel values, while genomic datasets may contain thousands of gene measurements.

Although having more features provides more information, it also introduces several challenges:

• Increased computational complexity

• Higher storage requirements

• Longer training times

• Risk of overfitting

• Difficulty in visualization

• Curse of Dimensionality

One solution to these problems is Dimensionality Reduction, which aims to reduce the number of features while retaining as much useful information as possible.

Among all dimensionality reduction techniques, Principal Component Analysis (PCA) is one of the most widely used and important methods.

However, before learning PCA mathematically, it is essential to understand the intuition behind it.

In this article, we will focus entirely on the intuition of PCA, understand why it was developed, examine the problem it solves, and build a strong conceptual foundation before exploring the mathematical details.

Why Do We Need PCA?

Consider a dataset containing information about students.

Notice something interesting.

Students who perform well in Mathematics also tend to perform well in Physics.

Similarly:

Students with lower Mathematics scores also tend to have lower Physics scores.

This indicates that the two features contain similar information.

In machine learning terms:

Features Are Correlated

When multiple features contain similar information, the dataset becomes redundant.

PCA helps remove this redundancy.

Understanding Redundant Information

Suppose we have the following features:

• Height in centimeters

• Height in inches

These two columns essentially describe the same information.

Adding both features does not provide much additional knowledge.

Similarly, many real-world datasets contain features that are strongly related to one another.

Examples include:

Such relationships create redundancy.

PCA attempts to capture the essential information using fewer dimensions.

Visualizing Data in Two Dimensions

Imagine a simple dataset with two features:

• Height

• Weight

If we plot the data points on a graph, we may observe something like:

•
    •
       •
          •
             •

The points form a diagonal pattern.

Although the data exists in two dimensions:

Height

Weight

most of the information lies along a single direction.

This observation forms the basis of PCA.

The Central Idea of PCA

The main idea behind PCA is:

Find The Direction
Containing Maximum Information

Instead of looking at the original axes:

X-Axis

Y-Axis

PCA searches for a new axis that best represents the data.

This new axis captures the greatest variation among observations.

Understanding Variance

To understand PCA, we must understand variance.

Variance measures:

How Much Data Points Vary

Consider two datasets.

Dataset A:

10

11

10

9

10

Dataset B:

1

20

5

25

15

Dataset B shows greater spread.

Therefore:

Higher Variance

PCA assumes that directions with higher variance contain more useful information.

Why Variance Matters

Suppose all students in a class score exactly:

80

in Mathematics.

The feature provides no ability to distinguish students.

Its variance is:

Zero

Now suppose scores range from:

40 To 100

The feature becomes informative because it helps differentiate students.

PCA prioritizes directions containing higher variance.

Finding the Best Direction

Consider a two-dimensional dataset.

The original axes are:

Feature 1

Feature 2

PCA asks:

Can We Rotate
The Coordinate System

to find a better representation?

Instead of measuring variation along the original axes, PCA identifies a new direction where the data spreads the most.

This direction becomes:

First Principal Component

First Principal Component

The First Principal Component is:

The Direction
With Maximum Variance

This direction captures the largest amount of information in the dataset.

Imagine rotating a line through the dataset.

The best line is the one along which the data points spread out the most.

That line becomes the first principal component.

Understanding PCA Through Photography

Imagine taking a photograph of a three-dimensional object.

The object exists in:

3 Dimensions

but the photograph captures only:

2 Dimensions

Some information is inevitably lost.

However, if you choose the camera angle carefully, you can preserve most of the important structure.

PCA performs a similar task.

It searches for the viewing angle that preserves the maximum information.

Second Principal Component

After finding the first principal component, PCA identifies another direction.

This direction must satisfy two conditions:

• Capture maximum remaining variance

• Be perpendicular to the first component

This becomes:

Second Principal Component

The second component explains information not already captured by the first component.

Principal Components Are New Features

A common misconception is that PCA selects existing features.

PCA does not choose original columns.

Instead, it creates:

New Features

called principal components.

These new features are combinations of the original variables.

For example:

PC1 = Combination Of Feature1 And Feature2

PC2 = Another Combination Of Feature1 And Feature2

The new features often capture the data more efficiently than the original features.

Dimensionality Reduction Using PCA

Suppose a dataset contains:

100 Features

After applying PCA, we may discover that:

95% Of Information

can be captured using only:

20 Principal Components

The transformation becomes:

100 Features
       ↓
PCA
       ↓
20 Features

This significantly reduces complexity.

Information Preservation

A common question is:

Does PCA Lose Information?

The answer is:

Yes

unless all principal components are retained.

However, PCA attempts to preserve as much information as possible.

For example:

The goal is to find a balance between simplicity and information retention.

Explained Variance

PCA measures how much information each component captures.

This is known as:

Explained Variance

Example:

The first two components explain:

85%

of the dataset's information.

This suggests that the remaining components contribute relatively little.

PCA for Visualization

One of the most common applications of PCA is visualization.

Suppose a dataset contains:

500 Features

Humans cannot visualize such a dataset directly.

PCA can reduce it to:

2 Dimensions

or

3 Dimensions

allowing patterns and clusters to be explored visually.

Real-World Example: Image Compression

Consider grayscale images of size:

100 × 100

Each image contains:

10,000 Pixels

Using PCA:

10,000 Features
        ↓
PCA
        ↓
500 Components

Most visual information can still be preserved.

This reduces storage requirements significantly.

Real-World Example: Customer Analytics

Suppose a business collects:

• Age

• Income

• Spending Score

• Number of Purchases

• Website Activity

• Product Preferences

Many of these features may be correlated.

PCA can combine related information into a smaller number of components.

This simplifies analysis while preserving important patterns.

Advantages of PCA

Reduces Dimensionality

PCA reduces the number of features while preserving useful information.

Removes Redundancy

Highly correlated features can be represented more efficiently.

Faster Model Training

Fewer features lead to reduced computational requirements.

Reduces Overfitting

Removing unnecessary dimensions can improve generalization.

Enables Visualization

High-dimensional datasets can be visualized in two or three dimensions.

Limitations of PCA

Information Loss

Some information is lost during dimensionality reduction.

Reduced Interpretability

Principal components are combinations of features and may be difficult to interpret.

Assumes Linear Relationships

PCA captures linear patterns but may miss complex nonlinear structures.

Sensitive to Feature Scaling

Features must usually be standardized before applying PCA.

Variance May Not Always Mean Importance

High variance does not necessarily imply high predictive value.

PCA vs Feature Selection

Many beginners confuse PCA with feature selection.

The two approaches are different.

PCA is a feature extraction technique rather than a feature selection technique.

When Should PCA Be Used?

PCA is particularly useful when:

• The dataset contains many features.

• Features are highly correlated.

• Visualization is required.

• Model training is slow due to dimensionality.

• Overfitting is caused by excessive features.

It may not be necessary when the dataset already contains a small number of independent features.