GPU & Compute Fundamentals

Modern Generative AI runs on an enormous amount of computation. Training a large model can involve more calculations than a human could do in many lifetimes — and the hardware that makes this possible is mainly the GPU. Understanding compute and GPUs explains why AI is so powerful, so expensive, and how it scales.

💡 In one line: AI needs massive parallel math, and GPUs — with thousands of cores working at once — are the hardware that delivers it.

Why Does AI Need So Much Compute?

A neural network is, at its heart, a giant pile of matrix multiplications — billions of multiply-and-add operations. Training makes this far heavier: the model processes huge datasets and adjusts billions of parameters, repeating the calculation millions of times. The total amount of math is staggering, which is why specialised hardware is essential.

CPU vs. GPU

The two main types of processor are built for different jobs:

• CPU (Central Processing Unit) — has a few very powerful cores. Brilliant at complex, sequential tasks done one after another.
• GPU (Graphics Processing Unit) — has thousands of simpler cores. Brilliant at doing the same operation on lots of data at once — that is, parallel work.

Neural-network math is exactly the kind of work that splits into thousands of identical small operations, so GPUs can do it far faster than CPUs. (GPUs were originally built for rendering graphics, which is also massively parallel — which is why they turned out to be perfect for AI.)

Remove

What is "Compute"?

Compute simply means the amount of calculation needed. It's measured in FLOPs — floating-point operations (the basic math operations like multiply and add).

• Training a large model takes a colossal total number of FLOPs.
• More compute means you can train bigger models on more data — which, per the scaling laws, tends to produce better models.

This is why compute is treated as a precious, strategic resource in AI.

Key Hardware Terms

Training vs. Inference Compute

• Training needs massive, sustained compute — often clusters of hundreds or thousands of GPUs running for weeks.
• Inference needs less compute per request, but it runs constantly for every user — so it still requires capable GPUs, and at scale its total compute can be huge.

Memory Matters (VRAM)

A model's parameters must fit in the GPU's memory (VRAM) to run. Roughly, at half precision a parameter takes ~2 bytes, so:

• A 7-billion-parameter model needs about 14 GB of VRAM just to load.
• Very large models don't fit on one GPU and must be split across many.

This is why quantization (storing parameters at lower precision) is so useful — it shrinks memory needs and lets bigger models run on smaller hardware.

Why It's Expensive

• GPUs are costly and power-hungry.
• Training frontier models can cost millions of dollars in compute and electricity.
• Scaling laws mean better models generally demand even more compute.

Compute is often the single biggest bottleneck — and cost — in building large AI systems.

Reducing Compute Needs

Common ways to do more with less:

• Quantization — lower-precision parameters (less memory, faster).
• Smaller / efficient models — right-size to the task.
• Mixed precision — use lower precision where accuracy allows.
• Batching — process many inputs together for efficiency.

Summary

• AI is built on massive parallel computation, mostly matrix multiplications.
• CPUs have a few powerful cores (sequential); GPUs have thousands of small cores (parallel) — ideal for AI.
• Compute is measured in FLOPs; more compute enables bigger, better models.
• A model must fit in GPU memory (VRAM) — a 7B model needs ~14 GB — so big models span many GPUs.
• Compute is expensive and power-hungry, and techniques like quantization and batching help reduce the load.