Introduction

Chain of Thought (CoT) prompting is a technique that encourages a model to work through a problem step by step, explicitly generating intermediate reasoning before arriving at a final answer, rather than jumping directly to a conclusion. Simply adding a phrase like "let's think step by step," or providing examples that demonstrate step-by-step reasoning, can dramatically improve accuracy on tasks involving arithmetic, logic, or multi-step problem solving — exactly the kinds of tasks covered in the earlier Reasoning Limitations topic.

Chain of Thought prompting doesn't change anything about the underlying model — it simply changes how the model is asked to respond, taking advantage of the fact that generating explicit intermediate steps gives the model's own next-token prediction process more "space" to work through a problem correctly, rather than trying to leap directly to a final answer in one shot.

Why Does Chain of Thought Prompting Matter?

Chain of Thought prompting helps to:

• Improve accuracy on multi-step reasoning, arithmetic, and logic problems
• Make a model's reasoning process visible and easier to evaluate or debug
• Reduce the risk of skipping necessary intermediate steps in complex problems
• Directly address some of the reasoning limitations covered in an earlier topic
• Provide a simple, low-cost technique requiring no model retraining
• Combine effectively with few-shot prompting for even stronger results

Why Chain of Thought Works: Connecting Back to Next Token Prediction

Whiteboard

As covered in the Next Token Prediction topic, a model generates
one token at a time, with each new token conditioned on everything
generated so far — including its own prior output.

When asked to jump straight to a final answer, the model has to
implicitly perform all necessary reasoning "invisibly" within a
single forward pass, with no intermediate results to build on.

When asked to reason step by step, each intermediate conclusion
becomes part of the visible context for generating the next step
— meaning the model can build on its own explicit prior reasoning,
rather than needing to solve the whole problem in one leap.

Zero-Shot Chain of Thought

The simplest form of CoT prompting requires no examples at all — just a phrase encouraging step-by-step reasoning.

Prompt:
"A store had 120 apples. They sold 45 in the morning and 38
in the afternoon. How many apples are left? Let's think step
by step."

Model Response:
"Step 1: Start with 120 apples.
Step 2: Subtract morning sales: 120 - 45 = 75.
Step 3: Subtract afternoon sales: 75 - 38 = 37.
The store has 37 apples left."

Simply appending "let's think step by step" (or similar phrasing) is often enough to trigger this more careful, explicit reasoning process, even without any examples demonstrating the pattern.

Few-Shot Chain of Thought

Combining CoT with few-shot prompting (covered in an earlier topic) provides even stronger guidance, by explicitly demonstrating the step-by-step reasoning pattern before presenting the new problem.

Prompt:
"Q: A bakery made 24 cupcakes and sold 3/4 of them. How many
are left?
A: 24 cupcakes total. 3/4 of 24 is 18, so 18 were sold.
24 - 18 = 6 cupcakes remain. The answer is 6.

Q: A parking lot has 80 cars. 25% left in the first hour and
another 30 left in the second hour. How many cars remain?
A:"

The example demonstrates not just the correct final answer, but
the full reasoning process the model should follow to reach it.

A Direct Comparison: With and Without CoT

Without CoT:
"What is 17% of 240, minus 12?"
→ Model may attempt to jump directly to a number, with a
  higher chance of arithmetic error

With CoT:
"What is 17% of 240, minus 12? Let's think step by step."
→ "17% of 240 = 0.17 × 240 = 40.8
    40.8 - 12 = 28.8
    The answer is 28.8"

Breaking the calculation into explicit steps reduces the
chance of an error compounding silently within a single,
unexplained final answer.

Where Chain of Thought Helps Most

Where Chain of Thought Adds Less Value

CoT prompting tends to add less benefit for:

- Simple factual recall questions (e.g., "What is the capital
  of France?") that don't require any actual reasoning process
- Tasks where the model already performs reliably well without it
- Cases where response length/cost is a higher priority than
  the marginal accuracy improvement CoT might provide

Applying CoT unnecessarily to simple tasks mostly just adds
extra tokens and latency without meaningfully improving results.

Chain of Thought vs Standard (Direct-Answer) Prompting

Key Properties of Chain of Thought Prompting

• CoT prompting encourages a model to generate explicit intermediate reasoning before its final answer.
• It can be applied in zero-shot form (simply requesting step-by-step reasoning) or reinforced with few-shot examples.
• CoT improves accuracy by giving the model's own next-token prediction process visible intermediate results to build on.
• The technique is most valuable for multi-step reasoning, arithmetic, and logic tasks, and less useful for simple factual recall.
• CoT increases token usage and response length compared to direct-answer prompting.

Where Is Chain of Thought Prompting Used?

Advantages

• Significantly improves accuracy on multi-step reasoning and arithmetic tasks
• Makes a model's reasoning process transparent and easier to verify
• Requires no model retraining — purely a prompting technique
• Can be combined with few-shot prompting for even stronger results
• Helps directly address some of the reasoning limitations covered in an earlier topic

Limitations

• Increases token usage and response length, raising cost and latency
• Doesn't fully eliminate reasoning errors — mistakes can still occur within the visible steps themselves
• Adds unnecessary length for simple tasks that don't actually require step-by-step reasoning
• Longer, more verbose responses may not always suit every application's user experience needs
• Still subject to the general hallucination risks covered in earlier topics

Real-World Examples

Best Practices

• Use CoT prompting specifically for multi-step, reasoning-heavy tasks, not simple factual queries.
• Start with the simple "let's think step by step" phrasing before investing in curated few-shot CoT examples.
• Combine CoT with few-shot examples when consistency and reliability matter more than simplicity.
• Review the generated reasoning steps themselves, not just the final answer, to catch errors early.
• Weigh the added token cost and latency against the accuracy benefit for your specific use case.

Interview Tip

A common interview question is:

"What is Chain of Thought prompting, and why does it improve accuracy on multi-step reasoning tasks?"

A strong answer is:

Chain of Thought prompting encourages a model to generate explicit intermediate reasoning steps before producing its final answer, rather than attempting to jump directly to a conclusion. It improves accuracy because language models generate text token by token, with each new token conditioned on everything generated so far — so when a model writes out its reasoning steps explicitly, each step becomes visible context that the next step can build on, rather than requiring the model to perform all necessary reasoning invisibly within a single generation. This is especially valuable for multi-step arithmetic and logic problems, where jumping directly to an answer increases the risk of compounding errors.

Connecting the explanation back to how next-token prediction actually works makes your answer stronger.

Conclusion

Chain of Thought prompting offers a simple but powerful way to improve a model's accuracy on complex, multi-step reasoning tasks, by making intermediate reasoning explicit rather than hidden within a single generation step. With this technique covered, the next topic explores self-consistency, which extends Chain of Thought further by generating multiple independent reasoning paths and selecting the most consistent answer among them.