For a long time, interacting with a Large Language Model (LLM) was surprisingly direct: you asked a question, and the model started generating words almost immediately. This fast, intuitive process was impressive for language generation, but fell short when it came to complex logic or mathematics. Models "spoke before they thought".
With the introduction of reasoning models, such as the OpenAI o1 series and DeepSeek-R1, this paradigm has changed. These models pause, build an internal chain of thought, and only provide an answer once they have mentally dissected the problem. In this article, we discuss how this works, what it costs in terms of latency and tokens, and for which specific tasks this actually makes a measurable difference.
The Shift: System 1 to System 2 Thinking
To understand what makes reasoning models different, we can borrow the well-known theory of psychologist Daniel Kahneman. He describes the human brain in two systems:
- System 1: Fast, instinctive, and automatic (for example, recognizing a face or speaking in your native language).
- System 2: Slow, deliberate, and logical (for example, solving a complex mathematical equation).
Regular LLMs are typical System 1 thinkers. They predict the next word in a sequence at lightning speed. If the task is complex and there is no obvious answer in the training data, the model quickly starts to hallucinate. After all, it has to respond immediately.
Reasoning models introduce System 2 thinking. Through advanced Reinforcement Learning techniques, these models are trained to map out a solution path first. They try different approaches, recognize their own mistakes (self-correction), and only when they are convinced of the logic do they start formulating the output for the user. If you want to brush up on the absolute basics of LLMs, read what an LLM actually is first.
The Hidden Costs: Latency and Reasoning Tokens
The magic of System 2 thinking is not free. It requires significantly more computing power per prompt, and that translates directly into two major pain points: time and money.
1. Latency (Waiting Time)
With a standard LLM, text starts appearing on your screen after a few milliseconds (Time to First Token, TTFT). With a reasoning model, this can take 10 to even 45 seconds, depending on the complexity of your question. For background processes or complex analyses, this is acceptable. However, for a real-time customer service chatbot, such a delay is disastrous for the user experience.
2. Reasoning Tokens
During the "thinking phase," the model actually generates text (the chain of thought), even though you often do not see it fully in your application. These internal deliberations are called reasoning tokens.
This has a major impact on costs. If you use an API, you pay not only for your input (the prompt) and the visible output, but also for every generated reasoning token. A short and concise final answer may be preceded by thousands of invisible reasoning tokens, making the price per request much higher than you might initially expect.
Important assumption: In this article, we assume that API providers maintain the current billing model where generated reasoning tokens are charged at the same rate as regular output tokens. Keep an eye on your provider's documentation, as AI pricing models change rapidly.
The 'Sweet Spot': Where Reasoning Models Excel
You should think of these models as expensive, thoughtful experts. You don't hire them for simple production work, but for complex, multi-step problems. For the following tasks, the extra cost in tokens and latency is well justified:
Programming and Architecture
For writing a simple Python script, any standard language model will do. But as soon as you deal with complex refactoring, designing database architectures, or debugging multi-threading issues in C++, a reasoning model shines. The model will first map out the dependencies and think through potential edge cases before writing any code. For a complete overview of suitable solutions for this, see our article on models for code.
Mathematics and Scientific Analysis
Reasoning models structurally achieve better (and measurable) results on academic benchmarks such as the AIME (American Invitational Mathematics Examination). They generate proofs step-by-step, check their own intermediate steps, and adjust their strategy if a calculation is about to get stuck.
Strategic Planning and Logic
For tasks that require abstract reasoning capabilities, such as drafting a complex legal argument, working out a chess strategy, or optimizing logistical routes. The internal thinking space gives the model the opportunity to "look ahead" at the consequences of certain choices.
The 'Bitter Spot': Where It Is Pure Waste
Just as important as knowing when to use a reasoning model is knowing when not to. Using a powerful reasoning model for simple tasks is like hiring a mathematics professor to calculate how much change you should get back at the bakery.
Avoid reasoning models for the following use cases:
- Text translation: Standard models are already virtually perfect at this and much cheaper.
- Summarization: Extracting the core from a text does not require deep, multi-step logic.
- Creative writing: Poems, marketing copy, or emails benefit from creativity, not rigid analytical reflections. Reasoning models tend to "over-engineer" these kinds of tasks, leading to dry, robotic texts.
- RAG (Retrieval-Augmented Generation) for simple FAQs: If the answer is already literally in your source document, no extra thinking time is needed.
Are you unsure which model fits your general task? Then consult our guide on choosing a model.
Managing the Thinking Budget in Practice
When developing AI applications, you don't always want the model to keep thinking indefinitely. Fortunately, APIs today offer ways to manage this "thinking budget."
Through the API of, for example, OpenAI, you can set the reasoning_effort parameter (often to low, medium, or high). A low effort forces the model to reach a conclusion faster, saving tokens and time. In addition, it is crucial to understand how limits work. The classic max_tokens parameter has been replaced or supplemented in many reasoning models by a limit that also includes the invisible tokens (such as max_completion_tokens). If you set this too strictly, you will interrupt the model's train of thought halfway through, and you won't get an answer back at all.
"Longer chains of thought are not always better. Sometimes a model gets stuck in a 'hallucination loop', where it endlessly keeps considering alternatives for a problem that is fundamentally very simple."
It is a misconception that more thinking time always leads to a better answer. If a prompt is ambiguous, a model can reason itself into a corner. Writing clear, unambiguous prompts therefore remains essential, even for models that can think for themselves.
Testing It Yourself: Determining the ROI of a Reasoning Model
Before migrating your entire production environment to an expensive reasoning model, it is crucial to validate the Return on Investment (ROI). Follow these three steps to determine if it is worth the extra cost for your specific task:
- Create an evaluation set: Collect 50 to 100 of your most challenging, real-world prompts. These should be problems that your current System 1 model (e.g., GPT-4o or Claude 3.5 Sonnet) regularly struggles with.
- A/B testing: Run these prompts through both your current model and a reasoning model. Measure not only the quality of the answer (is it correct?), but also record the API costs per call and the waiting time (TTFT). For a broad perspective, also check out our LLM comparison overview.
- Cost-benefit analysis: Compare the pure token costs with the saved development time or reduced error margin. If a reasoning model is 10 times more expensive, but it solves an architectural software bug that would have otherwise taken a developer 4 hours to fix, the ROI is highly positive. If it is simply 10 times more expensive to write a slightly better product summary, the business case is unsustainable.
Conclusion
Reasoning models represent a wonderful step forward in artificial intelligence. They shift the focus from "fast and fluent" to "thoughtful and accurate." Due to their iterative, internal thinking process, they are unparalleled in programming, mathematics, and strategic logic. However, due to the high hidden costs of reasoning tokens and the noticeable increase in latency, they require a conscious architectural choice. Deploy them as the scarce experts they are, and keep fast, regular models for your daily operational AI tasks.