Response time calculator for AI models
The total response time of an AI model largely determines how smooth an application feels to an end user. In practice, this wait time consists of two separate phases: the time until the first output appears and the speed at which the rest of the text is generated. With this interactive calculator, you can enter two different scenarios and compare them directly, for example a larger reasoning model with a low output speed versus a smaller, optimized model.
Calculated results
Scenario A
Scenario B
Note: All entered values are your own assumptions and theoretical parameters. The results shown are formula-based estimates and do not constitute a guaranteed measurement of a specific API provider.
How this calculator works and what the results mean
The tool above calculates the duration of interactions with Large Language Models (LLMs) based on a simple mathematical model. A request to a language model consists of two main steps: processing the input and generating the first output, followed by sequentially generating the remaining characters.
In the calculator, you enter the expected performance for two scenarios. The calculator breaks down the outcome into two perspectives: the classic wait time where a user waits until the entire response is ready, and the perceived wait time when a streaming connection is used. The results show at a glance how small changes in output speed or network delay accumulate across multiple requests within a full session.
Why TTFT and output speed are two separate variables
When assessing API performance, Time to First Token (TTFT) and output speed (often expressed in tokens or characters per second) are regularly lumped together under the general term 'speed'. However, these are two fundamentally different phases in processing by a language model, determined by entirely different hardware and software factors.
Time to first token (TTFT)
TTFT measures the time between the moment your application sends the request and the moment the very first letter appears on the screen. This phase consists of the network time to the server, processing and tokenizing the input prompt, the prefill phase on the graphics card (GPU), and generating the very first token.
When a prompt is very long (for example a document of dozens of pages), the TTFT rises significantly because the GPU must process the full context at once. During this phase, the user has not yet received any feedback and is looking at an empty screen or a loading indicator.
Generation speed in characters per second
Once the first token is generated, the model moves into the decoding phase. Here, the model generates token by token in an autoregressive loop: each newly generated character is added back to the context to determine the next character. The speed of this phase is mainly bounded by the memory bandwidth of the graphics cards on which the model runs.
A model can have a very slow TTFT (for example due to a busy queue or a huge input prompt), but then produce characters at a high rate afterward. Conversely, a small model can start talking immediately (low TTFT), but have a slow output speed due to limited infrastructure. Because both phases have different bottlenecks, they must always be analyzed separately.
The influence of streaming on perceived wait time
When an application waits for the full API response before showing anything on screen, the total wait time equals the sum of the TTFT and the total generation duration. For a response of 1,500 characters and a speed of 100 characters per second, this means the user quickly has to wait 15 to 16 seconds before a single letter is visible.
By using streaming responses via Server-Sent Events (SSE) the content is sent directly to the browser in small chunks as soon as it becomes available. This does not change the total physical duration of the API call, but the perception of the user changes dramatically. The user can start reading as soon as the first few sentences (for example 10% of the total) appear on screen. Because a person reads an average of about 20 to 30 characters per second, the user can already process the text while the model fills in the rest of the paragraphs in the background.
Output length: the most important lever for response time
The calculations in the tool show that the expected output length has the biggest impact on the total duration. In many development projects, a lot of time is spent selecting a faster API provider to lower the TTFT by 100 milliseconds, while the prompt instructions encourage unnecessarily long responses.
By explicitly asking in the system prompt for concise answers, bullet points, or structured JSON structures without unnecessary introductions, the output length can often be reduced by 50%. This directly halves the generation time per request. Anyone who wants to structurally lower the total response time achieves the most gains by strictly limiting the desired output length. For scenarios where the computing power runs on the user's own hardware, specific additional conditions also come into play; read more about this in the overview on small models on the device.
max_tokens or specify a maximum length in your prompt. This prevents a model from rambling and not only reduces response time but also lowers the cost per call. You can also calculate this with the model cost calculator.
Invisible reasoning steps and their impact on response time
A special category is formed by so-called reasoning models. These models internally generate a chain of thought before showing their final answer to the user. These internal reasoning steps are hidden from the interface in many API implementations.
For the end user, this phenomenon manifests itself as an extremely high TTFT. In the background, the model is already generating hundreds or thousands of 'invisible' tokens to work through the logical steps. Even if the final visible answer is only 200 characters long, the wait time for the first character can rise to 5 or 10 seconds. When comparing models, it is crucial to factor in this distinction. See also the in-depth analysis in which reasoning models are compared in terms of accuracy versus wait time.
Averages versus percentiles in speed measurements
The numbers you enter in this calculator are fixed values. In practice, however, the response time of an AI service is variable. Reporting response times based on an average often gives a misleading picture, because occasional outliers heavily and negatively affect the user experience.
In professional environments, measurements are therefore expressed in percentiles:
- p50 (Median): The response time under which 50% of requests stay. This represents the typical experience during a quiet moment.
- p90: The response time under which 90% of requests stay. This gives insight into delays during busier periods.
- p99: The response time for the slowest 1% of requests. This shows the extreme peaks caused by network hiccups, complex prompts, or overloaded servers.
When you enter a scenario in this calculator, it is wise not only to fill in your median, but also to calculate a scenario based on your provider's p99 values. More information on setting up reliable benchmarks can be found in the article on measuring speed for AI models.
Where this calculator stops: external factors
This calculator offers a theoretical framework to understand the influence of parameters. A calculation model in the browser can, however, never simulate the full reality of a production environment. The following external factors fall outside the scope of this calculator:
- Network distance and TLS handshakes: The physical distance between your server (or the user's browser) and the model provider's data center adds latency.
- Queues at the provider: During an unexpected spike in global usage, an API request can remain in a queue for seconds before the GPU can start on the prompt.
- Cold starts: If a model runs on your own server or a specialized platform and hasn't been called for a while, the model must first be loaded from storage into GPU memory.
- Network timeouts: With extremely slow responses, the connection can be prematurely terminated by intermediate proxies or API gateways. See the documentation on timeouts and cancellation for robustly handling these situations.
To obtain guaranteed figures for a specific application, running your own measurements and load tests on the live infrastructure is the only reliable method.
Further reading
- Model cost calculator - Calculates the cost per token and request
- Small on-device models - Local execution and wait times
- Reasoning models compared - Latency versus reasoning power
- Measuring speed - Guide to correct benchmarking of LLMs
- Streaming responses - Implementing SSE in API integrations
- Timeouts and cancellation - Dealing with slow API responses
