The impact of prompt formatting on inference costs

When designing production architectures for large language models, plenty of attention goes to model selection, quantization and hardware allocation. Yet a considerable share of operational spending arises before the model performs even a single reasoning step: in the syntactic construction and serialization of the prompt. The way instructions, context documents, source data and schema definitions are formatted directly dictates the number of input tokens and the stability of the KV cache.

At scale, every space, delimiter, JSON key or Markdown table translates into measurable GPU cycles and API invoices. To understand the financial dynamics of this, you need insight into vendor base rates; see the overview of per-token pricing models for the ratios between input, output and cache rates. In this article we analyze how formatting choices affect inference costs, which data formats are most cost-efficient, and how a well-considered prompt structure saves hundreds to thousands of euros per month.

The physics of tokenization: how formatting turns characters into tokens

A language model does not process a raw text string but a sequence of integers (token IDs). This conversion happens through algorithms such as Byte-Pair Encoding (BPE) or WordPiece. Byte-Pair Encoding builds a vocabulary by merging frequently occurring character sequences. Text that aligns closely with natural language clusters is encoded compactly: one token often represents a whole word or a syllable of 3 to 4 characters.

Formatting characters, however, show fundamentally different tokenization behavior. Symbols such as braces, square brackets, colons, tabs, quotation marks and repeated whitespace often do not appear in the vocabulary as combined clusters. A seemingly short syntax construction such as {"key": "value"} requires considerably more tokens per useful character than the plain text version key: value.

The mechanism of leading whitespace also plays a crucial role. Many tokenizers (including cl100k_base and o200k_base) attach a space to the word that follows it. When formatting interrupts the natural order of words and spaces through odd indentation or superfluous line breaks, the tokenizer splits familiar words into several separate sub-tokens. This phenomenon causes creeping inflation of input volume without any increase in semantic information density.

Data serialization compared: JSON, YAML, XML and Markdown

When structured data or documents are injected into a prompt, the choice of serialization format is decisive for token consumption. Developers reach for JSON by default because it is the dominant format in web applications. From a token perspective, however, JSON is extremely inefficient because of its mandatory braces, quotation marks around every field name and string value, and commas.

Alternatives such as YAML, compact XML or Markdown tables offer varying ratios between syntactic overhead and semantic precision. To make the differences visible, we compare an identical dataset of ten entities with five fields each across four common formats:

These measurements show that replacing standard JSON with YAML or compact Markdown tables yields a direct token reduction of 25% to 40% for tabular payloads. For applications analyzing millions of records through prompts every day, that saving translates straight to the bottom line.

Spaces, indentation and line breaks: the invisible token overhead

A common source of waste is passing formatted payloads with deep indentation (pretty-printing). In automated pipelines, a json.dumps(data, indent=2) or indent=4 call converts data structures into readable strings for human inspection. For an LLM, however, that visual hierarchy is superfluous and harmful to the cost structure.

Tokenizers often process sequences of spaces in blocks of two, three or four spaces, depending on the specific vocabulary. Once a nested object sits four levels deep (8 spaces), every line consumes 2 to 3 tokens purely on indentation. In a document of 500 lines of code or configuration data, this costs 1,000 to 1,500 tokens on empty space alone.

# Slecht: 28 tokens door inspringing en quotes
{
  "gebruiker": {
    "id": 1042,
    "status": "actief"
  }
}

# Optimaal: 14 tokens (50% reductie)
gebruiker:id=1042,status=actief

Stripping superfluous whitespace, tabs and double line breaks (minification) is a deterministic step that carries zero risk of hallucination while compressing the input payload directly. If you want to work through the total cost of large volumes of context, turn to the calculation bridge for long context windows to see how tokens accumulate as you scale up.

Delimiters and separators: context isolation versus token overhead

To separate instructions sharply from external data and mitigate prompt injection, delimiters are used such as triple quotation marks ("""), Markdown headings (###) or XML tags (<context>...</context>). The choice of delimiter has both a security and a cost dimension.

Classic models from Anthropic and OpenAI are explicitly trained to recognize XML tags as structural anchors. XML tags have a specific advantage: opening and closing tags (such as <doc id="1"> and </doc>) are usually tokenized as compact units of 2 to 3 tokens. They offer watertight separation without the visual noise of long separator lines.

Using long decorative separator lines, such as ==================== or --------------------, is a common anti-pattern. Because BPE tokenizers often chop repeated identical symbols into unpredictable subsequences of 2 to 4 characters, a decorative line of 40 dashes quickly costs 10 to 15 useless tokens. Multiplied across dozens of context chunks in a Retrieval-Augmented Generation (RAG) system, this runs into thousands of wasted tokens per interaction.

Prompt caching and prefix stability: the financial lever

The absolute game changer in modern inference costs is prompt caching (server-side KV caching). Cloud providers such as Anthropic, OpenAI, DeepSeek and Google offer discounts of 50% to 90% on input tokens that exactly match earlier requests. This is where the interplay between formatting and cost becomes extremely sharp: a small formatting error can invalidate the entire cache.

Prefix caching works strictly deterministically from the very first token of the prompt. As soon as a variable value sits at position 100 (such as a dynamic timestamp, a session ID or an arbitrary formatting change), the cache lapses for all tokens that come after it (positions 101 to 50,000). To benefit optimally from caching, the prompt must be strictly structured into a static prefix block and a dynamic tail block.

For a deeper exploration of how these caching mechanisms work technically, the reference on context caching in LLM APIs offers a complete overview of time-to-live settings and minimum token thresholds. In addition, the simulator for prompt caching cost savings and latency lets you interactively model how cache hit rates lower the effective token price.

+-------------------------------------------------------------+
| STATISCHE PREFIX (Gecachet: 80-90% korting)                 |
| - Systeemprompt en rolinstructies                           |
| - Vaste schema-definities en output-eisen                   |
| - Vaste Few-shot voorbeelden                                |
| - Statische referentiedocumenten                            |
+-------------------------------------------------------------+
                              |
                              v (Exacte byte-identieke scheiding)
+-------------------------------------------------------------+
| DYNAMISCHE SUFFIX (Niet gecachet: 100% inputtarief)         |
| - Huidige datum/tijd                                        |
| - Gebruikersvraag                                           |
| - Real-time opgevraagde RAG-context                         |
+-------------------------------------------------------------+

Anyone who arranges their prompt structure so that all static instructions come first, without dynamic formatting noise, immediately realizes a structural cost saving of 70% to 85% on the total input bill.

Structuring few-shot examples without wasting tokens

Few-shot prompting (in-context learning) is one of the most effective ways to guarantee output quality without fine-tuning the model. At the same time, examples are often the largest fixed cost item in the system prompt. A poorly formatted set of five examples can easily take up 2,000 tokens.

To keep few-shot examples compact and cost-efficient, three optimization rules apply:

First: eliminate verbose conversation labels. Replace cumbersome constructions such as Gebruiker: Vraag hier... \nAssistent: Het antwoord luidt als volgt... with compact markers such as Q: ... \nA: ... or compact XML nodes <ex q="..." a="..."/>.

Second: apply micro-formatting to the output examples. If the desired output is a JSON object, the examples in the prompt do not need whitespace and indentation. The model learns the pattern just as effectively from a minified string.

Third: limit examples to edge cases. A common mistake is including long, representative texts that consist mostly of bulk data. Synthetically shortening the input in examples to the minimum length needed to demonstrate the reasoning pattern halves the token consumption of the instruction block.

Structured output and schema injection: the impact on input and output tokens

When an LLM has to produce reliable JSON, APIs (such as OpenAI Structured Outputs or Anthropic Tool Use) require the specification of a JSON Schema. How that schema is supplied and enforced has considerable financial consequences.

With traditional prompting, developers write the schema out manually in the system prompt, often including long per-field descriptions ("description": "..."). This costs hundreds of input tokens per call. With modern native JSON modes, the schema is converted into a context-free grammar (CFG) or constrained decoding state machine on the inference server.

For an overarching view of prompt construction and context management we refer to the guide on context engineering, which explains why formatting and context selection form an integrated design process. Important here is the cost difference between input and output tokens: at virtually every provider, output tokens are 3 to 5 times more expensive than regular input tokens. Formatting that inflates the output unnecessarily (such as long field names in generated JSON) therefore hits the invoice disproportionately hard.

// Inefficiënte output (veel dure outputtokens)
{
  "transactie_identificatienummer": "TX-99812",
  "status_van_de_huidige_verwerking": "SUCCESS",
  "totaal_bedrag_inclusief_btw": 149.95
}

// Geoptimaliseerde output (60% minder outputtokens)
{
  "tx_id": "TX-99812",
  "status": "OK",
  "bedrag": 149.95
}

Measurement methods and benchmarking of formatting overhead

Making formatting overhead visible requires an empirical measurement method. Counting characters by hand gives a distorted picture because of the properties of BPE tokenizers. An effective measurement method uses the metric Token-to-Information Ratio (TIR):

TIR computes the ratio between the number of purely informative tokens (the raw data and core instructions) and the total number of tokens consumed, including syntax, whitespace and delimiters:

TIR = (Tokens_Nuttige_Payload / Tokens_Totale_Prompt) * 100%

In poorly optimized enterprise pipelines, TIR regularly sits below 40%, which means more than 60% of the input budget goes to structural overhead. Systematic profiling with specialized tokenizers (such as tiktoken for OpenAI or tokenizers from Hugging Face) makes it possible to pinpoint per pipeline step where the overhead arises.

To automate the financial calculation of these savings across different LLM models, you can use the interactive AI model cost calculator to compare scenarios with compressed payloads directly.

A strategic optimization protocol for production systems

Optimizing prompt formatting requires a structured approach to keep compression from causing loss of function or reduced accuracy. The following protocol secures maximum cost efficiency while preserving quality:

Step 1: Payload minification. Automatically remove all redundant whitespace, indentation and separator lines from dynamically injected context. Transform JSON objects into compact YAML or Markdown tables wherever possible.

Step 2: Prefix stabilization. Reorder the prompt: place all immutable components (system prompt, fixed documents, schemas, examples) at the start of the payload. Make sure dynamic data is only appended at the end in order to maximize cache hits.

Step 3: Key and schema compression. Shorten key names in both input schemas and expected output structures. Remove redundant field descriptions once the key name itself already carries enough semantic meaning for the model.

Step 4: A/B validation and regression testing. Run an evaluation set after every compression pass. Measure not only the token savings and latency gains achieved, but explicitly validate that the model's extraction and reasoning accuracy holds up.

By treating prompt formatting not as a trivial aesthetic detail but as a core component of the software architecture, organizations can cut their inference costs drastically without giving up reliability or model capability.