Skip to content
NLEN
Illustration: Models for summarizing long documents

Models for summarizing long documents

By Ivo Donker — compiled with AI assistance (Claude & Gemini) · Last updated: August 7, 2026

The automated summarization of long documents — such as annual reports, legal contracts, technical specifications, and scientific books — is one of the most requested applications of large language models (LLMs). Although modern models have increasingly larger context windows, effectively and reliably summarizing large source texts is far from trivial. In practice, developers and organizations run into challenges such as information loss, hallucinations, high latency, and exponentially rising costs.

In this article, we cover the technical and architectural conditions for summarizing long documents. We analyze why long texts pose a challenge for transformer-based models, compare processing strategies (processing long-context directly versus splitting via chunking), clarify the distinction with Retrieval-Augmented Generation (RAG), and provide a concrete framework for prompting, evaluation, and cost management.

1. Why long documents pose a fundamental challenge

At first glance, summarizing a document of a few hundred pages seems like a solved problem: you copy the text into the prompt of a model with a large context window and ask for a summary. However, the underlying mechanics of transformers bring specific limitations that directly affect the quality of the result.

Context window and theoretical capacity

The context window of an LLM determines the maximum number of tokens (words and characters) the model can receive in a single processing pass (inference pass). Where early models were limited to 2,048 or 4,096 tokens, modern enterprise models in 2026 support context windows ranging from 128,000 tokens to 1 to 2 million tokens. However, the fact that a model can theoretically *ingest* a document does not guarantee that it processes all the information in that text with equal precision.

Attention dilution and the 'Lost in the Middle' phenomenon

The self-attention mechanism of the transformer architecture calculates the relationship between every token and every other token in the input. With very long texts, the attention matrix becomes saturated. This leads to what is referred to in the literature as attention dilution (attention dilution). The model has trouble isolating subtle, crucial details when they are surrounded by tens of thousands of tokens of peripheral information.

This manifests concretely in the well-known 'Lost in the Middle' effect. Scientific research and empirical benchmarks such as the Needle in a Haystack test method show that models remember and process information at the beginning (primacy effect) and end (recency effect) of a long prompt significantly better than information located in the middle of the document. When an essential caveat or a critical financial clause is halfway through page 140, there is a real chance the summary will ignore that detail.

Cost and latency structure

Processing long documents has direct consequences for operational engineering:

2. Architecture choices: Long-context vs. Chunking vs. RAG

When designing a pipeline for summarizing long documents, a choice must be made between three primary processing strategies. The optimal choice depends on the purpose of the summary and the structure of the source.

Strategy A: Direct long-context processing

Here, the entire document is presented to the model as context in one go. The prompt contains the full document, followed by the specific instruction for the summary.

Strategy B: Splitting via chunking (Map-Reduce and Refine)

When a document exceeds the effective window, or when you require maximum precision per section, the document is split into smaller segments (chunks). There are two well-known patterns:

  1. Map-reduce:
    • Map step: Each segment of the document is fed to the model in parallel with the request to produce a partial summary.
    • Reduce step: The collected partial summaries are consolidated and merged into one final summary in a second step.
  2. Refine (iterative refinement):
    • The model processes the first segment and generates an initial summary.
    • The model then receives the second segment along with the initial summary, with the instruction to update the summary based on the new information. This process repeats for all segments.

Note on Map-Reduce: Information loss often occurs when merging partial summaries (the Reduce step). Nuances isolated in the Map step can still be filtered out during the final aggregation if the prompt does not strictly specify which details must be retained.

Strategy C: Retrieval-Augmented Generation (RAG)

A common misconception is using RAG to create a comprehensive summary. RAG is designed to find specific answers based on targeted queries (e.g., "What does the contract say about liability?").

For a general summary ("Give an overview of the main points in this document"), traditional RAG falls short. Vector searches (semantic search) only retrieve the top-K most relevant text blocks. As a result, the system inherently misses the parts of the document that don't score highly on the query, leading to an incomplete and skewed picture. RAG is extremely useful for question answering, but it is not suitable as the primary architecture for an overarching document summary, unless combined with hierarchical indexing or special map-reduce wrappers.

Property Direct Long-Context Chunking (Map-Reduce) RAG (Vector Search)
Global Context Excellent Moderate to good Poor (fragmented)
Detail retention Susceptible to mid-document dropout Very high per segment High, but only for relevant chunks
Suitable for Full-document analysis, coherent narrative Very large files, books Targeted question answering
Pipeline complexity Low (single API call) Medium/High (orchestration required) High (embedding, vector DB, retrieval)

3. Crucial model characteristics for document compression

Not every model is equally suited to summarization tasks. When selecting a model, organizations must look beyond the theoretical specifications on the label.

Effective vs. Stated Context Window

A model card may list a context window of 200,000 tokens, but the effective context window — the range within which the model can retrieve and process information with 99%+ accuracy — is often lower in practice. When evaluating a model for your applications it is necessary to test how retrieval and reasoning quality degrades as prompt length increases.

Instruction Following

When summarizing documents, it is crucial that a model adheres to strict constraints. Think of specific output formats (e.g., JSON with fixed fields), length limits, or the explicit instruction to no not use outside knowledge. Models with a high degree of instruction following are less easily distracted by the size of the input.

Hallucination and information compression

When a model has to compress a large amount of text into a short summary, the risk of hallucinations increases. This often takes the form of over-generalization or confabulation of connections: the model combines two facts from different chapters that in reality have nothing to do with each other. For an accurate assessment of this risk, we refer to the analysis on measuring hallucinations in LLMs.

4. Practical approach and prompt architecture

The way the prompt is structured has a direct impact on preventing information loss. The guidelines below improve the quality of the generated summary for long input.

System instructions and role definition

Give the model a clear role and define its boundaries. Explicitly instruct the model that it may only use information from the supplied source and that assumptions must be refused.

Position of the text and instruction

Because of the previously mentioned primacy and recency effects, the order within the prompt matters a great deal. Preferably place the processing instructions and the desired output format at the bottom of the prompt, after the document, or repeat the core instruction at the end.

[GESTRUCTUREERDE SYSTEEMINSTRUCTIE]
Je bent een senior data-analist. Je taak is het samenvatten van het onderstaande document.
Regels:
1. Gebruik UITSLUITEND feiten die expliciet in de tekst worden genoemd.
2. Structureer de samenvatting in drie secties: Hoofdpunten, Financiële Impact, en Risico's.
3. Als een sectie geen informatie bevat in de tekst, vermeld dan "Niet gespecificeerd".

--- BEGIN BRONTEKST ---
{{ HIER DE VOLLEDIGE TEKST VAN HET DOCUMENT }}
--- EINDE BRONTEKST ---

[HERHALING INSTRUCTIE & FORMAT]
Geef nu de samenvatting volgens de bovenstaande structuur op basis van de brontekst.

Enforcing structured output

Free-text summaries ("Give a summary of this article") lead to inconsistent results. Use JSON schemas or structured Markdown headers to force the model to explicitly isolate specific elements (such as dates, amounts, action items). This also makes the result easier to process automatically in downstream systems.

5. Quality evaluation of summaries

Evaluating the quality of a generated summary is complex, because there is rarely one 'perfect' summary. Still, there are objective criteria and methodologies for assessing summaries.

Quality criteria

Automated Metrics vs. LLM-as-a-Judge

Traditional n-gram metrics such as ROUGE and BLEU, which were widely used in classic NLP, fall short when assessing modern abstractive summaries. A model can write an excellent summary using completely different wording than a reference summary, resulting in a low ROUGE score.

In modern evaluation pipelines, the LLM-as-a-Judge principle is therefore often chosen. A secondary, advanced model is presented with the source text (or a representative sample of it) and the generated summary, and evaluates the summary against a detailed rubric on points such as factual accuracy and coverage.

6. Cost trade-offs and operational engineering

When setting up production processes for large volumes of documents, operating costs and processing times are decisive. Various strategies help optimize the Total Cost of Ownership (TCO).

Using lightweight models

Not every summary requires the largest and most expensive flagship model. A hybrid approach often delivers the best price-quality ratio: a smaller, faster model handles the initial Map step per segment, after which a more powerful model performs the final Reduce step to safeguard overall coherence. For specific files, deploying small models on your own hardware or dedicated servers can offer a cost-efficient alternative.

Pruning and pre-processing of input text

Preprocessing steps in the pipeline can significantly reduce the token count before the text is sent to the LLM:

By critically examining the architecture, model choice, and prompt design, organizations can build robust summarization systems that are both substantively reliable and financially scalable.