Choosing models for medical text analysis
Selecting a suitable language model for processing medical records, discharge letters, and clinical reports requires a fundamentally different approach than choosing a model for generic text processing. Medical texts have unique linguistic characteristics and extreme precision requirements. A misinterpretation in this domain can have direct consequences for safety and quality of care.
In this guide, we cover the specific complexity of Dutch-language medical records, the various processing tasks in healthcare practice, the crucial importance of negation detection, the trade-off between generic and domain-specific models, and setting up reliable evaluation on your own material.
What makes medical text so complex
Clinical documentation rarely follows the rules of standard prose. Anyone deploying a language model on medical records runs into a combination of linguistic and domain-specific obstacles that quickly derail a generally trained model.
1. Dense abbreviations and acronyms
Healthcare providers write quickly and use abbreviations extensively. The meaning often depends heavily on the context or the specific department. The abbreviation 'AP', for example, can stand for 'angina pectoris', 'alkaline phosphatase', or 'anterior-posterior'. A model must infer from the surrounding text which meaning is intended.
2. Negation and nuances of uncertainty
A significant part of a clinical record describes what the patient not has or does not have, or what is still uncertain. Sentences such as "no evidence of pulmonary embolism" or "differential diagnosis includes X, but Y cannot be ruled out" contain crucial negations and degrees of probability. General models sometimes tend to pick up on the term 'pulmonary embolism' while ignoring the negating context, leading to a faulty interpretation.
3. Telegram style and fragmentary notes
Especially in nursing handovers and emergency department reports, sentence structure is extremely compressed. Verbs and articles are omitted. Fragments such as "Patiënt 80j, pz cat, spv bl. dnk: urineweginfectie" require the model to independently interpret missing grammatical structures without distorting the factual content.
4. Multilingual mixed forms
Dutch-language medical documentation is rarely pure Dutch. Latin anatomical terms (such as arteria carotis) are interspersed with English medical terminology (such as heart failure or pacing), Dutch technical terms, and local hospital jargon. A model must be able to flawlessly parse this hybrid structure.
Types of medical text tasks and their model requirements
Not every task places the same demands on a language model. In practice, it is essential to distinguish between four main categories of text processing.
| Task type | Characteristics | Primary model requirements |
|---|---|---|
| Coding and classification | Assigning standardized codes (such as ICD-10, DBC, or SNOMED CT) to free text. | High precision, strict adherence to taxonomies, low tolerance for hallucination. |
| Summarizing records | Reducing a long patient history to a concise chronological summary. | Large context window, excellent negation detection, preservation of chronology. |
| Information extraction into fields | Structured retrieval of specific parameters (such as medication, dosage, lab values). | Strict JSON/XML structured output, high recall on entities. |
| Question answering on guidelines | Answering questions from healthcare providers based on protocols and NHG standards. | Strong RAG integration, accurate source attribution, factual consistency. |
As with models for document processing in the administrative sector, here too the task definition determines which architecture is most suitable. A model that excels at creative summaries is not automatically suited to structured extraction of medication lists.
Negation and uncertainty detection as a quality lever
In medical text analysis, negation detection is the most important quality lever. When a model extracts a symptom or diagnosis from a text, it must be able to determine with one hundred percent certainty whether that symptom is present, absent, or merely suspected.
The challenge lies in the variety of negating structures in Dutch. Consider the following examples:
- "Patiënt ontkent pijnen in de borst." (Direct negation)
- "Geen sprake van koorts of koude rillingen." (Scope of negation across multiple entities)
- "Er is geen reden om te twijfelen aan de diagnose." (Double negation)
- "Afwezigheid van afwijkingen op de X-foto." (Nominal negation)
Models that have not been specifically evaluated on negation detection often display a dangerous trait: they respond with extreme confidence while overlooking the negation. A model may state in its output that the patient has chest pain purely because that word combination appears in the source text. In a medical context, a false confirmation of an absent condition is just as harmful as missing a condition that is actually present.
General models versus domain-specific models
When selecting a model, organizations face a choice between generic large language models and models that have been further trained or fine-tuned specifically for the medical domain.
The strength of generic models with good prompting
Modern general models have an enormous base of knowledge and an excellent understanding of Dutch sentence structure. With the help of advanced prompting (such as few-shot examples that explicitly demonstrate negations and medical abbreviations), these models often achieve surprisingly good results. For tasks such as rewriting a medical text into a letter the patient can understand, a generic model with the right instructions is often more than sufficient.
When domain adaptation becomes necessary
Further training on domain-specific data (fine-tuning or training your own model) only becomes necessary when:
- The text is full of internal jargon and abbreviations specific to a single institution or region.
- The task requires strict output in a very specific medical format (such as SNOMED CT coding) where generic models too often deviate from the format.
- Extreme demands are placed on latency and cost, requiring a smaller, specifically fine-tuned model to be run locally.
When considering open-source models that need to be fine-tuned, it is essential to understand how the licenses and documentation are structured. For this, consult the guide on reading model cards and licenses to avoid legal and operational surprises with local use.
The role of context: RAG over model size
A common misconception is that a larger model automatically gives better medical answers. In practice, adding the right context via Retrieval-Augmented Generation (RAG) almost always delivers more benefit than switching to a larger model with more parameters.
Medical knowledge and guidelines change constantly. Letting a model answer questions about treatment protocols based on its internal weights (parametric memory) inevitably leads to outdated or inaccurate advice. By having the model work exclusively with the current guidelines (such as NHG standards or local hospital protocols) supplied via the context window through a RAG system, the risk of hallucinations is drastically reduced.
Key principle: Use the language model primarily as a reasoning and processing engine, not as an encyclopedia. Let the factual medical knowledge come from a managed, external source.
Deployment question and constraints: infrastructure and privacy
In medical text analysis, model choice is inseparably linked to the infrastructure in which the model runs. Privacy legislation (GDPR) and the guidelines around processing medical personal data set hard limits on what is technically and legally permitted.
1. Processing location
Sending non-pseudonymized patient records to external commercial APIs outside the European Economic Area is not permitted in virtually all cases. Organizations must choose between:
- A locally hosted model within the hospital's or healthcare institution's own infrastructure (on-premises).
- A dedicated, certified sovereign cloud environment within the EU that meets specific standards for the healthcare sector (such as NEN 7510 and ISO 27001).
2. Pseudonymization and data minimization
Before a medical text is processed by a language model, pseudonymization must often take place. Social security numbers (BSN), patient names, addresses, and dates of birth must be filtered out or replaced with fictitious identifiers. This process in turn places high demands on information extraction tools.
3. The human in the loop (Human-in-the-loop)
Who has final control over the output determines the risk profile. Make sure the process is set up so that a qualified healthcare provider can always verify the extracted or summarized information before it is stored in the Electronic Patient Record (EPD).
For a comprehensive legal and organizational risk inventory around data processing, it is advisable to carry out a AI risk analysis and DPIA before putting a model into production.
Supportive use versus decision-making
The legislative framework (including the European AI Act and regulations around medical devices, such as the MDR/IVDR) draws a sharp distinction between the intended purposes of a system.
Supportive and administrative use
When a model is deployed for administrative support — such as drafting a discharge letter that is fully reviewed by a physician, or sorting unstructured notes into the right fields of a form — the application generally falls into a lower risk profile. Here, the model functions as an advanced typing aid or assistant.
Diagnostics and decision-making
As soon as a model autonomously provides advice on treatments, calculates risk scores that directly influence clinical decisions, or suggests diagnoses based on a textual patient history, the system shifts into the domain of medical devices. Strict requirements apply here regarding certification, clinical evaluation, and continuous monitoring. The accuracy and transparency requirements for the chosen model are many times higher in this situation.
For a broader overview of current developments and the societal impact of artificial intelligence in healthcare, you can consult the article on AI in healthcare on the news platform.
Evaluating with your own material
Many organizations are tempted by public medical benchmarks and leaderboards on platforms such as Hugging Face. However, these figures say surprisingly little about performance on Dutch-language clinical texts in local practice.
Why public benchmarks fall short
Public medical benchmarks (such as MedQA or PubMedQA) consist mainly of English- and Chinese-language medical exam questions or scientific articles. This material differs fundamentally from the daily, often messy documentation in Dutch hospitals and general practices. A high score on an English medical exam offers no guarantee whatsoever of correctly processing a Dutch nursing handover.
Building your own test set
To make a responsible model choice, building an internal test set is necessary. Follow these steps:
- Select a representative sample: Collect approximately 100 to 200 anonymized or pseudonymized documents that represent actual practice (including typos, abbreviations, and telegram style).
- Establish a gold standard: Have medical specialists or experienced medical assistants manually create the desired output (for example, the correct summary, the extracted parameters, or the correct codes).
- Define specific evaluation metrics: Do not just assess general text quality, but explicitly measure the number of missed negations (false positives), the number of fabricated facts (hallucinations), and the accuracy of extracted entities.
Practical guidelines for setting up such evaluations in the language can be found in the overview on testing Dutch-language models.
Step-by-step plan for model selection
To arrive at a well-considered choice for medical text analysis, you can use the following step-by-step plan:
- Determine the task and the risk profile: Is the process purely supportive, or does it directly influence medical actions? Does the task require structured output or free text?
- Establish the constraints: Which legislation (NEN 7510, GDPR, AI Act) applies, and which processing location is required? Does this rule out external APIs?
- Set up a RAG infrastructure: Make sure medical knowledge, guidelines, and protocols are supplied via dynamic context rather than relying on the model's weights.
- Build your own test set: Assemble an internal set of curated Dutch cases, including complex negations and technical jargon.
- Compare models on your own test set: Test both general models via a secure environment and any local, open-source models on the assembled test set.
- Implement monitoring and human oversight: Set up the workflow so that a professional can always check and adjust the output.
We see similar selection processes in other sectors with high demands on confidentiality and precision, as described in the analysis on models for legal texts.
