Choosing models for customer service and chatbots
When developing automated customer systems, the emphasis often falls on choosing the most advanced large language models. In practice, however, the specific model choice for customer contact and chatbots rarely turns out to be the primary factor in success. After all, the behavior expected from a customer service bot is tightly bounded and strictly defined. Creativity, broad general knowledge, and complex reasoning chains are, within this domain, secondary to predictability, strict instruction adherence, and staying within set boundaries.
Still, the choice of the right type of language model is not irrelevant. A model's technical properties must closely match the specific behavioral profile that customer contact requires. A poorly chosen model can cause the chatbot to overstep its bounds, make incorrect assumptions, or slip into an unnatural tone. In this article, we analyze which model characteristics matter more than general intelligence in a customer service context, how latency affects the user experience, which specific pitfalls exist for the Dutch language, and why building the knowledge base and escalation path must always precede model selection. For a general overview of model categories, you can consult how to select a model per task.
Why model choice in customer service requires a specific profile
Customer service is an environment where mistakes have direct operational and reputational consequences. Where a model for brainstorming or text summarization benefits from a certain degree of generative freedom, a customer service application calls for the opposite: determinism and boundaries. Customer questions almost always relate to concrete procedures, product specifications, billing, or logistics.
In this context, the scope of permitted answers is narrow. A model must not try to fill in a missing detail with plausible-sounding information. The model's primary function is to interpret the customer's question, process the supplied documentation, and formulate a correct, polite, and concise answer within the imposed constraints. General benchmark scores in math, programming, or philosophical reasoning therefore give a misleading picture of a model's suitability for customer contact.
Key insight: In customer service, what matters is not how much the model knows from its training data, but how strictly the model can limit itself to the supplied context and rules.
Properties that matter more than general intelligence
When evaluating models for customer service applications, the priority shifts from raw processing capacity to specific behavioral parameters. The following four characteristics form the basis of a reliable chatbot.
1. Instruction Following
Instruction following refers to the degree to which a model exactly follows system instructions and imposed rules. A customer service prompt often contains rules such as: "Answer the question exclusively based on the source text below. If the source text does not contain the answer, say that you do not have the information and offer to connect the customer with a staff member." A model with high instruction adherence will not try to generate an answer from its general knowledge anyway, but will flawlessly carry out the instruction to refuse or hand off.
2. Refusing without being impolite
A customer service bot regularly receives questions that fall outside its scope, or questions that violate the organization's policy. The model must be able to enforce clear boundaries without the tone becoming unfriendly, curt, or mechanical. There is a subtle difference between a model's general safety refusal ("As an AI assistant, I can't help with that") and a professional customer service refusal ("I can't find that information in the system. I'd be happy to connect you with one of our staff members"). Models must be sufficiently fine-tuned to maintain this specific level of phrasing.
3. Consistency in register and form of address
A common problem in dialogue systems is inconsistent register. A model may address the customer formally in the first message and then unnoticeably switch to informal address in a follow-up message. This happens particularly when the conversation context grows longer and the model becomes influenced by the customer's language style. A suitable model maintains the form of address set in the system prompt throughout the entire session, regardless of how the customer communicates. You can read how to formulate these behavioral guardrails in the article on designing a chatbot persona.
4. Not improvising when context is missing
Hallucinations are unacceptable in a customer service environment. When a customer asks about the return period for a specific item and that period is not in the supplied knowledge base, the model must not make an estimate based on general e-commerce conventions. Models specifically calibrated to recognize uncertainty and directly report 'unknown' perform considerably better on this point than models trained to always formulate a substantive answer.
Latency as a product characteristic
When using language models for background tasks, such as document synthesis or data transformation, a response time of a few seconds more or less is rarely a problem. In a live chat interface, however, wait time is a direct factor in the user experience. High latency leads to user confusion, repeated input, or premature abandonment of the session.
Response time in dialogue systems consists of two components:
- Time-to-First-Token (TTFT): The time the model needs to process the request and send back the first word of the answer.
- Inter-token latency (generation speed): The speed at which subsequent words are generated.
Large models with tens or hundreds of billions of parameters generally need more computing power and time to generate tokens. Smaller models (for example, with 7 to 14 billion parameters) can, with the right hosting environment, produce answers with a fraction of the delay. Because the answer in a customer service context usually needs to be short and factual, a smaller model well supported by relevant context text often delivers a much more solid user experience than a very large but slower model.
| Model class | Processing speed (TTFT) | Instruction adherence | Suitability for customer contact |
|---|---|---|---|
| Large general model | Moderate to slow | Very high | Suitable for complex, multi-step tasks |
| Mid-sized optimized model | Fast | High | Optimal balance for standard questions |
| Small streamlined model | Very fast | Moderate to high (with RAG) | Excellent for direct routing and simple FAQs |
Dutch-language customer contact and language-specific pitfalls
Processing the Dutch language places specific demands on a model. Many language models are primarily trained on English-language data. Although these models can generally generate correct Dutch, the generated Dutch often shows subtle flaws that detract from the professionalism of a customer service interaction.
Unwanted anglicisms and sentence structures
Models with a strong English-language base sometimes translate thoughts from English directly into Dutch. This results in unnatural word usage, such as overusing passive constructions, unusual preposition constructions, or an excess of politeness forms that are common in English but come across as stiff or forced in Dutch (for example: "I hope this message finds you in good health").
Maintaining the correct register (formal vs. informal)
In Dutch, the distinction between formal ('u') and informal ('je/jij') address is essential. A company with an informal tone does not want the model to suddenly switch to formal address when a customer asks a formal question. Conversely, a business service provider needs the guarantee that the model consistently keeps the formal address. Models differ in their sensitivity to the customer's language style. A model that ignores the instruction given in the prompt and starts mirroring the customer's tone undermines brand identity. More background on how language models score on Dutch-language texts can be found in the comparison of Dutch-language content quality.
Retrieval over model size: the value of controlled context
A common misconception is that a larger model with broader general knowledge is better able to answer customer questions. In practical setups, adding a well-structured knowledge base via Retrieval-Augmented Generation (RAG) almost always yields more quality improvement than upgrading to a larger model class.
When a model answers based on the information in its parameters, there is a risk of outdated or incorrect facts. By having the model function solely as a reader and summarizer of explicitly supplied context documents, the processing becomes verifiable. We describe how the basic setup of such a context architecture works in the guide on RAG for beginners.
The value of 'I don't know'
In customer contact, a correctly formulated admission of not knowing is more valuable than a guessed answer. If a customer asks whether a specific product is in stock and the context does not contain that data, the model must directly state that it does not have that information. A model that learns to refuse when in doubt prevents customers from receiving incorrect information about deliveries, prices, or terms. A model's ability to recognize its own lack of context is an important selection criterion.
Escalation as a fundamental design requirement
No dialogue system can handle all customer questions independently. Escalation to a human staff member is not an exceptional situation but an essential part of the system design. The choice of model must also be based on how reliably the model can signal when escalation is necessary.
Escalation triggers include:
- Complex or multi-part questions: The question requires actions in external systems that the bot cannot perform.
- Emotional intensity: The customer expresses frustration or explicitly indicates a wish to speak with a human.
- Missing information: The knowledge base does not contain enough data to answer the question.
- Repeated confusion: The dialogue stalls because the bot still does not understand the customer's question after two turns.
Recognizing these situations is both a prompt and a model property. The model must be able to assess the context of the entire conversation and, at the right moments, send out a structured signal (such as a specific JSON output or tool call) that initiates the handoff to a staff member. Organizational aspects of this transition are covered extensively in the article on introducing chatbots into customer service organizations. For the technical integration options, you can also look at available AI tools for customer service.
Cost structures in a dialogue context
When estimating the operational costs of a chatbot environment, developers often look at the rates per token. In a dialogue application, however, the cost calculation plays out differently than for one-off text processing. This is due to the buildup of conversation history (history accumulation).
With every new turn in a chat conversation, the application resends the entire preceding dialogue, including the system prompt and supplied context, to the model. This means that turn 5 of a conversation consumes considerably more input tokens than turn 1.
Beurt 1: System Prompt + Context + Vraag 1 = 1.200 tokens
Beurt 2: System Prompt + Context + Historie + Vraag 2 = 1.500 tokens
Beurt 3: System Prompt + Context + Historie + Vraag 3 = 1.850 tokens
Totalen lopen cumulatief op bij elke dialoogstap.Because customer service conversations often consist of multiple short turns, costs are primarily driven by the token volume of the repeated context and history. A model that handles long prompts efficiently or supports context caching can, in practice, turn out cheaper than a model with a lower base rate but no caching functionality.
What you actually measure to select models
To determine which model performs best in a specific customer service environment, factual evaluation metrics must be set up. General benchmarks say little about performance on your specific domain data. The following three variables give a representative picture:
1. Grounding percentage (source attribution)
This measures what share of the generated answers can be directly traced back to the supplied source documents. Any answer that contains facts not present in the retrieved context counts as a negative score, regardless of whether the information happened to be correct.
2. Hallucination ratio with missing context
Test the model by deliberately posing questions whose answer is not in the supplied context. A good model achieves a score where it explicitly indicates in virtually all cases that the information is missing. A model that still makes assumptions is unsuitable for production.
3. Escalation precision and recall
Measure how accurately the model recognizes when a conversation needs to be handed off. Precision indicates whether escalations occur appropriately (no unnecessary handoffs), while recall indicates whether all situations that should have been escalated were actually caught.
The practical order of implementation
A common mistake is choosing a language model at the start of the development process. Model selection should only take place in the final phase of the process. The correct order of approach is as follows:
- Clean and structure the knowledge base: Make sure the documentation is up to date, concise, and free of contradictions. No model can answer correctly based on polluted sources.
- Determine escalation logic and constraints: Define hard rules for when a conversation must be handed off and which tone of voice should be used.
- Set up the retrieval mechanism: Build the search and selection logic that finds the right pieces of information for the question.
- Test and compare models: run a test set of representative customer questions across different models and evaluate on instruction adherence, latency, register consistency, and cost per conversation.
Following this order prevents you from trying to solve complexity with a heavier model, when the real problem actually lies in data quality or process design.
