Moderation and Safety Models: A Complete Guide for AI Applications
When building applications around Large Language Models (LLMs), the focus is often primarily on generating high-quality text, creativity, and intelligence. However, intercepting unwanted input and output is just as important. Without robust safety mechanisms, an application risks generating harmful, biased, or highly inappropriate content. This can lead to reputational damage, legal issues, and a downright poor user experience.
Moderation and safety models act as the digital bouncers of your AI application. They analyze what a user asks and what the main model answers, and intervene when this falls outside the established boundaries. In this article, we discuss what types of models and services exist, how the underlying techniques differ, and how to choose the right architecture for your specific application.
Why Are Moderation Models Indispensable in the World of LLMs?
Generative AI models are trained on massive amounts of text from the internet. In addition to valuable information, these datasets also contain toxic language, hate speech, dangerous instructions, and biases. Although developers apply techniques such as Reinforcement Learning from Human Feedback (RLHF) to make models safer (so-called alignment), no base model is infallible.
Users – whether with malicious intent or not – can attempt to bypass these built-in safety mechanisms using complex prompts. This is often referred to as 'jailbreaking' or 'prompt injection'. To defend against this, independent safety models are needed to act as an additional layer of protection. You can weigh which model to choose best fits your safety and performance requirements.
The Difference Between Input and Output Moderation (Input & Output Guardrails)
A robust safety system, often referred to by the term guardrails, typically works in two directions. It secures both the incoming traffic to the model and the outgoing traffic to the user.
Input Moderation (Prompt filtering)
With input moderation, the user's prompt is checked before it is sent to the main model. The goal is to immediately block prompts that ask for illegal activities, self-harm, hate speech, or the generation of malware. If a prompt is classified as unsafe, the call to the main model (often a heavy, computationally intensive model) is not even executed. This increases safety, saves costs on unnecessary API calls, and reduces waiting times for requests that would be rejected anyway. You can find more insight into the cost aspect in our article on choosing between free or paid solutions.
Output Moderation (Response filtering)
Even if a prompt seems completely innocent, an LLM can unexpectedly produce harmful, inappropriate, or 'hallucinated' answers. Output moderation checks the raw output of the main model before it is shown to the end user. If the generated text violates policy rules (for example, by unintentionally revealing personally identifiable information (PII)), the system can replace the response with a standard error message or a censored version.
Many developers assume that safety filtering barely affects response time. In practice, adding an output moderation step can increase the latency (delay) of the application by hundreds of milliseconds, especially if this is done via an external LLM-based model. This must be explicitly taken into account for real-time chat applications.
Two Primary Architectures: Classification vs. LLM-Based Moderation
Broadly speaking, there are two technical approaches to moderating text: models specifically trained to classify text, and generative models deployed as a referee (LLM-as-a-Judge).
1. Classification-Based Moderation Models
These are more traditional Machine Learning (ML) models (often based on architectures like BERT or RoBERTa) that are specifically trained for a single task: recognizing unwanted categories in text. They read the input and provide a score between 0 and 1 for various risk categories (e.g., violence: 0.95, hate speech: 0.12).
- Speed: Very high. Because they do not need to generate text, the latency is minimal (often less than 50 milliseconds).
- Costs: Low. They require much less computational power than generative models.
- Limitations: They are rigid. They understand context and nuance less effectively than a large language model. They can falsely flag sarcasm or medical discussions as harmful.
Well-known examples of this are the OpenAI Moderation API (free to use for OpenAI customers) and Google's Jigsaw Perspective API, which focuses heavily on detecting toxicity on platforms.
2. LLM-Based Moderation (LLM-as-a-Judge)
With this approach, you use another (often smaller and optimized) generative language model to evaluate the text based on an explicitly described policy or prompt (a so-called rubric). You send the input or output to this model with the instruction: "Read this text and determine if it violates the following rules..."
- Speed: Lower. The model has to process the context and often generate a judgment (and corresponding reasoning).
- Accuracy: Very high for complex, nuanced contexts. It handles complex policy rules exceptionally well.
- Customizability: You can easily adjust the policy by changing the instruction prompt of the 'judge', without needing to retrain the model.
A strong example of this is Meta's Llama Guard. This is a specially fine-tuned Llama model trained to classify safety risks based on a customizable taxonomy. More information on secure AI development can be found externally, such as on platforms covering AI governance and safety policy.
Specific Challenges for the Dutch Language
A crucial aspect when choosing a moderation model is language support. Many classification models and safety frameworks are primarily trained on English-language data. This brings specific challenges for Dutch-language applications.
If you use an English-oriented classification model for Dutch text, the chance of so-called 'false negatives' (unwanted text slipping through) increases significantly. Slang, street language, and specific Dutch cultural contexts are simply not recognized. For example, a word that is innocent in Dutch but has a double meaning in English might be falsely blocked (a false positive), or vice versa.
On the other hand, using LLM-as-a-Judge with large, multilingual models (such as GPT-4o, Claude 3.5 Sonnet, or a large Llama 3) often works surprisingly well for Dutch, as these models have a broad understanding of the language. However, smaller specialized models like Llama Guard may struggle with complex Dutch sentence structures unless they are specifically fine-tuned. View our in-depth Dutch language and LLM comparison for more insight into how models perform on Dutch-language content.
The Balance Between Safety and Usability: False Positives and Over-refusal
The biggest challenge when implementing moderation models is finding the balance between safety and user experience. If a moderation system is configured too strictly, it leads to over-refusal. This means the model refuses to answer legitimate requests.
Suppose you are building an educational AI for history. A user asks: "What were the strategic motivations of Nazi Germany in 1939?". A poorly tuned safety model might only detect the word "Nazi" or "Germany in 1939" and immediately block the request due to "hate speech or extremism". This significantly frustrates the user.
Tuning the safety thresholds is therefore an iterative process. With classification models, you can adjust the threshold value per category (e.g., only intervening at a toxicity score > 0.8 instead of 0.5). With LLM-based moderation, you can refine your system prompts by adding examples (few-shot prompting) of what is allowed within your specific domain.
How Do You Choose the Right Moderation Approach for Your Application?
The choice between services and models depends on your use case. Below, we outline three common scenarios and the most suitable moderation strategy.
Scenario 1: Internal Business Assistants (RAG)
If you are building a Retrieval-Augmented Generation (RAG) system for internal use, the likelihood of intentional 'jailbreaks' is relatively small (after all, employees just want to do their work). Here, the main concern is preventing data leaks and hallucinations.
- Recommended approach: Use a light form of input moderation. Focus output moderation on factual verification (e.g., with an LLM-as-a-judge checking if the response is strictly based on company documents) and filtering PII. Open-source tools like NVIDIA's NeMo Guardrails are highly suitable for this.
Scenario 2: Public Customer Service Chatbots
A bot on your public website can be used by anyone. Users will try to make the bot say silly things for viral screenshots. Latency is crucial here; users expect an immediate response.
- Recommended approach: A lightning-fast classification API (such as OpenAI Moderation API or a local fast classifier) on the input to immediately cut off toxicity and hate. For the output, you can perform a light lexical check or run the response through a fast, domain-specific classifier. Avoid heavy LLM checks on every output due to response time.
Scenario 3: Free Creative Applications or Social Networks
Think of platforms where users write stories together with AI or play interactive role-playing games. The context here is often highly nuanced and sometimes on the edge of what seems safe, but acceptable within the context of fiction.
- Recommended approach: Here, rigid classification models often fall short due to too many false positives. An LLM-based moderation model (such as Llama Guard) is necessary. You can write out the 'rubric' (the rules of the game for the judging model) exactly, so the model knows that violence in the context of a fantasy story is allowed, but instructions for real violence are strictly prohibited.
| Criterion | Classification-based (e.g., OpenAI Moderation) | LLM-based (e.g., Llama Guard) |
|---|---|---|
| Speed / Latency | Very fast (< 50ms) | Slower (100ms - 1s+) |
| Costs | Low or often free (via APIs) | Higher (generating/processing tokens costs money/compute) |
| Context Understanding | Limited (risk of false positives) | Very high (understands nuance and fiction) |
| Customizability | Rigid (fixed categories) | Flexible (customizable via prompts/rubrics) |
Conclusion and Next Steps
Securing generative AI applications is not a one-time task, but an ongoing process. Understand that no model is one hundred percent foolproof. Combining fast classification models for rough filtering at the front door with more intelligent LLM-based moderation for complex, context-dependent output currently offers the most robust line of defense.
Thoroughly test your moderation system with datasets specifically targeted at your Dutch audience, take the impact on response times into account, and continuously monitor for false positives to keep the user experience optimal. This way, you build applications that are not only intelligent, but also responsible and reliable.