Model selection for function calling and tool use
Deploying large language models for function calling and tool use requires a focused approach. Where generative tasks mainly rely on fluent language ability, task-specific interaction with external systems places strict demands on a model's accuracy. This article covers the criteria for selecting models that reliably generate structured data and drive APIs.
The fundamentals of tool use
Function calling enables a model to determine, based on a user query, which external function or API needs to be called. The model receives a definition of available functions in a specific schema, such as JSON Schema, and returns a structured call with the correct arguments. This differs fundamentally from free text generation, because the output must exactly comply with the syntax the target system expects.
A model that fails at this precision directly causes errors in the underlying application logic. Selecting the right model therefore requires insight into how models are trained to interpret schemas and correctly fill in parameters. The assumption that any large model is automatically suitable for this task turns out to be incorrect in practice; model architectures differ significantly in their ability to process complex and nested parameter schemas flawlessly.
Architectural requirements for models
Not every model has the necessary capabilities for reliable tool use. Specific architectural properties play a decisive role in the selection. Models must follow instructions meticulously while also retaining context across multiple successive steps in a process.
Syntax calibration and JSON generation
A model's output during function calling must be syntactically correct. A missing bracket or a misplaced quotation mark leads to a failed API call. Models specifically fine-tuned for generating valid JSON structures have a demonstrable advantage here. This is closely related to the techniques described on structured output, which goes deeper into enforcing fixed output formats.
The assumption here is that models that perform well on general programming tasks are also automatically better at function calling. Although there is a strong correlation, this premise shows exceptions in practice. Some models with strong programming capabilities still struggle with contextualizing API definitions expressed in natural language.
Context window and parameter complexity
As an application grows, the number of available tools also increases. A system can contain hundreds of different functions, each with dozens of parameters. The model must be able to process all these definitions within the context window without losing focus on the core instruction. Larger context windows provide room for extensive tool definitions, but also introduce risks of 'lost in the middle' behavior, where the model ignores instructions from the middle of the prompt.
The limitation of many open-source and proprietary models is that their accuracy drops significantly once the number of tools exceeds a certain threshold. This requires careful routing of tools instead of offering all functions to the model at once.
Evaluation criteria for selection
Determining the right model choice requires a structured evaluation. Measuring performance in tool use cannot rely solely on general benchmarks such as MMLU or HumanEval. Specific test suites are needed that measure the reliability of API calls.
Latency factors and throughput
Function calling is often part of synchronous user interactions, such as chatbots or automated assistants. The speed at which a model generates a tool call is therefore crucial. Smaller models are generally faster, but sometimes lose out on correctly understanding ambiguous parameter definitions. Larger models offer more reliability, but introduce delay that negatively affects the user experience.
An alternative to deploying one large, slow model is to use a tiered two-stage architecture. Here, a small, fast model determines which tool is needed, after which a larger model fills in the more complex arguments. This approach reduces total latency, but increases the complexity of the application architecture.
Reliability and determinism
Production environments require predictability. Models naturally exhibit a certain degree of stochastic behavior. Lowering the 'temperature' parameter to zero helps achieve deterministic results, but does not solve the problem that models sometimes hallucinate parameter values for fields that do not appear in the context.
For broader processes in which models autonomously make decisions across multiple successive steps, function calling is only one part of a larger whole. Anyone designing such a setup can find background information on autonomous workflows at models for agents, which discusses the constraints for agent systems.
Comparison of model types
The market currently broadly has three categories of models eligible for function calling: large proprietary cloud models, open-weight models, and fine-tuned task-specific variants.
| Model Category | Strengths | Weaknesses |
|---|---|---|
| Proprietary cloud models | High accuracy with complex schemas, large context | High costs, dependency on external provider, privacy risks |
| Open-weight models | Full control, can run locally, lower operational costs | Require substantial hardware, variable quality per model |
| Task-specific fine-tuned models | Optimal performance on a limited set of tools | High initial training costs, inflexible with changing APIs |
Proprietary models excel at understanding vague or poorly documented tool definitions. They independently correct small inaccuracies in the user's query. The downside is dependency on the API provider's uptime and possible compliance issues when sending sensitive business data to external servers.
Open-weight models offer the advantage of local hosting, which is necessary within sectors with strict privacy requirements. The weakness lies in configuration: correctly setting up the system prompts and parsing the output requires extra development work on the application side. In addition, open-weight models fall back more often into free text generation instead of adhering to the enforced schema.
Implementation risks and mitigation
Implementing function calling carries specific risks that are separate from general model performance. These risks are mainly found at the intersection of security and data validation.
A well-known risk is prompt injection via external data. When a model retrieves content from an external website or database and processes this data in the context, malicious input can contain instructions that induce the model to call unauthorized functions. The model could, for example, be tricked into calling a deletion function in a database.
To mitigate this risk, model output must never be blindly trusted. Every generated API call must pass through a deterministic validation layer before it is actually executed. This validation layer checks whether the requested action falls within the permissions of the current user and whether the parameters meet strict safety margins.
Practical guidelines for the choice
When making a definitive model choice for function calling, an organization should proceed iteratively. Don't start immediately with the largest and most expensive model, but begin with a representative test set of your own API definitions and common user queries.
Systematically measure what percentage of test queries results in a flawless tool call. Take edge cases into account here, such as missing required parameters or contradictory instructions. It often turns out that a smaller open-weight model, equipped with a precise system prompt and a structured parser, performs just as well as an expensive proprietary model on a narrowly scoped task.
When the technical setup of the API calls and the communication layers between application and model are further optimized, it is advisable to also evaluate the underlying protocols. Technical details on setting up structured communication lines can be found via function calling, which goes deeper into the technical protocols behind model-driven API integrations.
Choosing a model for function calling is therefore never a one-time decision, but a continuous process of testing, measuring, and adjusting as APIs and model versions succeed one another.
