Research Paper: Teaching Small Models When Not to Call Functions

By Dung Vo, Data Science R&D, UNEY

Our short paper, Teaching Small Models When Not to Call Functions: Structured Reasoning for Tool Refusal in Low-Resource Languages, was accepted at the Short Papers Track of SIGIR 2026 - the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, held in Melbourne, Australia, in July 2026. The work was done by Dung Vo, Thai Tran, and Tushar Semwal from UNEY Data Science R&D.

The paper focuses on a failure mode we kept seeing in small tool-using models:

A model may attempt a function call even when the available tools cannot solve the user's request.

This is a decision problem, not only a formatting problem.

Before a model calls a function, it has to answer a few concrete questions. Does the request need a tool? Is there a relevant tool? Is that tool sufficient? Are the required inputs present? Should the model call, ask for clarification, answer directly, or defer?

The paper is about that decision.

Problem: relevant tools are not always sufficient

Consider this request:

Assume the model has only this tool:

The tool is related to the request. It can inspect metrics. But it is not sufficient.

It can inspect one named service. It cannot scan all services. It cannot restart anything. The user also did not provide the required service_name.

A correct response should not invent a function call. It should explain the limitation or ask for the missing information:

I can check metrics for a specific service if you provide its name, but I do not have a tool that can scan all production services or restart them.

This is the core failure mode: small models often treat a partially relevant tool as if it were sufficient.

A hallucinated text answer is one kind of failure. A hallucinated tool call is different because the model is trying to act through an external interface.

The product constraint behind the research

We did not start from a benchmark curiosity. The problem came from a practical constraint.

Small models are attractive because they can reduce inference cost, latency, and deployment complexity. They are also relevant when inference needs to happen closer to the user, the device, or the data.

But small models are useful only if their decisions are reliable.

In product-facing AI systems, tool calls are not abstract benchmark outputs. They may represent retrieval, workflow actions, system queries, document operations, or other side effects. A model that calls a related but insufficient tool can create operational noise. A model that fabricates a missing tool can make the system appear more capable than it is. A model that fails to ask for missing inputs can turn an incomplete request into a wrong action.

This is the practical constraint behind the paper: small models need to stay efficient, but they also need to know when acting is not justified by the available tools.

Evaluation issue: malformed calls can look like refusals

The first issue was not the model. It was the measurement.

Some function-calling evaluations rely on format compliance. If the model outputs a malformed tool call and the parser fails, the evaluator may see "no valid tool call" and count the answer as a refusal.

But the model did not refuse. It attempted a call and failed to format it.

Example:

If no weather tool exists, this should be counted as an attempted hallucinated tool call. A parser-based evaluator may miss that intent because the call is malformed.

We therefore use intent-based evaluation for refusal-related metrics. The evaluator checks raw output for tool-call intent, including explicit tool tags, function-like syntax, JSON structures, bracket patterns, and direct function-call patterns.

After this correction, the baseline is much less safe than it first appears. Standard format-compliance evaluation inflated refusal-related scores by about 30 percentage points in our experiments.

The model was not refusing as often as standard evaluation suggested. Many malformed tool-call attempts were being counted as correct refusals.

For refusal tasks, we should measure what the model tried to do, not only whether the parser accepted the output.

Why standard fine-tuning is insufficient

A natural fix is to fine-tune the model on function-calling data.

This improves syntax. The model learns function names, argument formats, and schema patterns. It becomes better at producing valid calls.

But syntax is not judgment.

In our training mixture, most examples were positive tool-call examples. Only a small fraction explicitly represented irrelevance cases. This is common in tool-use datasets: models see many examples where tools should be used, and far fewer examples where tools should not be used.

In one setting, standard fine-tuning improved tool-call formatting but degraded live hallucination behavior by about 21 percentage points. The model became more fluent at calling tools, but not more careful about whether a tool should be called.

The issue is direct:

Standard fine-tuning teaches the model how to call tools. It does not reliably teach the model when not to call them.

For deployment, this distinction matters. A model that becomes better at producing valid calls can still be unsafe if it has not learned when the current toolset is insufficient.

Structured Reasoning Forms

We propose Structured Reasoning Forms: compact key-value reasoning schemas generated before the final response.

The goal is targeted reasoning, not longer reasoning.

For the production-service example, a reasoning form can look like this:

The key field is tool_sufficiency.

A tool can be relevant and still insufficient. That is the decision point standard tool-call training often leaves implicit.

The schema decomposes the decision into a small number of stages:

• task understanding
• ambiguity and missing inputs
• tool relevance
• tool sufficiency
• decision selection
• call planning
• hallucination risk

Each field supervises part of the decision. The model is not only trained to produce the final answer. It is trained to represent the decision process that should come before the answer.

This is why we treat the method as a data-supervision approach. We do not change the learning algorithm. We change the structure of the training signal.

When the model should call tools

The method is not meant to make models refuse by default.

For example:

Assume the available tools are:

Here, the tools are sufficient. The expected behavior is to call them.

A structured form can represent that decision:

The desired behavior is a sharper boundary between valid tool use and unsafe tool use.

Both mistakes hurt in practice. Calling tools too aggressively creates risk. Refusing too often makes the agent less useful. The useful behavior is the middle: act when the toolset is enough, ask when information is missing, and defer when the requested action is outside the current toolset.

Language setting

We evaluate across a high-resource language setting and lower-resource language settings.

The lower-resource settings cover different script conditions, including Latin-script and non-Latin-script settings, as well as alphabetic and abugida-based systems.

This framing is intentional. The contribution is not about one specific language pair. The broader question is whether small models can make reliable tool-use decisions across different linguistic and script conditions.

For multilingual agentic systems, evaluation in the easiest setting is not enough.

Training setup

We treat this as a data-supervision problem.

For the reasoning-form variant, we used Qwen 3 80B only as a schema formatter. It prepended structured reasoning forms to existing ground-truth targets, without generating final answers or changing labels.

So the method is not answer distillation from a larger model. The larger model only helps format the decision structure. The gain comes from making implicit tool-use decision logic explicit during training.

The training data contains 50,863 examples from multiple function-calling sources. Most of this data consists of positive tool-call examples, which reflects the imbalance described earlier. The structured form is added to teach the missing decision process around when tools should not be used.

At inference time, the model generates the reasoning form before the final response. That adds some overhead, but the structure keeps the reasoning bounded.

Main results

We evaluate two multilingual 1B models.

We also create new lower-resource function-calling benchmarks with 3,641 samples per setting, validated with help from native speakers. Malformed tool-call attempts are separated from true refusals so the model cannot receive refusal credit for formatting failure.

To keep the numbers comparable, the table below reports the same kind of information for each accuracy metric: the range after standard fine-tuning, the range after Structured Reasoning Forms, and the observed change across the evaluated model families and language/script settings. Because the experiments cover two 1B model families and multiple language/script settings, the table reports ranges rather than single numbers.

The important part is not only that refusal improves. The model keeps useful tool-calling capability instead of simply becoming over-conservative.

Reasoning cost is a separate question, so it is better to show it separately:

This is the trade-off we cared about from the start. In constrained systems, reasoning is useful only if it improves behavior without creating too much cost. A long reasoning trace that weakens tool-calling accuracy is not a good trade-off.

Does the reasoning content matter?

We test this with a randomized reasoning ablation.

If the model only needed extra tokens or a fixed reasoning schema, randomized reasoning should help almost as much as correct reasoning.
It does not.

The gap is the point. The model benefits from the logical content of the reasoning form, not only from seeing a template.

This is also why the form is kept explicit. The purpose is not to decorate the output with reasoning. The purpose is to supervise the decision variables that determine whether a tool call is justified.

Internal behavior

We also measure the model's tendency to emit a tool-call marker in cases where no suitable tool exists.

This is the strongest signal that the change is not just surface-level.

The model is not only learning to phrase a refusal. It becomes less likely to reach for a tool when the toolset is not enough.

Why this is useful for deployed agents

For deployed tool-using agents, valid function-call syntax is only one requirement. The model also needs to know when the current toolset is insufficient.

This affects several common failure cases:

• missing required arguments
• related but insufficient tools
• unavailable actions
• unsupported side effects
• tool calls that look plausible but cannot complete the task

This is the part that matters outside the benchmark.

At UNEY, the research direction is shaped by constraints that appear in real systems: latency, cost, privacy, multilingual use, and limited compute budgets. Small models are attractive under these constraints, but only if their tool-use decisions are reliable.

Structured Reasoning Forms give one practical way to supervise that decision without relying on long free-form reasoning.

The method is also compatible with further compression. In preliminary experiments, schema-level summaries reduced reasoning traces to about 71 to 90 tokens per turn. Repeated schema keys and categorical values may be compressible further with dedicated tokens.

For small models, that detail matters. A decision process that is accurate but too expensive may not survive contact with a real product.

Limitations and next steps

This is a short paper, not a complete solution to agentic reliability.

More work is needed on longer workflows, dependent tool calls, parallel calls, stronger side effects, latency measurement, and product-specific evaluations.

The multilingual direction should also be expanded to additional lower-resource settings and broader script families.

The main takeaway is narrower and more concrete:

Small tool-using models should be evaluated and trained not only on how to call functions, but also on when the available functions are not enough.

Acknowledgment

This work was done with Thai Tran and Tushar Semwal at UNEY Data Science R&D.

We thank UNEY leadership for supporting research connected to real deployment constraints, colleagues across product, engineering, and AI teams for practical feedback, the native speakers who helped validate the benchmark data, the reviewers for constructive feedback, and the open-source community for the models and datasets that made this work possible.

About SIGIR 2026

SIGIR 2026 is the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval - a CORE Rank A* venue and the flagship conference in the IR field. It will be held in Melbourne, Australia, from 20–24 July 2026, with tutorials and the Doctoral Colloquium on 20 July, the main conference from 21–24 July, and workshops on 24 July.

More info: SIGIR 2026 website · Program overview · Short Papers Track

Meet us at SIGIR 2026 to discuss in depth.