Tool and function calling: the LLM's hands

A production system was calling the model with a weather tool. The engineer thought of the city parameter as a string; the schema actually said "type": ["string", "null"]. The model worked perfectly in testing. In production, for a specific class of queries, it started passing { "city": null } — perfectly valid against that schema, and a crash at the API call site that took 45 minutes to trace because nothing upstream had logged the tool arguments. The fix was deleting one word from the schema: the "null". That's the level of precision tool calling requires.

What tool calling actually is

Tool calling (also called function calling) is a structured protocol that lets an LLM express intent to invoke an external function. The model does not execute anything. It returns a JSON object naming the function and the arguments. Your application code reads that object, calls the real function, and sends the result back.

This matters more than it sounds. The model is entirely sandboxed from real-world side effects. It hallucinates parameters, but your validator catches them before they reach the database. It picks the wrong tool, but your retry logic catches the error. Every real-world consequence is mediated by code you control. The model is giving you instructions. Executing them is your job.

The basic data flow never changes across providers:

1. You send a messages array plus a tools list. Each tool has a name, description, and a JSON Schema for its parameters.
2. The model either responds in plain text (stop_reason: end_turn) or emits one or more tool call blocks (stop_reason: tool_use on Anthropic; finish_reason: tool_calls on OpenAI).
3. You execute the requested function(s), collect the result(s), and append them as tool_result messages.
4. You call the model again with the updated message history.
5. Repeat until end_turn.

The sibling article The Tool-Use Round Trip: Agentic Loops, Stop Reasons, and Parallel Calls covers the multi-step mechanics in depth. This article focuses on getting the first call right.

Writing a minimal tool definition

Here is a working tool definition on the Anthropic API:

import anthropic

client = anthropic.Anthropic()

tools = [
    {
        "name": "get_weather",
        "description": (
            "Retrieve the current weather for a city. "
            "Use this when the user asks about weather conditions, temperature, or forecast. "
            "Do not call this for historical weather — it only returns current data."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {
                    "type": "string",
                    "description": "City name in English, e.g. 'Tokyo', 'New York', 'São Paulo'."
                },
                "unit": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "Temperature unit. Default to celsius unless the user specifies otherwise."
                }
            },
            "required": ["city", "unit"]
        }
    }
]

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}]
)

# Always check stop_reason first
if response.stop_reason == "tool_use":
    for block in response.content:
        if block.type == "tool_use":
            print(f"Tool: {block.name}")
            print(f"Input: {block.input}")

The equivalent on OpenAI uses functions (the legacy format) or the current tools parameter:

from openai import OpenAI

client = OpenAI()

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": (
                "Retrieve current weather for a city. "
                "Use for current conditions only, not historical data."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "City name in English."
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"]
                    }
                },
                "required": ["city", "unit"],
                "additionalProperties": False   # required for strict mode
            },
            "strict": True   # enforce schema at generation time
        }
    }
]

response = client.chat.completions.create(
    model="gpt-4o",
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}]
)

choice = response.choices[0]
# Check for refusal BEFORE parsing
if choice.message.refusal:
    raise ValueError(f"Model refused: {choice.message.refusal}")

if choice.finish_reason == "tool_calls":
    for tc in choice.message.tool_calls:
        print(f"Tool: {tc.function.name}")
        print(f"Args: {tc.function.arguments}")  # JSON string, not yet parsed

The structure is similar; the field names differ by provider.

Controlling whether the model calls anything: tool_choice

By default, passing tools is an offer, not a command. The model reads the schemas and decides for itself whether to answer in prose or emit a tool call. That default (auto on both providers) is right for open-ended assistants — and wrong for pipelines where a specific tool must fire. tool_choice is the knob.

On Anthropic, tool_choice takes an object:

• {"type": "auto"} — the default. The model may call a tool or answer in text.
• {"type": "any"} — the model must call some tool; plain-text answers are off the table.
• {"type": "tool", "name": "get_weather"} — the model must call this specific tool.
• {"type": "none"} — the model may not call any tool this turn, even though the schemas stay in context.

OpenAI's equivalents are "auto", "required", a named function object ({"type": "function", "function": {"name": "get_weather"}}), and "none".

Force a named tool when the tool is the output — extraction pipelines where you define a single record_entities tool and want a structured result on every call, or a router step that must always emit a classification. Use any/required when the model should act but you don't care which tool it picks. Two caveats. Forcing a tool means the model calls it even on inputs where calling it makes no sense — pair a forced tool with input validation, because "the model had no better option" produces confident garbage arguments. And on Anthropic, the value you pick changes the injected system prompt: the 290–804-token overhead cited below varies with tool_choice, and forcing a tool suppresses the model's usual pre-call reasoning text. Cheap, but not free.

The round-trip protocol, step by step

A single tool call is one turn. Most real applications need more.

flowchart TD
    A[Send messages + tools] --> B{stop_reason?}
    B -->|end_turn| C[Return final answer]
    B -->|tool_use| D[Parse tool_use blocks]
    D --> E[Execute each tool]
    E --> F[Append tool_results to messages]
    F --> G{Budget exceeded?}
    G -->|yes| H[Return partial result / error]
    G -->|no| A

    style H fill:#ff2e88,color:#fff
    style C fill:#15803d,color:#fff

Every iteration re-sends the full message history. The model has no memory between calls beyond what you pass in. This has two consequences. First, the context window grows on every step — a ten-step loop on a verbose tool can consume 30,000–60,000 tokens before the final answer. Second, there is no automatic termination. If the model enters a loop where each tool result prompts another tool call, it will continue until it hits your iteration cap or your token budget. Both caps are mandatory infrastructure, not optional guards.

The visualizer below shows what one agent loop looks like end-to-end, with the token budget counter running:

{ "type": "agent-loop", "scenario": "weather-trip", "title": "Tool-use round trip: weather lookup" }

Strict mode and what it actually guarantees

Strict mode (strict: true on both OpenAI and Anthropic as of 2025) enables constrained decoding on the server side: at each token position, a binary mask zeroes out any token that would produce output violating the schema. The model is physically unable to generate a field the schema doesn't define, or omit a required field, or produce the wrong type.

Strict mode requires two schema properties to be set:

• Every field must be listed under required.
• Every object must set "additionalProperties": false.

Missing either one causes a schema rejection error at request time on OpenAI. On Anthropic, the behavior may degrade silently.

What strict mode does not do: it cannot prevent the model from generating a valid-looking but wrong value. If the schema says "type": "string" for an order ID, the model can still produce "order_id": "ORD-99999" when order 99999 does not exist in your database. Schema conformance is necessary but not sufficient. Semantic validation — checking that the values make sense in your domain — is a second, separate layer you have to build.

For more on how constrained decoding works under the hood (token masking, xGrammar, the CRANE quality tradeoff), see Structured Outputs and Constrained Decoding: Guaranteeing Parseable JSON.

The token cost you're paying silently

Tool definitions are not free. Every tool schema you pass is injected into the context window on every API call. On Anthropic models, the system prompt for tool use itself adds 290–804 tokens depending on the model and whether tool_choice is set — before any schema content. Each tool's name, description, and parameter schema adds more.

Back-of-the-envelope: tool overhead per request

Anthropic tool-use system prompt injection: ~400 tokens (mid-estimate)
Each tool schema (name + description + params):
  - minimal tool: ~80 tokens
  - verbose tool with nested types: ~400 tokens
  - 10 tools × 200 tokens avg: ~2,000 tokens

Total overhead per request: ~2,400 tokens
At $3/M input tokens (illustrative, mid-2026): $0.0072 per request
At 100,000 requests/day: ~$720/day in tool definition overhead alone

A 10-step loop: that overhead × 10 = ~$7,200/day per 100k sessions
A 50-tool catalog (~10,400 tokens): ~$3,100/day per 100k requests —
  ~$31k/day if each session runs a 10-step loop

The implication: only pass tools the model actually needs for the current step. If a tool is irrelevant to this phase of the workflow, remove it from the call. Anthropic's Tool Search feature (2025) lets Claude discover and load tools on demand from a catalog rather than consuming context for all definitions upfront — a practical fix for large tool catalogs.

The context-window article explains why input token counts compound across multi-step loops.

Parallel tool calls

Both providers allow the model to request multiple tool calls in a single response. The model returns a list of tool_use blocks; you execute them concurrently, then send all results back in one turn.

# Anthropic: handle parallel tool_use blocks
tool_results = []
tasks = []

for block in response.content:
    if block.type == "tool_use":
        tasks.append((block.id, block.name, block.input))

# Execute concurrently (asyncio, threading, or a thread pool)
import asyncio

async def execute_all(tasks):
    results = await asyncio.gather(*[
        run_tool(name, input_data) for _, name, input_data in tasks
    ])
    return [
        {"type": "tool_result", "tool_use_id": tid, "content": str(result)}
        for (tid, _, _), result in zip(tasks, results)
    ]

The OpenAI incompatibility: strict: true (structured output mode) and parallel_tool_calls: true are mutually incompatible on OpenAI. If you have both enabled, you must set parallel_tool_calls: false. This is not a warning; the API will reject or silently degrade the combination. On Anthropic, parallel calls work fine with client-side tools.

sequenceDiagram
    participant A as App
    participant M as Model
    participant T1 as Weather API
    participant T2 as Calendar API

    A->>M: "Am I free Thursday and what's the weather?"
    M-->>A: tool_use [get_weather(Thursday), check_calendar(Thursday)]
    Note over M,A: Two tool_use blocks in one response
    par parallel execution
        A->>T1: get_weather("Thursday")
        A->>T2: check_calendar("Thursday")
    end
    T1-->>A: { temp: 18, condition: "rainy" }
    T2-->>A: { free: true, next_event: "5pm" }
    A->>M: tool_results [weather_result, calendar_result]
    M-->>A: "Thursday looks free until 5pm, though it'll be rainy at 18°C."

What breaks

This section is what the happy-path tutorials skip.

Hallucinated tool calls. The model can invent arguments that pass schema validation but reference entities that don't exist — an order ID that isn't in the database, a user ID it composed from context. Strict mode cannot catch this. The defense is domain validation before you execute: check that the IDs exist, that the values are in range, that the combination is coherent.

The refusal path. OpenAI Structured Outputs can return a refusal instead of a tool call block. Code that blindly parses choice.message.tool_calls without first checking choice.message.refusal throws an exception that looks like a parsing error but is actually a model policy decision. Always check the refusal field first.

choice = response.choices[0]
if choice.message.refusal:
    # Handle gracefully — this is a policy decision, not a bug
    return {"error": "model_refused", "detail": choice.message.refusal}

Truncated tool calls. A response that hits the max_tokens ceiling mid-generation comes back with stop_reason: max_tokens (finish_reason: length on OpenAI) — and if the model was partway through a tool call, the arguments are cut off mid-JSON. They will not parse, and worse, a lenient parser can salvage a syntactically-complete prefix that is semantically wrong. Both code samples above set max_tokens=1024, which is plenty for a weather lookup and not for a tool whose arguments include a long document excerpt. Treat max_tokens/length as its own branch: discard the arguments and retry with a raised ceiling. Large schemas and verbose arguments make this more likely, which is another quiet cost of bloated tool definitions.

Context rot in long loops. Each round trip re-sends the entire conversation history, including all previous tool arguments and results. By step eight of a ten-step loop, the model is reading 40,000 tokens of accumulated context to decide what to do next. Earlier instruction drifts into the middle of the context where models attend to it less reliably (see long-context loss-in-the-middle). Tool arguments become subtly less accurate. The system prompt's constraints are more likely to be ignored. The fix is aggressive summarization of completed tool results and a hard iteration cap.

Wrong tool selection at scale. When you have 30 tools in the context, the model's tool selection becomes less reliable. Two tools with overlapping descriptions will see split traffic — sometimes the model picks the right one, sometimes the wrong one. The model decides from the tool's name, description, and parameter schema — and the description is the strongest lever you control. Treat tool descriptions as prompt-engineered contracts: state the exact conditions under which this tool should be called, and the conditions under which it should not.

The OpenAI strict + parallel incompatibility. Worth repeating because it is silently destructive in code that evolves over time. Someone adds strict mode for data integrity. A month later, someone else enables parallel tool calls for speed. Neither change breaks anything in isolation. Together, they break the parallel calls silently or trigger an API error. Pin both settings explicitly and enforce them in code review.

Schema rejection at first request. OpenAI strict mode refuses schemas with required fields missing or without additionalProperties: false on objects. The error comes back at request time, not during parsing. If you're dynamically generating schemas from dataclasses or Pydantic models, add an export step that enforces these constraints.

For a deeper tour of production failure patterns, including loop budgeting strategies from DoorDash and ZenML's analysis of 1,200 production deployments, see Building Reliable Tool-Using Systems: Patterns From Production.

A worked validation layer

Strict mode gets you structural correctness. Domain validation gets you semantic correctness. Here is a minimal pattern that combines both:

import json
from pydantic import BaseModel, field_validator
import anthropic

class BookingArgs(BaseModel):
    customer_id: str
    date: str  # ISO 8601
    service: str

    @field_validator("customer_id")
    @classmethod
    def customer_must_exist(cls, v):
        # In real code: check database
        if not v.startswith("CUST-"):
            raise ValueError(f"Invalid customer_id format: {v}")
        return v

    @field_validator("date")
    @classmethod
    def date_must_be_future(cls, v):
        from datetime import date
        parsed = date.fromisoformat(v)
        if parsed <= date.today():
            raise ValueError(f"Booking date must be in the future, got {v}")
        return v

def handle_tool_call(block):
    if block.name == "create_booking":
        try:
            args = BookingArgs(**block.input)
        except Exception as e:
            # Return the validation error as a tool_result so the model
            # can correct its arguments on the next turn
            return {
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": f"Validation error: {e}. Please correct and try again.",
                "is_error": True
            }
        # Safe to execute
        result = create_booking(args.customer_id, args.date, args.service)
        return {
            "type": "tool_result",
            "tool_use_id": block.id,
            "content": json.dumps(result)
        }

Return validation errors as tool_result messages with is_error: true. The model reads the error and corrects its arguments on the next turn — this works better than re-prompting from scratch because the error is contextually adjacent to the failed call.

For Pydantic-first schema generation and the full validation-retry pattern, see Structured Data Extraction with LLMs.

When to use which approach

flowchart TD
    Q1{"Do you need the model\nto invoke an action?"}
    Q1 -->|Yes| Q2{"Does the action have\nside effects?"}
    Q1 -->|No| Q3{"Need typed output?"}
    Q2 -->|Yes| TC["Tool calling\n(strict mode on)"]
    Q2 -->|No| Q4{"Need parallel calls\non OpenAI?"}
    Q4 -->|Yes| TCnostrict["Tool calling\n(strict off)"]
    Q4 -->|No| TCstrict["Tool calling\n(strict on)"]
    Q3 -->|Yes| SO["Structured Outputs\n(response_format with schema)"]
    Q3 -->|No| JM["Plain text or JSON mode"]

    style TC fill:#a855f7,color:#fff
    style TCstrict fill:#a855f7,color:#fff
    style TCnostrict fill:#00e5ff,color:#0a0a0f
    style SO fill:#0e7490,color:#fff
    style JM fill:#15803d,color:#fff

The full comparison of JSON mode, structured outputs, and tool calling as distinct mechanisms is in JSON Mode vs Structured Outputs vs Tool Calling: What Actually Differs. The quick version:

Situation	Use
Model should call an external API or trigger an action	Tool calling
You want typed output from a document or prompt (no action)	Structured outputs / response_format
You need guaranteed schema conformance and don't need parallel calls (OpenAI)	Strict mode on
You need parallel calls on OpenAI	Strict mode off
Self-hosted inference (vLLM, SGLang) with xGrammar	Grammar-constrained generation with a JSON schema
Large tool catalog	Tool Search (Anthropic) or dynamic tool filtering per step

Programmatic tool calling (Anthropic, 2025)

For workflows with large tool sets, Anthropic's Programmatic Tool Calling (released under the advanced-tool-use-2025-11-20 header) changes the approach: instead of selecting from a menu of tools and generating arguments, Claude writes a short Python program that orchestrates tool calls. The program is executed in a sandboxed runtime, and the results are returned.

The practical benefit is token efficiency: at scale, passing 50 tool schemas on every turn is expensive. With programmatic tool calling, you pass a tool registry description and Claude generates the orchestration code. Anthropic's own benchmarks report roughly 38% reduction in billed input tokens on large tool sets compared to the standard tool-calling protocol.

This is a more advanced topic and assumes a sandboxed execution environment. The trade-off is complexity of the execution runtime versus context window savings at scale.

Designing your first schema

The most common beginner mistake is treating the description field as optional metadata. It is not. The description is the model's primary signal for deciding whether to call this tool — the name and parameter schema fill in the rest. A two-word description like "Gets weather" will cause the model to hallucinate parameter formats and to call the tool at exactly the wrong moments.

Three rules for production-ready tool descriptions:

1. State when to call it, not just what it does. "Use this when the user asks about current temperature or weather conditions. Do not use for historical data or forecasts."
2. State what you expect in the parameters. "City name in English, not abbreviated. Use 'New York' not 'NYC'." The model has no way to infer format constraints from type alone.
3. State what you won't do. "This tool only supports US addresses. For international addresses, use get_international_weather instead."

For schema patterns beyond the basics — namespacing tools in multi-agent systems, handling optional vs required parameters, returning useful values from tools rather than opaque IDs — see Designing Tool Schemas That Models Actually Understand.

The production checklist

Before shipping any tool-using system to production:

• Hard iteration cap set (10 steps is a reasonable default; adjust down for latency-sensitive workflows).
• Hard token budget per session set — total input + output tokens across all turns in a loop.
• stop_reason and message.refusal checked before any parsing.
• Domain validation (not just schema validation) runs before tool execution.
• Validation errors returned as structured tool_result messages, not raised as exceptions that kill the loop.
• Tool count per request reviewed: remove tools that are irrelevant to the current workflow step.
• Strict mode setting explicitly documented in code alongside the parallel_tool_calls setting, since the interaction between them is a footgun.
• Each tool call logged with: tool name, input arguments, result, latency, and token cost. You cannot debug a production agentic system without this trace.

The observability piece is covered in LLM Observability: Monitoring Quality, Cost, and Drift in Production. The full production reliability pattern — circuit breakers, fallback models, idempotency — is in Building Reliable Tool-Using Systems: Patterns From Production.

The judgment call

Tool calling is the right primitive whenever you need the model to make decisions that have consequences in the real world — a database query, a workflow kicked off on its say-so. It is not the right primitive for getting structured data out of a document where no action is involved; use structured outputs there instead.

The key discipline is maintaining the boundary between the model's decision layer and your execution layer. Every real-world consequence — the database write, most obviously — flows through code you control, with validation you write, with error handling you design. The model's tool call is a request. Granting it is your choice.

Start with one tool, strict mode on, an iteration cap of five, and full logging. Get that working before you add the second tool. Most production tool-use failures trace back to schemas that made sense to the engineer writing them but not to the model calling them, combined with a missing validation layer that was always going to be added "later."