Function calling explained: how LLM tools work in production

The name "function calling" makes it sound like the model runs your code. It doesn't. The model never executes anything. What actually happens is a structured conversation: the model asks your code to do something, your code does it, the model uses the result.

That distinction matters because it changes how you build with it and where it breaks.

What the model actually does

When you attach a tool definition to an API call, you give the model a menu of actions it can request. The definition has three parts: a name, a plain-English description of what it does, and a JSON schema describing the expected arguments.

A minimal tool definition looks like this:

{
  "name": "get_order_status",
  "description": "Returns the current status and estimated delivery date for an order. Use this when the user asks about their order.",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": { "type": "string" }
    },
    "required": ["order_id"]
  }
}

When the model's response includes a function call, it returns a structured block with the tool name and the arguments it wants to pass. Your code receives that, runs the actual function, and sends the result back in the next message. The model reads the result and generates its final response.

The model itself does exactly two things: decide whether to call a tool, and specify what arguments to pass. All execution is in your code.

The request-response loop in practice

A typical tool-calling exchange has four messages:

1. User message: "Where's my order #4821?"
2. Model response: { tool: "get_order_status", args: { order_id: "4821" } }
3. Tool result: { status: "In transit", delivery: "2026-07-12" }
4. Model response: "Your order is in transit and should arrive on July 12th."

In the API, step 2 comes back as a tool_use content block (Anthropic) or a tool_calls array (OpenAI). Your code extracts the function name and arguments, calls the actual function, and returns the result as a tool_result message. The model then has what it needs to produce the final answer.

If the task requires several tool calls, the loop repeats. Frontier models (Opus 4.8, GPT-5.5, Fable 5) can also request multiple tools in parallel on a single turn when the request calls for it.

One constraint worth knowing: function calling is synchronous from the model's perspective. It asks for one result, gets it, then continues. If your function takes 30 seconds to run, the whole response waits. Design slow functions accordingly. Alternatively, run them asynchronously and push the result when ready.

See the LLMTest API reference for the complete tool schema format, including Anthropic's tool_use format and the OpenAI-compatible tool_calls format the proxy supports.

Function calling vs plain prompting

Function calling is the right choice when your code needs to take an action or access live data based on what the model decides. The clearest cases:

• Live data access: Search results, database queries, weather, stock prices. The model can't know these from training; it needs to ask.
• Deterministic actions: "Create a GitHub issue" or "Book this meeting," where the model picks the action and your code executes it.
• Structured output routing: Classify an incoming request and route it to one of several handlers, each defined as a tool.

Plain prompting works better when you want unstructured text, when the output format is always the same (summaries, rewrites, drafts), or when you're already parsing the model's response with a JSON prefix in the prompt. Adding a tool schema for something that plain prompting handles in one pass adds API complexity with no benefit.

A practical rule: if you're writing "always respond with JSON in this exact format" in your system prompt, consider whether a tool definition would be cleaner. Tool definitions are typed, documented in the API schema, and easier to validate than free-form text parsing.

Three pitfalls that bite in production

Runaway loops. The most expensive failure mode. A tool returns an error or partial data. The model retries. Same result. Forty turns later, the session is dead and the cost isn't. The fix is basic but easy to skip: set a max-turn limit and a cost cap before the loop starts. Our post on stopping runaway agent loops in Node.js has the implementation, including how to bail out without losing partial results.

Schema drift. You update the function signature in your application code but forget to update the tool definition JSON sent to the API. The model passes arguments that match the schema it saw, which no longer matches your function's actual parameters. Your code throws a KeyError or undefined at runtime. The fix: generate tool schemas from your code, not separately. A TypeScript function with a Zod schema or a Python function with Pydantic can produce the tool definition automatically, keeping schema and implementation in sync.

Context bloat from parallel calls. Frontier models can request multiple tools in a single response turn. If you have 8 tools and a complex request, the model might call 6 of them at once. Each result gets appended to the context before the next turn. A tool that returns 2,000 tokens of data, called 6 times in parallel, adds 12,000 tokens to the next turn's cost. Monitor your tool result sizes in production and truncate aggressively. The context window guide shows how token costs compound over long sessions.

Function calling in RAG pipelines

One place function calling earns its complexity is retrieval-augmented generation. The retrieval step in a RAG pipeline maps naturally to a function call: you give the model a search_documents(query) tool, it calls the tool when it needs context, and uses the returned chunks to generate its answer.

This is more flexible than hard-coded retrieval before the prompt because the model can decide whether retrieval is even needed, what query to run, and whether to search again if the first result was insufficient. The cost tradeoff is real: each retrieval adds tokens and latency. For chat applications where retrieval is always needed, baking it into the prompt pipeline is cheaper. Tool-based retrieval wins when the need for external data is conditional.

FAQ

Is function calling the same as tool use? Same mechanism, different names. OpenAI coined "function calling." Anthropic calls it "tool use." The underlying flow (model returns a structured call object, your code executes it, you return the result) is identical across both APIs. Schema field names differ slightly between providers, but most LLM gateway proxies handle both formats transparently.

Can I use function calling without building a full agent? Yes. Single-step function calling is common and straightforward. The model returns one tool call, you run it, done. No loop, no agent state machine required. Agent loops come in only when the model needs to act on the result before giving a final answer, which is a narrower use case than most tutorials imply.

Which models support function calling? All current frontier models do: Claude Opus 4.8, Claude Sonnet 5, Claude Fable 5, GPT-5.5, GPT-5.6 Sol, Gemini 3.1 Pro. Most budget models do too: Haiku 4.5, GPT-4o-mini, Gemini 2.5 Flash. The mechanics are identical across tiers. You only need a frontier model when you want better judgment on when to call a tool, not for the calling mechanism itself.

How do I stop the model from calling the wrong tool? Two things move the needle. First, write descriptions that are explicit about scope: "queries the internal product CRM for a contact by email; does NOT search public web sources" beats "searches for a contact." Second, keep your tool count small. Models degrade noticeably above 20 tools. If you have 30, group them and expose only the relevant subset per request.

What's the cheapest model to develop and test with? claude-haiku-4-5 or gpt-4o-mini. The function-calling mechanics are identical to frontier models. Switch to Opus 4.8 or GPT-5.5 only when you need better judgment on which tool to call, not for the calling plumbing itself.

Do I need to handle the case where the model skips all tools? Yes. The model may decide no tool call is necessary and respond in plain text. Always check whether the response contains a tool call before parsing. If you're in a loop and the model responds with text, treat that as the loop's terminal condition — it's the model signaling "task complete."

The minimal working implementation

The simplest function-calling setup worth shipping has three parts: one tool with a precise description and tight schema, an execution loop that checks for tool calls on every model response, and a max-turn limit set before the loop starts.

Add parallel calls, schema validation, and cost caps after the basic flow works on real inputs. The pitfalls above hit hardest when you're scaling an already-working setup, not when building from scratch.

Track what your tools actually cost per invocation with LLMTest — it breaks down latency and spend at the individual tool-call level, so you can spot which tools are driving your API bill.