Serve a Local Tool-Calling Model for Your Agent with llama.cpp

Your agent needs a tool call. The model answers with a friendly paragraph that contains JSON, or worse, with a sentence about the weather instead of the structured get_current_weather call your code is waiting for. Running an LLM locally is the easy part now. Getting a local endpoint that returns OpenAI-shaped tool calls, every time, on the hardware sitting in front of you, is where most setups break.

This is a build guide for that second part. By the end you will have a llama-server instance that speaks /v1/chat/completions with tools enabled, a Python loop that executes those tool calls, and a way to measure whether the quantization you picked quietly made the model worse at calling tools.

Prerequisites

• Docker, or a C++ toolchain (CMake and a compiler) if you build from source. Prebuilt binaries for Linux, macOS and Windows are on the releases page too.
• Python 3.10 or newer, only if you want to convert your own weights from Hugging Face.
• Enough RAM or VRAM for the weights plus the KV cache. The upstream docs give the ballpark for Llama 3.1:

Model	Original size (bf16)	Quantized size (Q4_K_M)
8B	32.1 GB	4.9 GB
70B	280.9 GB	43.1 GB
405B	1,625.1 GB	249.1 GB

• Free disk space for the intermediate file. Conversion writes a full-precision GGUF before quantization, so you need room for that copy plus the final quant.

On the KV cache side, memory grows with -c (context size) and -np (parallel slots), and -ctk / -ctv let you store the cache at a lower precision than the default f16. Recent builds also default --fit to on, which adjusts unset arguments so the model fits in device memory, with -fitt / --fit-target controlling the margin in MiB.

Step 1: pick the quant before you download anything

Quantization trades weight precision for size. What that trade costs in speed at modern formats is smaller than most people expect. Here is the upstream benchmark table for Llama 3.1 8B, measured on the llama.cpp maintainers' machine:

Quant	Size (GiB)	Prompt t/s @512	Generation t/s @128
F16	14.96	923.49	29.17
Q8_0	7.95	865.09	50.93
Q6_K	6.14	812.01	58.67
Q5_K_M	5.33	758.69	67.23
Q4_K_M	4.58	821.81	71.93
Q3_K_M	3.74	783.44	71.68
Q2_K	2.95	784.45	79.85

The F16 model is the slowest row and the biggest file. The 4-bit quants run more than twice as fast in generation. Those numbers are from one rig, so treat them as a shape rather than a promise, but the lesson holds: going smaller buys you RAM, not speed. The docs are also blunt that a smaller quant can degrade overall quality, with the impact on speed and memory described as negligible.

Practical defaults:

• Start with Q4_K_M. It is the balance point most people land on, and it is the default qualifier when you download from Hugging Face without specifying one.
• Move up to Q6_K or Q8_0 if you have the memory and your tasks are precision-sensitive.
• Drop to Q3_K_M or Q2_K only when you have no other option, and use an importance matrix when you do.

Step 2: get a GGUF file

Two paths. Pick one.

Download an existing quant. llama-server takes a Hugging Face repo directly, no manual download step:

llama-server -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M

If you leave the quant off, it defaults to Q4_K_M and falls back to the first file in the repo. Files land in the local cache, -cl / --cache-list shows what is already there, and --offline stops it from reaching the network.

Convert your own fine-tune. This is the path when the model you want does not have a GGUF build yet:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
python3 -m pip install -r requirements.txt

python convert_hf_to_gguf.py   --outfile my-model-bf16.gguf   --outtype bf16   --remote <org>/<model>

./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M

convert_hf_to_gguf.py --remote pulls the weights from Hugging Face; point it at a local directory instead and drop the flag. If your model takes images or audio, the LLM part is not enough, you need a separate mmproj file for the multimodal encoder and projector.

For low-bit quants, an importance matrix is the difference between a usable 3-bit model and a broken one. llama-imatrix computes it from a calibration text file, and llama-quantize applies it:

./build/bin/llama-imatrix -m my-model-bf16.gguf -f calibration.txt -o imatrix.gguf -ngl 99
./build/bin/llama-quantize --imatrix imatrix.gguf my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M

There is also a Hugging Face Space (ggml-org/gguf-my-repo) that builds quants for you with no local setup, synced from llama.cpp main every six hours.

Step 3: serve it with tools enabled

llama-server --jinja -fa   -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M   -c 8192 -np 2   --host 0.0.0.0 --port 8080   --api-key "$LOCAL_KEY"   --metrics

What each flag is doing:

• --jinja turns on the Jinja chat template, which is what OpenAI-style function calling is built on. Current builds default it to enabled, but passing it explicitly costs nothing and saves you a debugging session on an older binary.
• -fa enables flash attention. The docs mark the default as auto, so being explicit is a choice about RAM versus speed, not a correctness requirement.
• -c 8192 -np 2 gives you two slots of 8K context. Every extra slot adds KV cache memory.
• --api-key matters more than it looks. Without it the server has no authentication at all, and 0.0.0.0 on a public interface means your GPU is open to whoever finds the port.
• --metrics exposes a Prometheus-compatible endpoint.
• --sleep-idle-seconds 300 unloads the model and its KV cache after five minutes of idle, and reloads on the next request. Useful on a laptop or a shared box.

Docker, if you prefer not to build:

docker run --gpus all -p 8080:8080 -v ~/models:/models   ghcr.io/ggml-org/llama.cpp:server-cuda13   -m /models/my-model-Q4_K_M.gguf -c 8192 -np 2 --host 0.0.0.0 --port 8080 --jinja -fa

The server image contains only llama-server, and the CUDA, Vulkan, ROCm, SYCL and OpenVINO variants are published under their own tags.

Before writing a single line of agent code, check that the server agrees with you about tools:

curl -s localhost:8080/health
curl -s localhost:8080/props | jq '.chat_template_tool_use'

If chat_template_tool_use is empty, the model's bundled template has no tool support, and your tool definitions will be ignored no matter how nicely you format them. The server logs Chat format: Generic in that case. Fix it by pointing at a tool_use template file from models/templates/ in the repo, with --chat-template-file. The docs also suggest --chat-template chatml as a rough fallback for models with no official tool template.

Step 4: verify tool calling with curl

Do this before the agent loop. It isolates server-side problems from client-side ones.

curl -s http://localhost:8080/v1/chat/completions   -H "Authorization: Bearer $LOCAL_KEY"   -H "Content-Type: application/json"   -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "system", "content": "You are a chatbot that uses tools. Do not overthink things."},
      {"role": "user", "content": "What is the weather in Istanbul?"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string", "description": "The city and country, e.g. Paris, France"}
          },
          "required": ["location"]
        }
      }
    }]
  }'

The answer you want has finish_reason: "tool" and a tool_calls array in the message. Anything else tells you something specific. If the arguments arrive as prose inside content, the template is falling back to the generic handler and you need a proper tool template. If the model answers the question directly and never calls the tool, add a line to the system prompt that says tools exist and when to use them. Small instruction-tuned models are often cautious about calling tools unless told.

Step 5: an agent loop that runs the tool

The client side is plain OpenAI SDK code with a different base URL. Nothing about the loop is llama.cpp-specific, which is the point of the OpenAI-compatible endpoint.

import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")

def get_current_weather(location: str) -> str:
    # Your real implementation goes here.
    return json.dumps({"location": location, "temp_c": 17})

TOOLS = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]
HANDLERS = {"get_current_weather": get_current_weather}

messages = [
    {"role": "system", "content": "You call tools when they can answer the question."},
    {"role": "user", "content": "What is the weather in Istanbul?"},
]

final = "no answer produced"
for step in range(6):
    resp = client.chat.completions.create(
        model="local",
        messages=messages,
        tools=TOOLS,
        tool_choice="auto",
        temperature=0,
    )
    msg = resp.choices[0].message
    messages.append(msg.model_dump(exclude_none=True))

    if not msg.tool_calls:
        final = msg.content or ""
        break

    for call in msg.tool_calls:
        args = json.loads(call.function.arguments or "{}")
        result = HANDLERS[call.function.name](**args)
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": result,
        })
else:
    final = "step limit reached without a final answer"

print(final)

Three things in that snippet are doing real work. The range(6) cap stops a model that keeps calling tools forever from burning your evening. model_dump(exclude_none=True) keeps the assistant message serializable for servers that are strict about unknown fields. The else clause on the for loop catches the step limit instead of silently printing a tool result as if it were the answer.

Two flags worth knowing about here. "parallel_tool_calls": true in the request payload enables multiple tool calls in one turn on models that support it, and it is off by default. If you run a reasoning model, --reasoning-format deepseek on the server puts the thinking text in message.reasoning_content instead of polluting message.content, and --reasoning-budget caps how many tokens the model is allowed to think.

Step 6: measure the quant, do not assume it

Tool calling is exactly the kind of task where a cheaper quant fails quietly. You will not get an exception, you will get a plausible wrong argument.

llama-bench answers the throughput question without a chat client in the way:

./build/bin/llama-bench -m my-model-Q4_K_M.gguf -p 512 -n 128 -r 5 -o md --progress
./build/bin/llama-bench -m my-model-Q4_K_M.gguf -ngl 99 -t 8 -p 512 -n 128

-p is prompt tokens, -n is generated tokens, -r is repetitions. Run the first command with each quant you are considering and keep the output in the repo. Run the second with different -ngl and -t values to find where offloading stops helping on your hardware.

For the correctness half, write 15 or 20 prompts where the only right answer is a tool call with exact arguments, then run that set against each candidate quant and each KV cache type. The upstream docs carry an explicit warning here: extreme KV quantizations such as -ctk q4_0 can substantially degrade a model's tool calling performance. That warning is about the cache, not the weights, and it is easy to hit while congratulating yourself on saving 300MB.

Built-in tools and MCP, if you want shortcuts

llama-server can act as the agent host itself. --tools all enables built-in tools (read_file, write_file, edit_file, grep_search, file_glob_search, exec_shell_command, get_info), or you pass a comma-separated subset. --tools-runtime docker:<image> runs those tools inside a container instead of on the host, and the same flag accepts podman: and ssh: targets.

MCP servers plug in through a Cursor-format JSON file:

{
  "mcpServers": {
    "example": { "command": "/path/to/server", "args": [] }
  }
}

llama-server -m my-model-Q4_K_M.gguf --mcp-servers-config mcp.json

Only the stdio transport is supported. Each server is spawned once at startup to list its tools, then stopped and respawned on demand, and its tools show up as <server>_<tool> in GET /tools. The timeout_ms key sets the per-call timeout, 30 seconds by default.

The security note in the docs deserves repeating: do not enable these in untrusted environments. The child process runs with the same privileges as the server, exec_shell_command is a shell by definition, and CORS restrictions only bind browsers. Any client that can reach the port can reach the tool. Keep --api-key set and bind to localhost or a private interface unless you actually meant to publish it.

When to use llama.cpp vs Ollama vs vLLM

Tool	What it is	Reach for it when
llama.cpp	The C/C++ inference engine and GGUF tooling. Runs on CPU plus CUDA, Metal, Vulkan, SYCL, ROCm and more.	You want flag-level control (KV cache type, batch size, offload, grammars, JSON schema) or you are deploying to a box without a datacenter GPU.
Ollama	A distribution and UX layer; its own README lists llama.cpp as a supported backend. Model library, ollama run, its own REST API with OpenAI compatibility.	Ease of use matters more than tuning. Fastest path from zero to a chat model on a laptop.
vLLM	A Python serving stack, originally from the Sky Computing Lab at Berkeley, built around PagedAttention and continuous batching.	Many concurrent requests on GPUs and throughput is the constraint. It speaks the OpenAI API and supports tool calling parsers and 200+ architectures.

The honest split is this. llama.cpp gives you the most control per byte of RAM and the widest hardware support. Ollama wraps it in something pleasant. vLLM is the one you pick when one GPU has to serve a crowd.

Next steps

Pin the build tag you tested. llama.cpp ships continuous nightly builds (tagged bNNNN, with b11046 current as of this writing) alongside versioned v0.* releases, and the argument surface moves. Record your quant, your KV cache type, your -c and -np, and your 20 tool-calling prompts in the same repo as the agent. When something regresses after an upgrade, that record turns a vague feeling into a diff. Then add --metrics to whatever already watches your services, so an idle model that never unloads and a slot count set too high stop being invisible.

References

• Function calling (llama.cpp docs)
• llama-server README and full flag list
• quantize README, including benchmark and size tables
• llama-bench README
• Docker images and usage (llama.cpp docs)
• Ollama README (supported backends)
• vLLM documentation