Tool calling and structured JSON with llama.cpp on a Radeon GPU

What a tool call is

A tool call is an OpenAI-shaped /v1/chat/completions request that includes a tools array. The model returns tool_calls instead of (or before) a normal message. llama.cpp parses that when --jinja is on and the chat template understands tools. Function-calling.md: native handlers exist for several families (Llama 3.1 / 3.3, GPT-OSS Harmony, Hermes 2/3, Functionary, Mistral Nemo, Command R7B, DeepSeek R1). Unrecognized templates fall back to Generic — logs say Chat format: Generic. Generic may use more tokens. Parallel / multiple tool calls stay off unless the payload sets "parallel_tool_calls": true, and only some templates support them.

Official path: llama.cpp function-calling.md · coding assistant · harnesses.

--jinja and templates

function-calling.md: start llama-server with --jinja. The server README (checked 2026-10-14) lists --jinja / --no-jinja and says the default is enabled. The house pin still passes --jinja so a rebuilt binary cannot silently drop it. The template comes from the GGUF metadata unless you override it. --chat-template-file loads a Jinja file. --chat-template names a built-in (chatml is the documented “many models, YMMV” fallback when there is no official tool_use template). Inspect GET /props for chat_template and chat_template_tool_use. --skip-chat-parsing dumps reasoning and tool markup into content instead of parsing them.

Official Qwen3.6-35B-A3B card documents tool use (vLLM / sglang parsers on that card). llama.cpp function-calling.md does not name this 3.6 file in the native-handler table. After launch, read the server log for Chat format and GET http://127.0.0.1:8080/props. Do not invent a handler name or a --chat-template-file path for the house GGUF. Llama 3.3 70B is listed as Llama 3.x native. GPT-OSS is listed as Harmony. Gemma 3 27B and GLM 5.3 Flash are not named on that table — treat them as not documented there.

• --jinja / --no-jinja

  Use the Jinja chat-template engine. function-calling.md requires --jinja for OpenAI-style tools. Server README default is enabled (checked 2026-10-14). House pin still passes --jinja.

  llama.cpp function-calling.md; server README — --jinja (checked 2026-10-14)

• --chat-template / --chat-template-file

  Override the GGUF metadata template. Built-in names include chatml (documented YMMV fallback). A file path loads a custom Jinja. Inspect /props for chat_template and chat_template_tool_use before you invent a filename.

  llama.cpp server README — --chat-template, --chat-template-file; function-calling.md

• --skip-chat-parsing

  Force a pure content parser. Reasoning and tool markup stay in message.content instead of being split out. Useful when think-tags leak into a harness that expects tool_calls.

  llama.cpp server README — --skip-chat-parsing

• parallel_tool_calls (request body)

  Multiple / parallel tool calls. Official default is off. Enable with "parallel_tool_calls": true. Only some templates support it; verification is based on the Jinja template.

  llama.cpp function-calling.md; server README — parallel_tool_calls

• response_format / json_schema / grammar

  Constrain the whole reply to JSON or a GBNF grammar. Server README: {"type":"json_object"} or a schema object. -j / --json-schema and --grammar-file on the CLI. Not a substitute for a tools array.

  llama.cpp server README — response_format, json_schema, grammar; grammars/README.md

Structured JSON

Tool calling injects the tool schema into the prompt. Constrained decoding is a different knob: it forces the next tokens to match a grammar. Official grammars/README.md: JSON Schema converts to GBNF via json-schema-to-grammar; the schema is not injected except during tool calling, so describe a free-form JSON shape in the prompt. Use response_format {"type":"json_object"} or a schema-constrained object when the whole reply must be JSON (calculator output, recipe export) and you are not in a tool loop. Use --grammar / --grammar-file when the output is not JSON. Use tools + --jinja when the harness expects tool_calls. A grammar can stall — llama.cpp issue #4218. Do not invent an overhead tok/s; measure ${TC_PACK}.

Official: server README · grammars/README.md · GBNF · JSON schema. Measure pack grammar-ab/1 on Stack Lab — do not invent an overhead tok/s.

House launch

the house default is Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL, llama.cpp Vulkan, one R9700, 262144 context, f16 KV. Add --jinja for tools. House /v1 is http://127.0.0.1:8080/v1 with model id qwen — same pin as /setup/coding-assistant. function-calling.md cautions that extreme KV quants (example: q4_0 KV) can degrade tool calling. House KV stays f16. A tools payload and multi-turn history eat context — size -c on /long-context.

House Hermes on the local desktop also talks to Strata on :8083 for the Flash fast-agent path. llama.cpp function-calling.md does not document Strata. Treat Strata tool calls as not verified here. Attach notes stay on /harnesses/hermes and /backends/strata.

Daily-driver recipe: qwen36-35b.llamacpp-vulkan.262k · Coding clients: /setup/coding-assistant · Hermes: /harnesses/hermes · Strata (not verified for tools): /backends/strata.

# House daily-driver shape with --jinja. Not a measured tool-call pin.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  --jinja \
  -ngl 99 \
  -c 262144 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

# Shape only. Official tools payload from docs/function-calling.md,
# pointed at the house /v1 (model id qwen). Not a measured success rate.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role":"user","content":"What is the weather in Istanbul?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type":"string","description":"City and country/state"}
          },
          "required": ["location"]
        }
      }
    }]
  }'

# Shape only. response_format from the llama.cpp server README.
# Describe the fields in the prompt — the schema is not injected.
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role":"user","content":"Return one JSON object with a name string."}],
    "response_format": {"type":"json_object"}
  }'

Shape only. Official tools payload from function-calling.md, pointed at house qwen on 127.0.0.1:8080. Not a measured success rate.

Troubleshooting

• Tools ignored

  Confirm --jinja, then GET /props for a tools-aware chat_template / chat_template_tool_use. Read the log for Chat format: Generic versus a native handler. A missing or default-only template often means the model writes prose. Do not invent a --chat-template-file for the house 35B — start from the GGUF metadata.

• Malformed JSON

  Tool arguments are model text unless you also constrain them. For a whole-reply JSON object, use response_format or --json-schema. For a custom shape, use GBNF. Measure parse success on Stack Lab grammar-ab/1. Do not copy a free-form 35B tok/s as a grammar figure.

• Think-tags leaking into tool calls

  Reasoning tokens can land in content when the parser and the template disagree. Official server README: --skip-chat-parsing keeps everything in content; reasoning_effort none disables thinking when the template honors it. The official Qwen3.6 card documents enable_thinking on other engines — that is their cite, not a llama.cpp flag we invent.

• Template mismatch

  function-calling.md: override with --chat-template-file, or try --chat-template chatml when there is no official tool_use template (YMMV). Wrong template is the usual “tools ignored” or “markup in content” cause. Check /props after every override.

• Long-context drift

  Tool schemas and multi-turn tool results sit in the KV cache. Official UD-Q5 1-GPU llama.cpp decode already drops from 102.39 tok/s at 4K to 44.4 at 256K — same file, no tools. Size -c on /long-context. Extreme KV quants can degrade tool calling (function-calling.md). House KV stays f16.

Measured 35B — no tools

Official Qwen3.6-35B-A3B llama.cpp Vulkan cells from existing run cards. These launches have no tools array. Tool-call and grammar cells stay unmeasured. UD-Q3_K_XL is a labeled estimate. Full matrix: /benchmarks.

VRAM fit

Calculator estimate for Qwen3.6-35B-A3B on llama.cpp, f16 KV. A tools payload does not change weights. UD-Q3_K_XL is a labeled estimate. Do not read leftover as a tool-call speed cell.

One R9700 · 32K

One R9700 · 262K

KV grows. Tool schemas sit in that window. Size -c on /long-context.

System RAM

A tools array does not add GGUF weight; it does add prompt tokens. When -ngl leaves layers on the host, shopper lines stay Minimum system RAM needed (full-system estimate rounded to a real kit, with a breakdown — labeled estimate until a card overlays it) and RAM needed at full context (cache included when measured llama.cpp peaks exist). House 35B UD-Q5_K_XL on one 32 GB R9700 does not offload. The 16 GB example is house 35B Q4_K_M on an RX 7800 XT at 32K selected — estimate unless a measured card overlays the kit.

FAQ

• Do I need --jinja for tool calls?

  Yes for the documented OpenAI-style path. function-calling.md starts llama-server with --jinja. The server README default is enabled (checked 2026-10-14); the house pin still passes the flag. Then send a tools array on /v1/chat/completions.

  Coding assistant · Harnesses · llama.cpp

• Tools array or JSON Schema — which one?

  Use tools + --jinja when the harness expects tool_calls (Hermes, Cline, Continue). Use response_format / --json-schema / GBNF when the whole reply must be one JSON object and you are not in a tool loop. The schema is not injected except during tool calling. Grammar cost stays not measured yet — run Stack Lab grammar-ab/1.

  GBNF · Stack Lab · amdai-bench

• Can a Q3 file call tools?

  UD-Q3_K_XL may appear as a labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). House files stay Q6 / Q5 / Q4. A template flag does not change that. Q1/Q2 stay off.

  Quants · Q6 / Q5 / Q4 + Q3 estimate · Calculator

Sources

• llama.cpp function-calling.md
• llama.cpp server README
• llama.cpp GBNF guide
• llama.cpp json-schema-to-grammar.cpp
• Qwen3.6-35B-A3B model card
• Llama 3.3 70B model card
• gpt-oss-20b model card
• llama.cpp repository
• AMDai — coding assistant
• AMDai — harnesses
• AMDai — Open WebUI
• AMDai — backends
• AMDai — llama.cpp
• AMDai — Strata
• AMDai — Hermes
• AMDai — speed up llama.cpp
• AMDai — long context
• AMDai — Stack Lab
• AMDai — amdai-bench