Tuning Fork

Tune agent skills to your own model.

See it work tells the story behind the figure.

Tuning Fork (tuningfork) sits between your agent and your LLM, allowing you to record and experiment on every call the agent makes. When a skill fails, tuningfork helps to find the step where the execution goes wrong, and shows how the model's responses cause the failure, down to the odds of each word in the model's reasoning. Then, tuningfork provides the framework to refine the skill via surgical edits, backed by statistical testing, so you can tune agent skills to your own model.

It works with Hermes Agent and Claude Code on llama.cpp, from a terminal, a web UI, or an agent over MCP. Any model that llama.cpp serves can be recorded and tested. Seeing the odds of each word works for Gemma 4, Qwen3.6, Qwen3.5, Qwen3-Coder, Qwen3 and Qwen2.5 so far, and the list will grow as the repo evolves (how to serve them).

Tuning Fork is pre-alpha: commands and their output may still change.

Why

The problem: You have a skill (agentskills.io-style SKILL.md) for your agent, but the skill is flaky: it works three times out of five. So you reword the skill, run the task again, and squint at the agent session history trying to see if the edits work. The process is repeated every time the agent fails to execute the skill properly, which then turns into what seems like a never-ending whack-a-mole. Even worse, when you change your LLM, most edits seem to lose their effect. This problem happens because:

• Wording sensitivity is real, and model-specific. Open models swing hard on meaning-preserving format changes, and the best format does not transfer between models (Sclar et al., ICLR 2024). Every model swap or quantization change means partial retuning.

• Luck and noise play a role. The edited skill passed after your edits, great, but are you sure that it's your edit and not just that the model generated a different token by luck and that token steered the execution to a success? The model could also have been getting lucky on other steps in previous runs so far, so it can fail in other ways during future runs despite your edits.

• You don't have an efficient tool. Squinting at the agent session history to figure out what went wrong is the same painful and inefficient approach as in the early days of software development, when people did desk checks and diagnostic printouts.

The missing tool: If we imagine the LLM as the CPU of a computer, a running agent as software, and skills as source code written in natural language, then tuningfork supplies you with the debugger you have been missing:

• Record every model call, with the exact context the model saw, or import a conversation your agent already had.

• Localize the first step where successful and failed runs diverge.

• Resample that step N times from a checkpoint, its history held fixed, and count what the model does.

• Look inside the step: the model's own odds at each token of its answer, and the word where its reasoning forks.

• A/B the wording there, and report the difference with a confidence interval and a significance test instead of a bare percentage.

• Break a live call before or after the model, and read or edit its whole context.

tuningfork does not just stop and edit LLM traffic; it provides a framework to give you clarity on how the model interprets agent skills, and make you confident that your edits are actually improving the skill in a way that is tuned to your model.

Related MCP server: log-probe-mcp

Quickstart

tuningfork runs on Linux, or on Windows inside WSL 2 (macOS support planned), with Python 3.12 or later.

It is not on PyPI yet: install it from GitHub with uv, which fetches Python itself if you need it.

Under WSL, keep the checkout on the Linux file system (~/…), never under /mnt/c, where SQLite's locking does not work; on Windows has the steps, and a way to keep the checkout on Windows instead.

Install Tuning Fork (tuningfork)

git clone https://github.com/hhuang91/tuningfork.git
cd tuningfork
uv sync
source .venv/bin/activate

The last line activates the environment, so that tuningfork and python are the environment's own.

Try it without a GPU (five-minute UI walkthrough)

python tests/browser/demo.py lab /tmp/tuningfork-lab 8790    # made-up runs, a scripted model

Open the web UI at the address it prints. The runs are made up, nine of the guide's example task (tally-report) with a resample and an A/B, but tuningfork's own proxy serves them, in front of a scripted model that answers at once, so every button works. To make the A/B in the GIF under See it work, open run tally-report/1#demo.2, its step 1, and A/B a wording…. The scripted model writes the keys the check wants only for a wording that names them in these exact words: row_count and amount_total (step by step). Part 1 of the guide tours runs like these from the terminal, then stops a live call at a breakpoint and changes it, still with no model.

On your own agent

You need a GPU that fits your model, llama.cpp's llama-server, and a harness: Hermes Agent or Claude Code (the versions we use). On Linux, install bubblewrap too (sudo apt install bubblewrap), so that every run sees the same folders: the model reads them (why it matters).

1. Start the model server, with the flags that make its numbers repeat (why). This is the guide's, Gemma 4 31B in about 24 GiB:

   llama-server -hf lmstudio-community/gemma-4-31B-it-GGUF:Q4_K_M --no-mmproj \
     --alias gemma-4-31b --host 127.0.0.1 --port 8081 \
     -ngl 999 -c 131072 -np 1 -ctk q4_0 -ctv q4_0 --jinja --ctx-checkpoints 0 --no-cache-prompt

2. Start tuningfork in front of it. It records to ~/.tuningfork and prints the web UI's address:

   tuningfork serve --upstream http://127.0.0.1:8081

   --upstream is the model server it forwards each call to. The harnesses tuningfork launches are pointed at tuningfork's own proxy for you. To record running agents as you use them, connect them by setting their base URL (where they send their model calls) to tuningfork's proxy: http://127.0.0.1:8787/v1 for agents that use an OpenAI-style API, such as Hermes Agent, or http://127.0.0.1:8787 for those that use Anthropic's, such as Claude Code. --port (or TUNINGFORK_PORT) moves the proxy to another port, and TUNINGFORK_PROXY then tells the other commands where it is. Keep it on 127.0.0.1, the default: anything that can reach the port can read your recordings and send calls to your model (what it guards). From another machine, forward the port over SSH instead: ssh -L 8787:127.0.0.1:8787 <host>.

3. Check the server, in a second terminal with the venv activated: can it do what the harnesses need, and does it process every prompt whole?

   tuningfork connections check

4. Record runs of a task — a prompt, its input files and a pass/fail check (the format):

   tuningfork batch examples/tally-report.evals.json -n 8 --harness hermes    # or --harness claude-code

   Or start from what your agent did: a conversation it already had imports as a run (tuningfork import, from a Hermes session file or export), and any run, imported or recorded as you used it, becomes a task of its own (tuningfork scenario <run> <name>).

5. Look at them in the web UI, at the address serve printed, or in a terminal: what you can do.

docs/setup.md has the rest: other servers, the harnesses, the sandbox, a pinned clock, and protecting a server that matters.

How it works

tuningfork is a recording proxy at the seam between the harness and the model server. It speaks both OpenAI chat completions and Anthropic Messages, so a harness that takes a custom base URL goes through it unchanged.

flowchart LR
    harness["Agent harness<br>Hermes Agent<br>Claude Code"] -- "OpenAI or<br>Anthropic API" --> proxy["tuningfork<br>recording proxy"]
    proxy --> server["Inference server<br>llama.cpp"]
    proxy --> store[("Recordings<br>each call's exact context")]
    store --> tools["CLI<br>web UI<br>MCP tools"]

Model requests are stateless: every request carries the model's whole context. So the proxy sees the full state of the program at every step without touching the harness's code, and any step can be sent again — as recorded, or with the skill's text swapped for an edit. A fork goes further: it relaunches the harness on the same task, answers its calls from the recording up to the step, and lets it run live from there to the end, where the task's check decides.

The work goes round one loop, from a skill to its next version, and the commands below follow it:

flowchart LR
    record["1 · Record<br>every call kept"] --> localize["2 · Localize<br>where runs go wrong"]
    localize --> measure["3 · Measure<br>resample · A/B"]
    measure --> improve["4 · Improve<br>screen · validate"]
    improve --> approve["5 · Approve<br>you decide"]
    approve -- "the skill's next version" --> record

What you can do

Grouped by the question each answers; each is run as tuningfork <command>. Every command takes --help, and docs/commands.md lists them all with their main options. The web UI does nearly all of it with a click, and an agent can do the reading, measuring and refining itself, over MCP (docs/agents.md).

Does my skill work?

What happened in a run?

Where do the runs go wrong?

Bad luck, or the wording?

A step is named by its number, or by what the model did there: write_file is the first step where it called that tool.

Which token decided it?

These need llama.cpp, and a model whose raw output tuningfork reads: Gemma 4, Qwen2.5, Qwen3, Qwen3.5, Qwen3.6 or Qwen3-Coder.

What is the model doing right now?

How do I fix the skill?

A refinement declares what the step should do before anything is proposed, screens each edit against the original, validates a survivor with full runs — held-out tasks included — and writes it into the skill only when you approve it.

Setting up, and keeping things tidy

The web UI

tuningfork serve serves a web UI at the address it prints, and tuningfork ui serves the same pages, read-only, over recordings with no proxy running. Everything above is in it, and it can start the work as well as show it: a batch of runs, a resample, an A/B, a fork, a refinement.

The page changes nothing without the proxy's access token, which is in the address serve prints; tuningfork token prints it again.

Let an agent drive it

Every operation is also a tool for an agent, over the Model Context Protocol (tuningfork mcp). Hermes Agent, Claude Code or OpenJarvis can count what the failed runs' checks said, find where a task's runs split, read that step, measure it, and screen edits of its own. No agent is offered anything that approves an edit, sends your recordings to another model, or shows it a held-out task's verdicts: those stay with you.

docs/agents.md sets it up for each client. examples/hermes-architecture-diagram/ is a case study: Hermes improving one of its own skills with tuningfork's tools, beside Hermes's own self-improvement loop, on the same local model.

See it work

The toy skill in examples/find-the-file/ summarizes a CSV into report.json, and says that if the file is not there, the agent should say so and stop. The request names data.csv, but the file is in a subfolder. Ten runs through Hermes Agent, on Gemma 4 31B:

$ tuningfork batch examples/find-the-file/evals.json -n 10 --harness hermes --model gemma-4-31b
...
4/10  40%  [17%-69%] passed

Every run first read data.csv where the request names it, and found nothing. Then it either searched for the file and passed, or gave up and failed. Bad luck, or the wording? Sent to the model twenty more times with its history held fixed, that step gives up 16 times. Scored, its answer shows where the reasoning forks. After "…but it was not found.", the next word is "According" at 80%, which goes on "…I should say so and stop", or "Wait" at 7%, which goes on "…search for it first?". The model wavers there, so one line of the skill can tip it. Tested at that step, toward a target declared before the first sample:

$ tuningfork ab <run> 1 --variant examples/find-the-file/fix/SKILL.md \
    --target-rules '[{"if": "calls", "value": "search_files"}]'
...
target (declared before sampling): a behaviour that calls search_files
  original  4/20  20% [8%-42%]
  variant   20/20 100% [84%-100%]
  Fisher exact p = 1.5e-07 → significant at α=0.05, variant higher

Full runs agree: with the edit in every run, 10/10 pass [72%-100%], against 4/10 before (Fisher p = 0.011). Every rate comes with its 95% interval, and an A/B without a declared target is labelled exploratory and corrected for every comparison it makes. The example shows each command with its output.

In the web UI, an A/B is a few clicks from a run's step (here on the demo's scripted model, which answers at once; the quickstart starts it):

Guides

• Tuning Fork, step by step — every part of the tool, on a toy skill made to fail one way: a tour with no GPU, the real experiment with Claude Code and Hermes Agent, a look inside a step, and a fix proposed, screened, validated and approved.

• A step where the model wavers — the runs behind the figure at the top: where the runs part ways, how often the model goes each way, where its reasoning forks, and the line that tips it.

• A real skill, fixed at the step that fails — Hermes Agent's own design-md skill, from the failing step to an approved edit and the runs after it.

• The same model, two ways to improve a skill — Hermes's own self-improvement loop against Hermes with tuningfork's tools.

• Reference: setup · commands · agents over MCP · writing a harness adapter · tuningfork glossary for the words.

Works with

What is particular to a harness or a server lives behind an adapter, never in the core: docs/adapters/TEMPLATE.md is where a new harness starts, and tuningfork probe <url> writes a profile of a server that you can contribute back.

Contributing

The test suite runs against a scriptable mock server, with no GPU and no model: uv run pytest. CONTRIBUTING.md says how to set up and what review looks for; SECURITY.md says how to report a vulnerability, and what tuningfork does and does not guard. What changed in each version is in CHANGELOG.md.

License

Apache-2.0: see LICENSE, and NOTICE for the work it follows.