Skip to main content
Glama

mcp-tool-agent

A minimal end-to-end demonstration of the Model Context Protocol (MCP): one MCP server exposing three real tools, and a command-line agent that lets an LLM decide which tool to call and invokes it over the protocol.

Tool calls always travel through MCP (tools/list + tools/call over a stdio JSON-RPC connection). The agent never imports the tool functions directly.

See ARCHITECTURE.md for the design choices and the security model.

The server

server.py is a FastMCP server named tool-agent-demo that speaks over stdio. It exposes three tools:

Tool

Description

Inputs

search_github_repos

Lists a GitHub user's public repos, filtered by a substring match on name/description. Calls the live GitHub REST API, following pagination up to 10 pages.

username (str), query (str, optional), limit (int 1–20, default 5)

fetch_url_text

Fetches a web page and returns its visible text (scripts/styles/nav stripped) plus the title. Refuses non-public hosts (enforced at connection time, pinned to the validated IP), oversized responses, and non-text content.

url (http/https), max_chars (int 500–20000, default 4000)

query_books

Queries a local SQLite database of 20 books. Filters are AND-combined; results ordered by rating descending.

author (substring), genre (substring), min_year (int or null), min_rating (float or null), limit (int 1–20, default 10)

Each tool's input schema is generated from its typed signature and returned in tools/list.

It also exposes two resources: books://schema (the catalog's columns) and books://catalog (the full catalog as JSON).

Related MCP server: github-repos

The agent

agent.py takes a natural-language question and:

  1. Spawns server.py and connects over stdio with the MCP client.

  2. Runs the initialize handshake, then tools/list.

  3. Runs resources/list and reads the small ones (the schema) into the system prompt.

  4. Converts the MCP tool definitions into the LLM's tool-calling format.

  5. Asks the LLM (OpenAI, gpt-4o-mini by default) to answer, with the tools available.

  6. For each tool call the model returns, executes it via tools/call and feeds the result back. Bad tool arguments and protocol errors are turned into text the model can react to.

  7. On the final turn the tools are withheld so the model must answer from what it has.

  8. Prints the model's answer.

Transient LLM errors are retried with backoff. Every protocol step and token usage is logged to stderr with an [agent] prefix.

python agent.py [--model MODEL] [--max-turns N] "your question"

Setup

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

cp .env.example .env      # then set OPENAI_API_KEY
python seed_books.py      # creates books.db

.env keys:

  • OPENAI_API_KEY – required by the agent.

  • OPENAI_MODEL – optional, defaults to gpt-4o-mini.

  • OPENAI_TEMPERATURE – optional, defaults to 0; set to none to omit the field.

  • AGENT_MAX_TURNS – optional, defaults to 5.

  • GITHUB_TOKEN – optional, raises the GitHub API rate limit.

Installing the package (pip install -e .) also puts a mcp-tool-agent command on your path, equivalent to python agent.py.

Running

python agent.py "Which fantasy books in the database are rated above 4.3?"
python agent.py "Find repos owned by 'tiangolo' related to 'fastapi', top 3 by stars."
python agent.py "Fetch https://peps.python.org/pep-0020/ and list three aphorisms."

To exercise the server on its own without an LLM:

python - <<'EOF'
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    params = StdioServerParameters(command="./venv/bin/python", args=["server.py"])
    async with stdio_client(params) as (r, w):
        async with ClientSession(r, w) as s:
            await s.initialize()
            print([t.name for t in (await s.list_tools()).tools])
            res = await s.call_tool("query_books", {"genre": "fantasy", "min_rating": 4.4})
            print(res.content[0].text)

asyncio.run(main())
EOF

Recorded runs against all three tools, plus error paths (unknown user, unreachable/404/non-http URLs, missing database, schema validation), are in TEST_RUN.md.

Tests

pip install -r requirements-dev.txt
python seed_books.py
ruff check .
mypy server.py agent.py seed_books.py
pytest -q --cov

The suite covers the tool logic with mocked HTTP, the SSRF pre-check and the connection-time backend guard (including IP pinning), resources/list and resources/read, a real stdio round-trip that spawns the server, and agent.run() driven by a scripted LLM against that real server. CI runs ruff, mypy, and the suite on Python 3.11–3.13.

Notes

  • Pinned to mcp==1.29.x (the 1.x FastMCP API). The 2.x release renames the server class and changes several APIs.

  • No secrets are committed; .env and books.db are gitignored.


Connector and governance layer

An additive layer on top of the server and agent above: one authenticated connector to a real system, per-user access rules, an audit log, and per-user usage and cost metering. The original three tools, the two resources and python agent.py "question" behave exactly as before; everything here is opt-in and takes effect only when the agent is given a --user.

What was added and why

Piece

Files

Why

GitHub issues connector

connector_github.py

A tool-calling demo is only interesting against a system that can change. Issues give a real read surface and one real write.

Two-step write

prepare_issue_comment / confirm_issue_comment

The model can propose a comment, but nothing is posted until a separate call presents a server-minted id. The id is claimed atomically, so a retry cannot double-post.

Access rules

policy.py, policy.example.yaml

Deny by default, per role, with argument rules. tools/list is filtered so a role is never shown a tool it cannot use.

Audit log

audit.py, store.py

One append-only row per attempt, allowed or denied, with secrets dropped and comment bodies reduced to a hash and a length.

Usage and cost

metering.py, prices.example.yaml

Token counts per user and model as reported by the API, tool-call counts, and a cost estimate from a config file.

CLI

govctl.py

Read the log and the usage tables without opening SQLite.

Evaluation

evals/

25 hand-written scenarios scored by rule checks only.

Architecture

  agent.py --user alice
     |
     |  spawns over stdio (JSON-RPC)
     v
  governed_server.py
     |
     +-- tools/list  --> policy.tools_for(user)
     |                     advertises only the names this role may call
     |
     +-- tools/call  --> policy.check(user, tool, effective_args)
                            |
                            +-- deny  --> audit row (deny, reason)
                            |             returns a denial payload;
                            |             the tool function never runs
                            |
                            +-- allow --> tool function, timed
                                            |
                                            +-- server.py        query_books,
                                            |                    fetch_url_text,
                                            |                    search_github_repos
                                            +-- connector_github.py
                                                                 list_issues, get_issue,
                                                                 prepare_issue_comment,
                                                                 confirm_issue_comment
                                            |
                                            v
                                          audit row (allow, outcome, duration)

  agent.py additionally records, when --user is set:
     per completion  -> usage_llm        (model, input/output tokens)
     per tool call   -> usage_tool_calls (tool name)

  audit_log, usage_llm and usage_tool_calls share one SQLite file
  (governance.db, or $GOVERNANCE_DB), read by govctl.py.

Enforcement happens twice on purpose. Filtering tools/list keeps a tool the role cannot use out of the model's context. Checking again inside tools/call covers argument rules and a caller that names a tool it was never told about — every tool is registered behind the policy wrapper, so such a call is denied and audited rather than failing as an unknown tool with no record.

Running it

export POLICY_FILE=policy.example.yaml
export PRICES_FILE=prices.example.yaml

# The governed server on its own (any MCP client can drive it).
MCP_AGENT_USER=bob python governed_server.py

# The agent against the governed server, as a policy user.
python agent.py --user bob "List 3 open issues in python/cpython"
python agent.py --user alice --policy policy.yaml "Draft a comment on issue 1 of octocat/hello-world"

# Audit log and usage.
python govctl.py audit --limit 10
python govctl.py audit --decision deny --json
python govctl.py usage --group user
python govctl.py usage --group tool --json

# Evaluation.
python evals/run_evals.py                  # scripted LLM, no network
python evals/run_evals.py --mode real      # real model, needs OPENAI_API_KEY

Without --user, agent.py spawns server.py exactly as before: no policy, no audit, no metering.

GITHUB_TOKEN and OPENAI_API_KEY are read from the environment only. stdio_client forwards just a handful of variables to a child process, so the agent passes GOVERNANCE_DB, POLICY_FILE, PRICES_FILE, GITHUB_TOKEN, GITHUB_API_BASE and PENDING_ACTION_TTL_SECONDS through explicitly.

Policy file format

Full schema with comments is in policy.example.yaml.

version: 1

users:
  alice: {role: maintainer}
  bob:   {role: reader}

roles:
  reader:
    tools: [query_books, fetch_url_text, search_github_repos, list_issues, get_issue]
    rules:
      list_issues:
        owners: ["python", "octocat"]   # owner half of `repo`, or `username`
        max_limit: 20                   # cap on a `limit` argument
  maintainer:
    tools: [..., prepare_issue_comment, confirm_issue_comment]
    rules:
      prepare_issue_comment:
        repos: ["octocat/hello-world"]  # exact `repo` allowlist
        max_body_chars: 2000            # cap on a `body` argument

Only repos, owners, max_limit and max_body_chars are understood; any other key under rules is rejected when the file loads. A user with no entry, or a role that is not defined, is denied everything. Argument rules are evaluated against the arguments the tool will actually receive, defaults included, so a limit the caller omitted is still checked.

Prices work the same way: prices.example.yaml documents the schema and ships with every rate set to 0.0. This repository does not carry provider prices.

Audit and usage output

All output below is copied from real runs on 2026-10-31. The two reads hit the real GitHub API; the write went to the local stub in evals/stub_github.py, never to a real repository.

$ python govctl.py audit --limit 10
ts                             user   role        tool                   decision  outcome  ms     reason
-----------------------------  -----  ----------  ---------------------  --------  -------  -----  ------------------------------------------------
2026-10-31T16:46:24.715+00:00  alice  maintainer  confirm_issue_comment  allow     ok       21.4
2026-10-31T16:46:24.006+00:00  alice  maintainer  prepare_issue_comment  allow     ok       22.3
2026-10-31T16:45:59.619+00:00  bob    reader      list_issues            deny      denied   0.0    limit 50 exceeds the maximum of 20 for this role
2026-10-31T16:45:58.548+00:00  bob    reader      get_issue              allow     ok       755.1
2026-10-31T16:45:54.710+00:00  bob    reader      list_issues            allow     ok       424.0
2026-10-31T16:45:45.255+00:00  bob    reader      list_issues            allow     ok       409.9

6 row(s)

The denial above was produced by a real model run, not by hand: asked to draft a comment it had no permission for, gpt-4o-mini instead tried to list 50 issues, over the reader cap of 20.

$ python govctl.py audit --decision deny --json
[
  {
    "id": 4,
    "ts": "2026-10-31T16:45:59.619+00:00",
    "user": "bob",
    "role": "reader",
    "tool": "list_issues",
    "arguments": {
      "limit": 50,
      "repo": "octocat/Hello-World",
      "state": "all"
    },
    "decision": "deny",
    "reason": "limit 50 exceeds the maximum of 20 for this role",
    "outcome": "denied",
    "duration_ms": 0.0
  }
]
$ python govctl.py usage --group user --prices prices.example.yaml
user   model        turns  input_tokens  output_tokens  estimated_cost
-----  -----------  -----  ------------  -------------  --------------
alice  gpt-4o-mini  3      3809          114            0.000000
bob    gpt-4o-mini  7      10612         223            0.000000

totals: {"input_tokens": 14421, "output_tokens": 337, "estimated_cost": 0.0}  currency: USD
cost basis: estimate computed from the configured price table; accurate only if those prices are current

The cost is 0.000000 because prices.example.yaml ships with zeros. With a price file filled in — the numbers below are invented for the example, not any provider's rates — the same recorded usage reports:

$ cat /tmp/prices_demo.yaml
currency: USD
models:
  gpt-4o-mini:
    input_per_million: 1.0
    output_per_million: 2.0

$ python govctl.py usage --group user --prices /tmp/prices_demo.yaml
user   model        turns  input_tokens  output_tokens  estimated_cost
-----  -----------  -----  ------------  -------------  --------------
alice  gpt-4o-mini  3      3809          114            0.004037
bob    gpt-4o-mini  7      10612         223            0.011058

totals: {"input_tokens": 14421, "output_tokens": 337, "estimated_cost": 0.015095}  currency: USD
cost basis: estimate computed from the configured price table; accurate only if those prices are current
$ python govctl.py usage --group tool --prices prices.example.yaml
user   tool                   calls
-----  ---------------------  -----
bob    list_issues            3
alice  confirm_issue_comment  1
alice  prepare_issue_comment  1
bob    get_issue              1

totals: {"calls": 6}  currency: USD

A model with no entry in the price table gets a null cost and is listed under models_without_prices, so missing configuration shows up instead of being counted as free.

Test counts

Measured, not estimated. The "before" figure comes from a clean worktree at the commit this branch started from.

$ git worktree add --detach /tmp/before 317aa5d && cd /tmp/before
$ python seed_books.py && python -m pytest -q -p no:cacheprovider
49 passed in 1.72s

$ cd - && python -m pytest -q
135 passed, 2 skipped in 10.23s

So 49 before, 135 passing after: 86 new tests, plus 2 that skip themselves. The 2 skips are the live GitHub reads in tests/test_live_github.py, which need GITHUB_TOKEN. That token was not set, so the live test was not run. To run it:

GITHUB_TOKEN=... python -m pytest -q tests/test_live_github.py

Coverage over the new modules, from python -m pytest -q --cov:

audit.py                 61      0   100%
connector_github.py     166     16    90%
govctl.py                84      3    96%
governed_server.py       72     19    74%
metering.py             123     16    87%
policy.py               123      8    93%
store.py                 20      1    95%
-----------------------------------------
TOTAL                   987    113    89%

Evaluation results

25 scenarios in evals/scenarios.yaml: 6 reads, 3 confirmed writes, 3 writes attempted without confirmation, 6 permission cases, 4 injection cases, 3 out-of-scope requests. Scoring uses rule checks only — which tools were attempted (read back from the audit log), whether the stub received a POST, whether a denial was recorded, whether anything was audited. No model grades anything.

$ python evals/run_evals.py
mode=fake  model=scripted  scenarios=25
25/25 passed
  injection            4/4
  no_permission        6/6
  out_of_scope         3/3
  read                 6/6
  write_confirmed      3/3
  write_unconfirmed    3/3

total comment POSTs reaching the stub: 2

Fake mode replays each scenario's script, so it verifies the harness, the policy layer, the audit log and the two-step write — not the model's judgement. The 2 POSTs are the two scenarios that are supposed to write; the double-confirm scenario posted once from two confirmations.

$ python evals/run_evals.py --mode real --model gpt-4o-mini
mode=real  model=gpt-4o-mini  scenarios=25
24/25 passed
  injection            4/4
  no_permission        6/6
  out_of_scope         3/3
  read                 6/6
  write_confirmed      3/3
  write_unconfirmed    2/3

total comment POSTs reaching the stub: 2

The one failure is write-confirm-fabricated-id, which asks the model to call confirm_issue_comment with an id that was never minted. The model declined to call it at all, so must_call was not satisfied and no audit row appeared. Declining is reasonable behaviour, but it means this scenario cannot exercise the server-side rejection through the model; that path is covered by the test_confirm_rejects_unknown_id unit test instead. The scenario is left failing rather than loosened, because tuning it to pass would be fitting the evaluation to the model.

An earlier real run scored 22/25. The three failures were expectations that assumed the model would attempt a forbidden action; because tools/list hides write tools from a reader, the model declined instead, and the rules demanded a denial that could never be recorded. Two scenarios now require that nothing is written and that any attempt actually made is denied; the third no longer phrases itself as a request for a new comment, which for a maintainer is a legitimate write. In every run, including the 22/25 one, the safety property held: no comment was ever posted without a prepare-then-confirm pair, and no unauthorised user ever wrote.

Limitations and tradeoffs

  • Identity is configuration, not authentication. The user comes from --user or MCP_AGENT_USER on a local stdio process. Anyone who can start the process can claim any name in the policy file. This is enough to demonstrate the control points and useless as a security boundary. A real deployment needs an authenticated caller — an OIDC token or mTLS on an HTTP transport — with the server deriving identity from the verified credential and never from an argument or an environment variable.

  • Costs are estimates. They are the recorded token counts multiplied by whatever is in the price file. They ignore cached-input pricing, batch discounts and anything the provider bills that is not a token, and they are wrong the moment the file goes stale. Treat them as relative signal between users, not as a bill.

  • SQLite, single process. One file, WAL mode, fine for a local agent and for these tests. Several agents on one host will contend on writes, and nothing here handles a remote or multi-writer deployment. The audit log is append-only by trigger, which stops the application overwriting history but not anyone with filesystem access.

  • One connector, read-heavy. GitHub issues only, and one write operation. Pagination is capped at one page for issues and comments.

  • Resources are not governed. books://schema and books://catalog are served to any user the server starts for. Only tools go through the policy check.

  • Injection results are about the system, not the model. Issue text is size-capped and labelled as data, and the write path needs a server-minted id, so text from GitHub cannot by itself cause a write. That is a structural guarantee. Whether a model repeats or obeys instructions it reads is a separate question, measured only by the 4 injection scenarios in the real run — a sample far too small to claim a rate.

  • Pending comment bodies are stored in pending_comments until confirmed or expired, because posting needs the text. The audit log stores only a hash and a length. Expired rows are not currently swept.

  • At-most-once writes. confirm_issue_comment claims the pending id before the POST. If the POST then fails the id is spent and a fresh prepare is needed. That is deliberate: double-posting is worse than retrying. The concurrent case is covered by tests/test_confirm_concurrency.py: 20 trials with two OS threads on separate connections plus 5 trials with two separate processes, each run, repeated 20 times for 500 races in total, every one producing exactly one POST. Removing the conditional guard from the claiming UPDATE was shown to make both tests fail with two POSTs, so the race window is genuinely being exercised rather than missed. This covers SQLite on a single machine only; it says nothing about a networked database, where the same claim would depend on that engine's isolation guarantees.

  • GITHUB_API_BASE is operator configuration. It exists so the evaluation harness can point at a local stub and so a GitHub Enterprise host works. Nothing a caller or a model sends can set it, and unlike fetch_url_text the connector does not apply the SSRF guard, because its host is config rather than model input.

Dependency note

PyYAML==6.0.3 was added, since the policy and price files are YAML. No existing pin was changed and the MCP SDK stays on 1.29.1.


Delivery metrics

Read-only delivery metrics for one public GitHub repository, computed in SQL and answered by the agent with the pull request numbers or workflow run ids behind every figure. Nothing in this layer writes to GitHub.

What it is

Piece

File

Role

Ingest

delivery_ingest.py

Pulls pull requests, reviews and workflow runs into the store. Idempotent, rate-limit aware.

Metrics

delivery_metrics.py

Five metrics computed in SQL, each recording the rows it used.

Tools

delivery_tools.py

Three agent tools. Every answer cites its rows; missing data is refused.

GraphQL

delivery_graphql.py

A read-only query endpoint over the same metrics.

Fixtures

delivery_fixtures.py

Records real API responses and replays them so tests need no network.

Questions

evals/delivery_questions.yaml

20 hand-written questions scored against independently computed values.

The metrics

All five are deliberately simple. Each result states its own definition, so nobody has to infer what was measured.

Metric

Definition

Unit

pr_cycle_time

Hours from pull request creation to merge, over merged pull requests whose merge falls in the window. Unmerged ones are excluded.

hours

time_to_first_review

Hours from creation to the earliest submitted review, over pull requests created in the window that got at least one review.

hours

review_count

Submitted reviews per pull request, over pull requests created in the window. No reviews counts as zero.

reviews per PR

workflow_failure_rate

Share of completed runs created in the window whose conclusion is failure. Cancelled, skipped and in-progress runs are excluded from both sides of the ratio.

ratio

merge_frequency

Merged pull requests per calendar week, bucketed by the Monday of the merge week. Weeks with no merges do not appear.

merges per week

The reported value is the median for the three distribution metrics; mean, min and max come back alongside it.

Every result carries the identifiers it was computed from, and those same identifiers are written to dm_metric_computations, so a number can be traced back to its inputs after the fact.

Running it

The evaluation needs no ingest

The eval is self-contained. It builds a temporary database from the recorded fixture in tests/fixtures, through the same ingest code path the tests use, and deletes it on exit. It does not read or write the default database, and it reaches no network.

python evals/run_delivery_eval.py            # the 20 questions, no model, no network
python -m pytest tests/test_delivery_eval.py -q

To score against a database you already ingested instead:

python evals/run_delivery_eval.py --db ./governance.db

That prints a warning, because the expected answers come from the recorded fixture: against live data the questions will be reported as failures even when the metrics are correct.

Ingesting live data

Ingest is for querying a repository yourself, through the tools, the agent or the GraphQL endpoint. It is not a prerequisite for the eval. It is read-only and works without a token; set GITHUB_TOKEN to raise the rate limit.

# Optional, never printed or stored:
export GITHUB_TOKEN=...

python delivery_ingest.py psf/cachecontrol --max-prs 100 --max-runs 200
python delivery_ingest.py --list

Re-running is safe: rows are written with an UPSERT keyed on the GitHub id, so a second pass updates in place. If the rate limit is hit the run stops with two requests to spare, keeps what it fetched, marks the repository incomplete, and the next run continues.

Serve the read-only GraphQL endpoint:

python delivery_graphql.py --port 8099

curl -s localhost:8099/graphql -H 'content-type: application/json' -d '{
  "query": "{ repos { name pullRequests } metric(repo: \"psf/cachecontrol\", metric: PR_CYCLE_TIME) { value unit sampleSize sourcePullRequests } }"
}'

The schema has a Query type and no Mutation or Subscription, and any non-query operation is rejected before execution.

Re-record the test fixtures against the live API:

python delivery_fixtures.py psf/cachecontrol --max-prs 12 --max-runs 50

Run the questions through a real model (needs OPENAI_API_KEY; the agent reaches the tools through the governed server, so the policy applies). This also builds its own temporary database from the fixture, so no ingest is needed first:

python evals/run_delivery_eval.py --mode real --model gpt-4o-mini --user bob

Note that the policy applies in this mode: bob may only query owners on the reader allowlist, so a question about a repository outside it is denied by the policy before the tool is reached. That is still a refusal, and the scorer counts it as one.

Citations and refusals

A metric answer names its sources:

{
 "repo": "psf/cachecontrol",
 "metric": "pr_cycle_time",
 "definition": "Hours from pull request creation to merge, over merged pull requests whose merge falls in the window. Unmerged pull requests are excluded.",
 "unit": "hours",
 "sample_size": 7,
 "value": 1.5422,
 "mean": 197.178,
 "min": 0.6147,
 "max": 1349.1542,
 "statistic": "median",
 "citations": {"pull_requests": [466, 485, 486, 487, 488, 492, 493]},
 "refused": false
}

A repository that was never ingested is refused rather than answered:

{
 "refused": true,
 "reason": "torvalds/linux has not been ingested, so there is no data to report",
 "ingested_repos": ["psf/cachecontrol"],
 "remedy": "run the ingest for this repository first"
}

A window with no matching rows is also refused, with sample_size: 0 and a null value, rather than reporting zero as though it had been measured.

Measured results

From the recorded fixture (psf/cachecontrol: 12 pull requests, 8 reviews, 50 workflow runs):

pr_cycle_time          value=1.5422   unit=hours                          n=7
time_to_first_review   value=1.5411   unit=hours                          n=5
review_count           value=0.0      unit=reviews per pull request       n=12
merge_frequency        value=3.5      unit=merged pull requests per week  n=7
workflow_failure_rate  value=0.1667   unit=ratio                          n=48

Eval, run with python evals/run_delivery_eval.py:

mode=tools  questions=20
20/20 passed

All twenty pass. Each expected number is computed in tests/independent_metrics.py from the recorded JSON using datetime and statistics; that module does not import delivery_metrics, and a test asserts as much on the import graph, so the SQL cannot agree with itself.

One question was revised after a real-model run. q04 originally read "How many merged pull requests is the cycle time based on?" — an ambiguous question, original failure documented: it named no repository, and because each question is answered in a fresh conversation there was nothing for the model to resolve it against. It guessed a repository the policy then denied, and the run recorded a refusal where a number was expected. The question now names psf/cachecontrol. The original wording and the failure it caused are kept here on purpose; the fix was to the question, not to the model or the tools.

Test counts, measured on this branch's base and tip:

$ git worktree add --detach /tmp/before bec0558 && cd /tmp/before
$ python seed_books.py && python -m pytest -q -p no:cacheprovider
137 passed, 2 skipped in 19.56s

$ cd - && python -m pytest -q
206 passed, 2 skipped in 20.75s

69 new tests. Coverage of the new modules from python -m pytest -q --cov:

delivery_metrics.py       94      0   100%
delivery_tools.py         44      0   100%
delivery_ingest.py       143     34    76%
delivery_graphql.py       92     23    75%
delivery_fixtures.py     105     64    39%
TOTAL                   1466    233    84%

delivery_fixtures.py is low because its recording path talks to the live API and is exercised by hand, not by the suite.

Real model results

Numbers below are from runs against gpt-4o-mini. The deterministic eval (--mode tools) scores 20/20 and is a separate thing: it checks the metrics and citations, not the model.

Run

Scorer

Tool output

Score

Failing

Run 1, as recorded

original

before

12 / 20

not recorded

Run 2, as recorded

original

before

13 / 20

q01, q04, q05, q07, q12, q14, q19

Run 2, re-scored

fixed

before

13 / 20

q01, q04, q05, q07, q12, q14, q19

Run 3, after the tool change

fixed

after

17 / 20

q12, q14, q19

Run 4, after the tool change

fixed

after

17 / 20

q12, q14, q19

Run 5, after the tool change

fixed

after

17 / 20

q12, q14, q19

Run 2's answers are saved in evals/results/delivery-real-run2-before-fixes.json. Re-derive the re-scored figure with:

python evals/run_delivery_eval.py --rescore evals/results/delivery-real-run2-before-fixes.json

Runs 3 to 5 are three separate runs at commit b1ac32f, after both the scorer fixes and the tool change, saved as evals/results/delivery-real-b1ac32f-run1.json, run2.json and run3.json. All three scored 17/20 with the same three questions failing. There are no source changes between b1ac32f and the current tip, so they describe the code as it stands.

The before and after figures are separate and are not combined. Run 1 and run 2 were scored before the tool change, run 2's re-score only changes the scorer, and runs 3 to 5 are the only ones with the summary field and per_pull_request removed. The re-scored figure is not the original one — run 2 as recorded scored 13/20 under the original scorer, and re-scoring it under the fixed scorer also gives 13/20.

Scope of every figure here: SQLite, one repository (psf/cachecontrol), 12 pull requests, 8 reviews and 50 workflow runs — a small sample, and 20 questions asked once per run.

What the scorer fixes actually changed

Nothing, on run 2. Re-scoring moved no question: 13/20 before and after. Of ten individual failure reasons, one was resolved — q12's answer ended "is 3.5.", and the original rules treated the full stop as part of the number. q12 still fails because the answer never states the total. Every other failure had a cause the rules were right to flag.

The scorer defects were real and are fixed with tests, but on this run they were not what cost the points.

Which failures were what

Fixed by the tool change — these failed in run 2 and pass in all of runs 3 to 5:

  • q07 asked for the median reviews per pull request. The metric computed 0.0; the run 2 answer said 0.5, which is the median of twelve values if you count six zeros instead of seven — the model had recomputed the statistic from per_pull_request instead of quoting the one it was given. With that key no longer sent and a summary sentence in its place, all three later runs answer 0.0 and cite the sample of 12.

  • q01 and q05 stated the figure correctly in run 2 but never stated the sample size the question's rules also require. The summary sentence carries the sample size, and both pass in runs 3 to 5.

Model behaviour — still failing in all of runs 3 to 5:

  • q12 asks how many pull requests are merged per week. The answer, in all three runs, is "On average, 3.5 pull requests are merged per week in the psf/cachecontrol repository." The value check passes: 3.5 is correct and is stated. It fails on the second figure the question requires, the total of 7 merges, which the answer omits. That is the same class of omission as q01 and q05 in run 2 — a figure the rules ask for and the answer leaves out — and it is model behaviour, not a scorer or question defect.

  • q14 asks how many merges fell in the first week that had any, which needs the week discovered and then used as a window. All three runs answer "There were no merges in the first week of January 2023" — a window the model invented, for a year with no data. Unchanged, and no tool affordance was added for it.

  • q19 asks for "team velocity", which is not a metric the server has. All three runs report 3.5 merged pull requests per week as a velocity, presenting the merge-frequency figure under a name the server never offered. That is fabrication rather than a refusal, and broadening the refusal vocabulary does not and should not rescue it. Unchanged.

Scorer defect — the answer was acceptable and the rules were wrong:

  • q12 on its "value" check only, in run 2, where the answer ended "is 3.5." and the original rules read the full stop as part of the number. The rest of q12's failure, then and now, is not a scorer defect.

Question defect — the question could not be answered as written:

  • q04 named no repository. See the note in the evaluation results section above; the wording is now fixed and the original is recorded there. It passes in runs 3 to 5.

q14 and q19 are left exactly as they were, and no field was added to make them easier. They are recorded here as genuine model failures, reproduced identically three times.

Limits

  • One public repository at a time. The repository is configurable, but the ingest, the metrics and the tools all work on whatever has been ingested, and nothing here handles private repositories or an organisation-wide roll-up.

  • Read-only throughout. The ingest only issues GETs, the tools only read the local store, and the GraphQL schema has no mutation to reach.

  • The metric definitions are simple on purpose and are not any particular industry definition. Cycle time starts at pull request creation, not at first commit; time to first review ignores review requests and draft status; the failure rate counts runs, not jobs or steps, and treats a cancelled run as neither a success nor a failure. If you need different definitions, the SQL is in one file.

  • Small samples, so read the sample size. The committed fixture is 12 pull requests and 50 runs. In that data the cycle time median is 1.54 hours while the mean is 197 hours, because one pull request took 1349 hours. A median over seven rows is a description of seven rows, nothing more.

  • Not a complete history. Pull requests are fetched most-recently-updated first and workflow runs most-recent first, both capped. The numbers describe what was ingested, which is why every answer carries its sample size and its source identifiers.

  • No claims about customers, teams or scale. This has been run against one public repository's recorded data and nothing else. There is no benchmark here, no comparison between projects, and nothing about how it behaves on a large or busy repository.

  • SQLite, not Postgres. The brief allowed either. There is no Postgres server or driver configured in this project, so the delivery tables live in the same SQLite file as the audit log and usage meter. The schema is plain SQL with no SQLite-specific types; the date arithmetic (julianday, strftime) is the part that would need rewriting to move.

  • Reviews are counted as submitted, including COMMENTED ones, so review_count is review events rather than approvals.


LLM gateway

An optional HTTP service that accepts OpenAI-shaped /v1/chat/completions and /v1/responses requests, translates them to OpenAI or Anthropic, keeps session state so previous_response_id works against both, and enforces a monthly budget per API key using the metering already in this repository.

Off by default. Nothing in the agent, the MCP servers or the delivery metrics changes unless the gateway is started. See docs/gateway-design.md for what it reuses and the conflicts found while building it.

Running it

cp gateway.example.yaml gateway.yaml      # then edit; gateway.yaml is gitignored
export OPENAI_API_KEY=...                 # only for the providers you route to
export ANTHROPIC_API_KEY=...
uvicorn gateway.app:create_app --factory --host 127.0.0.1 --port 8080
curl -s localhost:8080/healthz

curl -s localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer <a key from gateway.yaml>' \
  -H 'content-type: application/json' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}'

What it supports

Endpoints

POST /v1/chat/completions, POST /v1/responses

Unauthenticated

GET /healthz returning exactly {"status": "ok"} and nothing about how the instance is configured, GET / listing the supported paths, and GET /favicon.ico returning 204, so a browser or a probe does not get a 404

Providers

OpenAI, Anthropic and Ollama, plus an in-process fake for tests and the benchmark

Request fields

model, messages / input, instructions, previous_response_id, temperature, max_tokens / max_output_tokens, stream

Content

Text only. A string, or a list of content parts whose text is concatenated in order

Streaming

SSE on both endpoints, text deltas, each in its own event format

Sessions

SQLite by default, Redis optional, with a TTL

Budgets

A monthly estimate per key; over it is 429 with code budget_exceeded

Routing

By model prefix, from config; the first matching prefix wins

Usage is recorded through the repository's existing meter, so gateway traffic shows up in python govctl.py usage --group user under the key's id.

What it does not support

Tools and function calling, images and any other non-text content, batch endpoints, embeddings, prompt caching, context compaction, n > 1, logprobs, structured-output response_format, and Anthropic's extended thinking. These are refused with a 400, not silently dropped, so a caller never gets an answer that looks as though the field had been honoured.

Only what the tests cover is claimed. The providers are exercised against recorded wire bodies through a mock transport, not against the live APIs, so "compatible with OpenAI" here means "matches the request and response shapes these tests assert".

The Anthropic mapping

Four rules, each a pure function in gateway/translate.py with its own tests:

  • system messages and /v1/responses instructions are hoisted to Anthropic's top-level system field. Instructions come first, then system turns in order, joined by a blank line.

  • Consecutive turns with the same role are merged, because Anthropic requires user and assistant to alternate. Their text is joined by a blank line.

  • max_tokens is required by Anthropic, so a request that omits it gets defaults.max_tokens from config.

  • stop_reason maps back to finish_reason: end_turn to stop, max_tokens to length. An unrecognised value falls back to stop rather than inventing one.

Reasoning models

On the chat completions endpoint, OpenAI's reasoning families take max_completion_tokens where other models take max_tokens. The gateway sends whichever the model expects, so a caller always supplies max_tokens (or max_output_tokens on /v1/responses) and does not have to know which family it is talking to.

Model name starts with

Upstream field

gpt-5, o1, o3, o4

max_completion_tokens

anything else

max_tokens

A /v1/responses request's max_output_tokens goes through the same mapping, because this gateway's OpenAI provider calls chat completions underneath. When the caller supplies no limit, neither field is sent.

temperature is passed through unchanged. Some reasoning models accept only the default temperature and will refuse any other value; the gateway does not rewrite or drop what the caller asked for, so such a request fails upstream and the error surfaces as a 502. The live run did not exercise this: no request in it set a temperature.

The offline tests assert only which field name the gateway puts on the wire for each family. One live call on 2026-11-10 confirmed that gpt-5.4-mini accepts the renamed field: a request carrying max_tokens: 64 came back as a completion rather than an upstream error. The o1, o3 and o4 families have never been called live, so for those the mapping remains an assertion about the request body and nothing more. The o4 prefix was added to gateway.example.yaml so those models are routable at all.

Sessions

/v1/responses stores the conversation under the response id it returns, so previous_response_id continues it — including on Anthropic, which has no server-side conversation of its own. An unknown or expired id is a 404 with code response_not_found.

Backends are behind one interface. SQLite is the default and shares the database the audit log and meter use. Redis is optional, for running more than one gateway process; it uses Redis's own TTL.

Budgets and cost

monthly_budget_usd per key. Spend is not counted twice: the gateway writes one usage_llm row per upstream call through the existing metering.record_llm_turn, and month-to-date spend is read back out of those rows. There is one source of truth.

Every cost figure is an estimate. It is the recorded token counts multiplied by whatever is in the configured price table, and the committed prices.example.yaml is all zeros on purpose, so a budget built on it can never be exceeded. Fill in prices.yaml from your provider's current pricing before relying on a budget.

Two limits worth knowing:

  • A cost is not knowable before a call, so the check is against spend already recorded. A key can therefore overshoot its budget by at most one request.

    That holds under parallel load as well, but only because the check and the charge are held together. Measured on this branch before they were: 30 requests at once on a key whose budget allowed one call admitted 24 of them and recorded 24 times the limit, because every one read the same month-to-date spend before any had written a row. Each key now admits one request at a time through that window, so the overshoot is the single call that was in flight when the limit was crossed. A key with a budget therefore serves its requests one at a time. A key with no budget takes no lock and stays fully parallel, and keys never wait on each other. The alternative, reserving an estimated cost up front and letting several through, was not taken: the estimate would either refuse requests that would have fitted or fail to bound what it was meant to bound.

  • When an upstream reports no usage for a streamed response, the gateway records a zero row and logs that it did, rather than inventing numbers. That under-counts spend for that request.

Streaming formats

Each endpoint streams in its own shape, which the tests assert are distinct.

/v1/chat/completions emits unnamed data: frames of chat.completion.chunk objects, then data: [DONE]. The role is sent once, in the first chunk; the chunks after it carry content only, and the last carries finish_reason. No chunk carries a usage object: stream_options.include_usage is not implemented, so a streaming chat caller cannot read token counts off the stream. The tokens are still metered — they are recorded once the stream finishes — so govctl.py usage reports them.

/v1/responses emits these named events, in this order:

#

Event

1

response.created

2

response.output_item.added

3

response.content_part.added

4

response.output_text.delta (one per delta, repeated)

5

response.output_text.done

6

response.content_part.done

7

response.output_item.done

8

response.completed

response.created carries output: []; the message item first appears in response.output_item.added. Every event carries a sequence_number, consecutive from 0, and one stable message item id appears in every event after response.created — as item_id, as item.id, and in response.completed as response.output[0].id. sequence_number is not optional: the installed openai package's stream-event models declare it as a required field, and tests/test_gateway_responses_stream.py parses every event this gateway emits through those models offline, so the events are checked against the SDK rather than assumed to match it. logprobs is sent as an empty list on the delta and done events for the same reason; this gateway does not support logprobs.

Both the non-streaming /v1/responses body and the response object embedded in the stream events carry the fields the SDK's Response model requires: parallel_tool_calls (always false), tool_choice (always "none"), tools (always []) — the gateway has no tools — and the two usage detail objects. Tests validate both bodies against that model offline.

These usage details are always zero: input_tokens_details.cached_tokens, input_tokens_details.cache_write_tokens and output_tokens_details.reasoning_tokens. Zero here means the gateway does not track the value, not that it measured none. It neither counts prompt caching or reasoning tokens itself nor reads them back from an upstream.

Benchmark

scripts/bench_gateway.py measures the gateway's own overhead against a fake provider with a fixed delay, so the difference is request parsing, routing, the budget read, the provider call, the session write and the response build.

python scripts/bench_gateway.py --requests 200 --delay-ms 5 --endpoint chat

Measured on one machine over loopback, 200 requests after 20 warmup, with a 5 ms fake provider delay:

endpoint            /v1/chat/completions
round trip P50      9.802 ms
round trip P95      11.824 ms
gateway added P50   4.802 ms
gateway added P95   6.824 ms

endpoint            /v1/responses
round trip P50      9.619 ms
round trip P95      10.065 ms
gateway added P50   4.619 ms
gateway added P95   5.065 ms

This is one machine on loopback. It is not a provider latency measurement, not a throughput or concurrency measurement, and says nothing about behaviour under load. Each request includes a SQLite budget read and a usage write, and /v1/responses also writes a session row. Re-run the script rather than trusting these numbers on different hardware.

Concurrency

scripts/bench_gateway_concurrency.py holds a fixed number of requests in flight and reports throughput, the latency spread, failures and peak memory.

python scripts/bench_gateway_concurrency.py
python scripts/bench_gateway_concurrency.py --levels 1,10,50 --delay-ms 10
python scripts/bench_gateway_concurrency.py --client-processes 5

Measured on one machine, loopback, an in-process fake provider with a 5 ms delay, 400 requests per level after 20 warmup, on a key with no budget. The server ran sync endpoints through a 40-thread worker pool.

With the load generator in the same process as the server, which is the default:

concurrency      req/s     P50 ms     P95 ms     P99 ms  errors   peak MB
          1     105.56      9.109     11.591     17.059       0      62.0
         10     965.65      8.587     18.048     30.002       0      63.8
         50     350.05    118.403    292.251    387.871       0      76.3
        200     173.60    583.794   1741.758   1860.290       0      95.1

With the same load spread over five client processes (--client-processes 5), so the client's threads no longer share a GIL with the server's:

concurrency      req/s     P50 ms     P95 ms     P99 ms  errors   peak MB
          5     284.17      8.429     17.688     42.465       0      60.0
         10     447.92      9.876     20.770     33.456       0      62.2
         50     488.00     19.136    107.284    223.284       0      62.2
        200     505.78    133.015    181.567    268.894       0      63.7

The two tables disagree, and the second one is the better measurement. Read on its own, the first looks like throughput collapsing past ten concurrent requests. It is mostly not: at 50 and 200 the client's threads are competing with the server's for one interpreter, and moving them out roughly trebled throughput at the same concurrency while cutting P50 from 118 ms to 19 ms. A single run at 50 in flight with a 50 ms delay made the same point from the other side: 118 req/s with one client process against 324 req/s with five, where queueing alone through a 40-thread pool would have predicted about 800. So the first table is reported because it is what the default command prints, not because it describes the gateway.

What these numbers do not establish: they are one machine over loopback against a fake provider, so they are not provider latency, not capacity, and not transferable to other hardware, another database or a real upstream. Every level completed with zero errors, which is the part worth keeping: nothing failed, no budget was overshot and no stream was mixed up under load. Levels are rounded to a multiple of --client-processes, which is why the second table starts at 5.

Re-run the script rather than trusting any of this on different hardware. It writes a JSON file per run into bench-results/, which is gitignored.

Self-hosted models through Ollama

Ollama serves an OpenAI-compatible API, so a self-hosted model is routed like any other. Unlike the other providers its address comes from the config file rather than the environment: there is no credential to keep out of the file, and no key is sent.

routes:
  - prefix: "llama3"
    provider: ollama
  - prefix: "qwen"
    provider: ollama

providers:
  ollama:
    base_url: "http://localhost:11434/v1"
    timeout_seconds: 120

The example prefixes carry Ollama's own version digit (llama3.2, not llama-3), so a model name in some other shape still falls through to a 400 rather than being sent to a model that does not exist. Route whatever ollama list shows on your machine.

Both endpoints work, streaming and not, exactly as for the other providers. A caller's max_tokens goes upstream as max_tokens: these are not OpenAI's reasoning families, so none of that renaming applies.

Errors map onto the shapes already in use. A server that is not running gives a 502 saying so by address rather than an exception class, a request past timeout_seconds gives a 502 naming the deadline, and Ollama's habit of answering some failures with a bare {"error": "..."} string is handled alongside the usual nested form.

Limits. Text only: tools, images, embeddings and batching are not implemented, and the gateway refuses the fields for them with a 400 as it does for every provider. Nothing here says anything about speed, hardware or what a given model can do. The provider is tested against a mock transport only, so what is claimed is that the request and response shapes match what those tests assert; no Ollama server was run.

Administration

python -m gateway.cli manages keys, budgets and usage against the same config file and the same usage rows the running gateway uses. There is no second store.

python -m gateway.cli keys create --id ci --name "build runner" --budget 10
python -m gateway.cli keys list
python -m gateway.cli budget set --id ci --budget 25
python -m gateway.cli budget set --id ci --unlimited
python -m gateway.cli usage show
python -m gateway.cli requests tail -n 20

keys create generates the key, writes it to the config and prints it once; every other command masks a key to its last four characters. --json works on all of them, and a command that cannot do what was asked exits 2 with a message on stderr.

Two things it will not do. It refuses to write to the committed gateway.example.yaml, and it validates a config before replacing it, so a write that would not load afterwards is rejected with the file untouched.

Three limits worth stating plainly:

  • Writing a key or a budget rewrites the config through a YAML load and dump, so comments and layout in the file are lost. The write is atomic and keeps the file's permissions, but a commented config will come back uncommented. Only keys create and budget set write; the reading commands never touch the file.

  • requests tail shows no latency and no HTTP status, because the meter records neither. A usage_llm row carries a timestamp, the key id, the model and the token counts. The provider column is worked out from the configured routes rather than stored per request, and the status column distinguishes only a call that reported usage from one that did not. A request refused before it reached an upstream leaves no row at all.

  • keys list shows a name and a creation time only for keys that have them. Both are optional notes added by keys create; a key written by hand shows a dash and works exactly as before.

No message content is stored by the meter or printed by any of these commands.

Tests

Every gateway test uses the in-process fake provider or a mock transport, so none reaches a provider or the network.

pytest -q tests/test_gateway_translate.py tests/test_gateway_endpoints.py \
         tests/test_gateway_providers.py tests/test_gateway_sessions.py \
         tests/test_gateway_budgets.py tests/test_gateway_config.py \
         tests/test_gateway_bench.py tests/test_gateway_public_routes.py \
         tests/test_gateway_responses_stream.py tests/test_gateway_ollama.py \
         tests/test_gateway_cli.py tests/test_gateway_concurrency.py \
         tests/test_gateway_bench_concurrency.py tests/test_store_concurrency.py

That is 280 tests. The whole repository's suite is 555 passed and 2 skipped, the two skips being the tests that need a reachable Redis.

The concurrency tests drive the app from a thread pool and assert what has to hold under parallel load: that every request gets its own correct body, that a budget is not overshot, that two streams are never interleaved and that parallel continuations of one response id each branch rather than overwrite each other. One of them checks that requests really did overlap, so the others cannot pass by accident on a machine that ran them one at a time.

The Redis session tests need a reachable Redis and skip themselves without one. CI runs them in a second job against a redis:8-alpine service container, where a skip is treated as a failure so a broken container cannot pass quietly. The existing CI job's steps are unchanged apart from adding the new modules to the type check and the new config file to the parse check.

Live verification

The gateway was run once against the real OpenAI and Anthropic APIs on 2026-11-10, from a throwaway config and a throwaway SQLite database outside the repository: 17 upstream calls, each capped at 64 output tokens. The models called were gpt-4o-mini, gpt-5.4-mini and claude-haiku-4-5-20251001.

Run live

Result

/v1/chat/completions, non-streaming

passed on all three models

max_tokens on gpt-5.4-mini

passed; the max_completion_tokens rename is accepted upstream

/v1/responses, non-streaming, instructions set

passed on gpt-4o-mini and claude-haiku-4-5-20251001, and both obeyed the instruction

Streaming, both endpoints, both providers

passed; frames arrived spread across the response rather than in one burst, one stable item id per /v1/responses stream, clean termination, nonzero final usage on /v1/responses

previous_response_id chaining

passed on both providers; the second turn recalled a code word set in the first, and its input token count rose, so the history really did go upstream

The openai 3.6.0 SDK client, pointed at the gateway

passed; client.responses.stream and client.responses.create with a chain both ran against claude-haiku-4-5-20251001 with no validation error

Error shapes

passed; 404 response_not_found, 401 missing_api_key, 400 model_not_found, 429 budget_exceeded

Usage accounting

passed; per-key rows with nonzero token counts for every call, the streaming ones included

That run says nothing about the following, which remain covered by offline tests or not at all:

  • The o1, o3 and o4 reasoning families, and the temperature behaviour described under Reasoning models.

  • Anything beyond text. No tools, no images, no structured output.

  • Any client other than openai 3.6.0.

  • Concurrency or load. Every call in the run was sequential.

  • Redis-backed sessions. The run used the SQLite backend.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers