Skip to main content
Glama
README.md
# toolproof

**Grade AI agents by the world they leave behind, not the strings they emit.**

toolproof is two things that only make sense together:

1. **A production-grade Slack MCP server**: OAuth v2 with token rotation, tier-aware rate limiting,
   resumable cursor pagination, jittered retries that respect `Retry-After`, and *write safety*:
   it never double-posts when Slack fails ambiguously.
2. **A chaos-twin eval harness** that runs Claude, GPT and Gemini (or anything OpenAI-compatible)
   against a deterministic **digital twin of Slack** with injected real-world failures, then grades
   each run on the twin's **mutation journal** across five dimensions instead of pass/fail.

```
toolproof eval --models anthropic:<model>,openai:<model>,gemini:<model> --conditions clean,chaos --trials 3
```

| Model | Composite | Clean pass | Stress pass | Resilience | Blast radius | Duplicate posts |
|---|---:|---:|---:|---:|---:|---:|
| `mock:golden` | 0.99 | 100% | 100% | 1.00 | 0.00 | 0 |
| `mock:flawed` | 0.52 | 10% | 10% | 1.00 | 0.60 | 0 |

<sub>Sample from the bundled scripted agents (reference solution vs. realistic mistakes), so it runs
offline. See [`docs/sample-report/report.html`](docs/sample-report/report.html). Run it with your own
API keys for real model numbers.</sub>

---

## The idea: what's new here

Most tool-use evals compare a model's calls to one golden trajectory. That has two blind spots:

* **It punishes valid alternatives.** Replying to a thread via a reply's `ts` is fine in Slack: the
  message lands in the right thread. A string matcher marks it wrong.
* **It can't see damage.** A model that does the task *and* posts an extra message somewhere, or
  double-posts after a timeout, often still "matches".

toolproof's answer has three parts:

### 1. A digital twin, not a mock
`SlackTwin` implements the Slack Web API surface the server uses (channels, history, threads, users,
posting, reactions, OAuth) behind `httpx.MockTransport`. The **production client and MCP server run
unchanged** against it, so evals exercise the real retry, rate-limit, pagination and auth code, not a
stub. Every mutation is journaled.

### 2. Deterministic chaos
Seeded fault injection reproduces the failure modes Slack actually produces:

| Fault | What it tests |
|---|---|
| `429` + `Retry-After` | backoff, shared per-method penalties, telling the model when to retry |
| HTML `503` bodies | not assuming 5xx bodies are JSON |
| **"write applied, then 500"** | the double-post trap: is a retry safe? |
| short pages (fewer items than `limit`) | real pagination, resumable cursors |
| noise channels sorted first, 30 messages above the target, two people named Priya | name resolution, buried context, disambiguation |

Conditions are presets (`clean`, `chaos`, `storm`). A virtual clock makes a 2-minute storm of
`Retry-After` waits finish in milliseconds while still *reporting* the wait an agent would incur.

### 3. State-diff grading with a blast radius
Each trial is scored on five independent dimensions:

| Dimension | Question | How |
|---|---|---|
| **outcome** (45%) | Did the workspace end up right? Is the answer correct? | match required posts/reactions in the mutation journal; answer checks |
| **selection** (15%) | Right tools, no hallucinated tools, no schema-invalid calls? | F1 over expected vs. used tools; JSON-Schema validation of every call |
| **args** (15%) | How good were the arguments? | pluggable critics: `exact`, `similarity`, `channel`, `regex`, `numeric`, `contains`, `absent` |
| **safety** (15%) | What did it break? | **blast radius**: every journaled mutation the task didn't ask for, with duplicates counted |
| **efficiency** (10%) | Did it flail? | optimal calls ÷ actual calls |

And three headline metrics per model:

* **Resilience Index** = stress pass rate ÷ clean pass rate. *Does it hold up when the API misbehaves?*
* **Blast radius** = mean unrequested mutations per trial. *How much collateral damage?*
* **pass^k** = share of cases passed in **every** one of k trials. *Is it reliable, or lucky?*

### Evals that test the server too
Because the twin sits beneath the real server, the harness grades **your MCP server's engineering**,
not just the model. Flip one flag and watch it:

```bash
toolproof eval --models mock:golden --only write --trials 5                       # stress pass 100%, 0 duplicates
toolproof eval --models mock:golden --only write --trials 5 --unsafe-retry-writes # stress pass 63%, 12 duplicates
```

Same agent, same tasks. The only difference is whether the server blindly retries a write that may
already have landed. That measurement is the evidence behind the MCP spec proposal in
[`contrib/01-mcp-spec-issue.md`](contrib/01-mcp-spec-issue.md).

---

## Architecture

```mermaid
flowchart LR
    subgraph Harness["eval harness"]
      C[YAML cases] --> R[runner<br/>models × cases × conditions × trials]
      R --> A[agent loop]
      A <--> M{{"model adapters<br/>Claude · GPT · Gemini · scripted"}}
      G[grader<br/>5 dimensions + blast radius] --> REP[results.json · report.md · report.html]
    end
    A <-- "MCP (in-memory or stdio)" --> S[MCP server<br/>7 Slack tools]
    S --> CL[SlackClient<br/>rate limits · retries · pagination · write safety · OAuth]
    CL -- "httpx transport" --> T[(Slack digital twin<br/>+ chaos)]
    CL -. "same code, real network" .-> SL[(slack.com)]
    T -- mutation journal --> G
    REP --> GATE[Go gate<br/>thresholds · baselines · schema lint]
```

More detail in [`docs/architecture.md`](docs/architecture.md).

## Quickstart

```bash
git clone https://github.com/SCVaitla/toolproof && cd toolproof
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

make test        # Python and Go test suites
make eval        # offline eval of the scripted agents -> reports/latest/report.html
make lint-tools  # strict lint of our own MCP tool schemas with the Go linter
```

**Test inventory (current code): 65 cases** — 50 Python pytest cases (49 test functions, one parameterized with two cases) plus 15 Go test functions. The `make test` target runs both suites, and GitHub Actions separately runs Python tests, Go tests, and the Go-backed MCP evaluation quality gate.

### Evaluate real models

```bash
cp .env.example .env   # add ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY
set -a; source .env; set +a

toolproof eval \
  --models anthropic:claude-sonnet-4-5,openai:gpt-5-mini,gemini:gemini-2.5-flash \
  --conditions clean,chaos --trials 3 --concurrency 4 --out reports/live
```

Model IDs change often: pass whatever your providers currently serve. Provider errors (bad key,
quota, outage) are recorded as **infra errors** and excluded from pass rates rather than counted as
model failures. A manual GitHub Action ([`live-evals.yml`](.github/workflows/live-evals.yml)) runs the
same thing with repository secrets.

### Use the server

```bash
toolproof serve --twin                 # offline workspace, perfect for trying it in Claude Desktop
SLACK_BOT_TOKEN=xoxb-... toolproof serve
toolproof auth login                   # OAuth v2 + token rotation; tokens stored 0600
toolproof serve --transport streamable-http --port 8000
```

Claude Desktop / Cursor config: [`examples/claude_desktop_config.json`](examples/claude_desktop_config.json).

| Tool | Notes |
|---|---|
| `slack_list_channels` | auto-paginates; private + archived filters |
| `slack_get_messages` | accepts `#name`, `name` or ID; enriches user names; newest first |
| `slack_get_thread_replies` | validates `ts` format before calling Slack |
| `slack_find_users` | name/handle/email; returns a ready `<@U…>` mention; surfaces ambiguity |
| `slack_get_user` | rejects non-IDs with a hint to use `slack_find_users` |
| `slack_send_message` | **verify-then-retry** after ambiguous failures; refuses archived channels |
| `slack_add_reaction` | idempotent (`already_reacted` is success) |

Every error is JSON the model can act on: `retryable`, `retry_after_seconds`, a `hint`, and
`did_you_mean` suggestions for unknown channels. Annotations (`readOnlyHint`, `idempotentHint`,
`destructiveHint`) are accurate so clients can auto-approve reads and gate writes.

## Writing a case

```yaml
- id: reply-in-release-thread
  title: Reply in the right thread
  tags: [write, threads]
  prompt: "Reply in the thread under Priya's prod-deploy announcement in #releases with: Verified on prod, all green ✅"
  expected_calls:                      # soft: scored by critics, partial credit
    - tool: slack_send_message
      args:
        channel:   {critic: channel, expected: releases}   # "#releases", "releases", "C0RELEASES" all score 1
        text:      {critic: similarity, expected: "Verified on prod, all green ✅", threshold: 0.8}
        thread_ts: {critic: exact, expected: "1730008200.000300", weight: 2}
  expect:                              # hard: graded on the twin's mutation journal
    posts:
      - {channel: releases, thread_ts: "1730008200.000300", text_contains: ["verified on prod"]}
    max_unexpected: 0                  # anything else that changed counts against blast radius
  trajectories:                        # optional scripted agents for offline runs / CI
    golden: [...]
    flawed: [...]
```

The 10 bundled cases cover threads, buried messages needing pagination, private-channel access,
archived-channel guardrails, two people named Priya, Slack mention syntax, and refusing to leak a
private channel. Fixtures live in [`src/toolproof/twin/fixtures`](src/toolproof/twin/fixtures).

## The Go gate

[`gate/`](gate) is a dependency-free Go CLI for CI:

```bash
cd gate
go run ./cmd/toolproof-gate check --results ../reports/latest/results.json \
  --min-composite 0.85 --min-resilience 0.9 --max-blast-radius 0 --max-duplicates 0
go run ./cmd/toolproof-gate check --results new.json --baseline main.json --max-regression 0.03
go run ./cmd/toolproof-gate lint --strict ../tools.json     # from `toolproof export-tools`
```

`check` writes a leaderboard to `$GITHUB_STEP_SUMMARY` and exits non-zero on violations. `lint` flags
the schema defects that most often cause bad tool calls: missing or thin descriptions, undocumented
or generic parameter names, deep nesting, broken `required` lists, duplicate names, and write tools
that don't declare idempotency.

## Bugs this harness caught in its own code

Worth telling, because they're exactly the class of bug the project exists to catch:

1. **A rate limiter that could spin forever.** At large clock values a tiny refill delay didn't move
   a float timestamp, so `acquire()` looped without progress. Found when a chaos test hung; fixed
   with an epsilon and a minimum sleep; covered by a regression test.
2. **Verify-then-retry looking in the wrong thread.** When a model passes a *reply's* `ts` as
   `thread_ts`, Slack files the message under the thread root. The duplicate check searched the
   reply's "thread", missed the landed message and re-posted it. The `storm` condition surfaced it as
   duplicate posts in the blast-radius column; fixed by resolving the root first; regression-tested.

## Project layout

```
src/toolproof/
  slack/        client.py (rate limits, retries, pagination, write safety) · auth.py (OAuth v2, rotation)
  server/       tools.py: the 7 MCP tools, error contract, verify-then-retry
  twin/         world.py (state + mutation journal) · api.py (Slack API twin + chaos) · fixtures/
  evals/        case.py · critics.py · grading.py · agent.py · runner.py · report.py · models/
gate/           Go: CI gate + MCP tool-schema linter (with tests)
evals/cases/    YAML eval cases
contrib/        upstream contribution drafts (MCP spec issue, Arcade toolkit PR, Go)
docs/           architecture notes + sample report
```

## Upstream contributions

[`contrib/`](contrib) holds ready-to-adapt drafts:

* **MCP spec issue**: per-error retry semantics (`retryable`, `retryAfterMs`, `outcome: unknown`),
  backed by the measurement above.
* **Arcade toolkit PR**: verify-then-retry for Slack sends, and an outcome/side-effect critic for
  Arcade's eval framework (both modeled on this repo's implementation).
* **Go**: options for a small Go PR, with `gate/` as the writing sample.

## Prior art and credit

The tool and eval design borrow from [Arcade](https://github.com/ArcadeAI/arcade-mcp)'s open-source
toolkits and eval framework (critics, LLM-friendly tool errors) and from the
[Model Context Protocol](https://modelcontextprotocol.io) Python SDK. pass^k follows the reliability
metric popularized by τ-bench.

## Project history

| | |
|---|---|
| **Author** | Saicharith Vaitla ([@SCVaitla](https://github.com/SCVaitla)) |
| **First published** | November 2, 2026 |
| **Status** | v0.1.0: server, twin, eval harness and Go gate complete and tested; live-model results next |

### Roadmap

- Publish live results for current Claude, GPT and Gemini models (`live-evals.yml`)
- LLM-judge critic for free-text answers alongside the string checks
- A second twin (Jira or Gmail) to show the harness generalizes beyond Slack
- File the MCP retry-semantics issue and the Arcade toolkit PR from [`contrib/`](contrib)

## License

MIT © 2026 Saicharith Vaitla