Skip to main content
Glama
agent-kit-server

agent-kit-server

by jayu1023
README.md
# agent-kit-server

The backend [`agent_kit`](https://pub.dev/packages/agent_kit) was missing.

`agent_kit` ships the Flutter widgets for the non-text parts of an agent UI —
tool-call cards, thinking bubbles, citation chips, plan timelines — and is
backend-agnostic, which means every adopter has had to write the agent loop
themselves. This is that loop: a LangGraph agent whose SSE stream decodes
directly into `AgentEvent`, with four real tools, a human approval gate before
any write, and the same tools exposed over MCP.

```dart
AgentView(stream: agentKitServerEvents(question: 'How does CSAT compare across plans?'))
```

It runs with **no API key, no network and no Docker**: the default planner is
deterministic and `web_search` searches a bundled corpus. `make install && make test`
is the whole setup.

**Docs:** [`docs/EXPLAINER.md`](docs/EXPLAINER.md) — how each piece works and why.
[`docs/INTERVIEW_GUIDE.md`](docs/INTERVIEW_GUIDE.md) — how to present and defend it.
[`docs/TRACE.md`](docs/TRACE.md) — real captured span trees, including the approval pause.

---

## The graph

```
                  ┌──────────────────────────────────────────────┐
  POST /runs ────▶│ plan                                         │
                  │  heuristic | llm → PlanStep[] + Action[]      │
                  │  emits: plan, thinking                        │
                  └───────────────────┬──────────────────────────┘
                                      ▼
                  ┌──────────────────────────────────────────────┐
         ┌───────▶│ act                                          │
         │        │  resolve args (late-binding resolvers)        │
         │        │  write tool? ──▶ interrupt() ──▶ PAUSE        │
         │        │  emits: plan(running), tool_call(pending|…)    │
         │        └───────────────────┬──────────────────────────┘
         │                            ▼
         │        ┌──────────────────────────────────────────────┐
         │        │ observe                                      │
         │        │  run_tool: timeout + bounded retry            │
         │        │  emits: tool_call(terminal), plan, citations   │
         │        └───────────────────┬──────────────────────────┘
         │                            ▼
         │        ┌──────────────────────────────────────────────┐
         │        │ reflect                                      │
         └────────┤  advance cursor · enforce MAX_STEPS           │
       more steps │  emits: thinking                              │
                  └───────────────────┬──────────────────────────┘
                                      ▼ halt_reason set
                  ┌──────────────────────────────────────────────┐
                  │ answer → control: done                       │
                  └──────────────────────────────────────────────┘

  POST /runs/{id}/approve ──▶ Command(resume=…) ──▶ re-enters `act`
```

| File | Responsibility |
|---|---|
| `app/events.py` | the `agent_kit` wire protocol — the **only** module that knows the Dart shapes |
| `app/state.py` | the graph's state; what survives the approval round-trip |
| `app/planner.py` | heuristic + LLM planners, the tool allowlist, late-binding resolvers |
| `app/graph.py` | the four nodes, the interrupt, the loop guard (`single` mode) |
| `app/specialists.py` | specialist definitions: tool sets, budgets, dispatch order |
| `app/supervisor.py` | the multi-agent graph: route → dispatch → synthesize |
| `app/guards.py` | tool timeout, bounded retry, the permanent/transient split |
| `app/tools/` | `web_search`, `query_tickets`, `calculator`, `save_report` |
| `app/main.py` | `POST /runs` (SSE), `POST /runs/{id}/approve`, `GET /runs/{id}`, `/tools`, `/health` |
| `mcp_server.py` | the same tool objects over MCP stdio — no duplicated definitions |
| `app/tracing.py` | OpenTelemetry spans; file / OTLP exporters, off by default |
| `app/trace_tree.py` | renders a JSONL trace as the span trees in `docs/TRACE.md` |
| `clients/flutter_smoke/` | Dart tests that decode real server output with the real package |

---

## Quick start

```bash
make venv install
make test                    # 132 python tests
make test-dart               # regenerate the fixture, then 8 Dart wire tests

make dev                     # uvicorn on :8000
make ask Q="How does average first response time compare across plans?"
make ask Q="What is our CSAT by plan and save a report about it"   # pauses
make approve                 # or: make approve OK=false
```

---

## The four tools

| Tool | What it does | Sandbox |
|---|---|---|
| `web_search` | passages from the policy corpus | provider-pluggable: `offline` (default, deterministic) or `tavily` |
| `query_tickets` | one read-only SELECT over a 35-row ticket table | SQLite `authorizer` callback, single-statement check, forced `LIMIT` |
| `calculator` | arithmetic | AST allowlist — no names, attributes or calls are reachable |
| `save_report` | **writes a file** | path-traversal check; **requires human approval** |

Schemas are generated from the Pydantic args models, never hand-written, so a
schema cannot drift from a function signature. `GET /tools` serves them, the LLM
planner is given them, and MCP advertises them — one definition, three consumers.

### The sandboxes are the interesting part

`calculator` does not use `eval()`, because `__import__('os').system('rm -rf /')`
is a valid Python expression and an LLM will eventually emit one. It walks the
AST and rejects every node type that is not a number, an operator or a
parenthesis, so there is no reachable path to a call. `2 ** 10 ** 9` is rejected
separately — that one is a denial of service, not an injection.

`query_tickets` installs a SQLite `set_authorizer` callback that permits only
`SQLITE_SELECT`/`SQLITE_READ`/`SQLITE_FUNCTION`. The database refuses writes
itself, which is a stronger guarantee than a regex over the SQL string. The
single-statement check stops `SELECT 1; DROP TABLE tickets`, and a `LIMIT` is
appended when the model omits one.

`save_report` treats the model-chosen filename as untrusted input:
`title="../../.ssh/authorized_keys"` is slugified and then checked with
`is_relative_to` against the reports directory.

---

## Guards, and where each one lives

Deliberately in the graph and in one shared execution path — never bolted onto
individual call sites, because a guard you have to remember is a guard you will
forget on one branch.

| Guard | Where | What it stops |
|---|---|---|
| step budget (`MAX_STEPS`) | `reflect()` | an agent that never terminates |
| per-tool timeout | `guards.run_tool` | one hung tool holding a request open |
| bounded retry + **permanent/transient split** | `guards.run_tool` | retrying a bad argument forever |
| approval before writes | `act()`, via `interrupt()` | a side effect nobody sanctioned |
| tool allowlist + arg schema validation | `planner._validate` | a hallucinated tool or argument ever executing |
| failures do not cascade | `reflect()` | one bad step discarding a usable answer |

**The permanent/transient split is the one worth arguing about.** Retrying
`SELECT * FROM ticekts` three times produces the same typo three times, spends
the step budget and hides the real problem. So `ValueError`/`TypeError`/
`ValidationError` fail immediately and are reported back in a legible form the
model can correct from; timeouts and transport errors get exponential backoff.

**Validation is a guardrail, not error handling.** The LLM planner's output is
checked against the real tool registry *before* anything runs — an unknown tool
name or a schema violation discards the whole plan at the planning boundary. That
is the difference between "the model can call tools" and "the model can call
*these* tools with *these* arguments".

---

## Human-in-the-loop

`save_report` is in `WRITE_TOOLS`, so the graph interrupts before it runs.
Approval is a **separate HTTP request**, not a reply on the open stream:

```bash
curl -N -X POST :8000/runs -d '{"question":"... and save a report about it","run_id":"r1"}'
# … event: control  {"kind":"approval_required","tool":"save_report","args":{…}}
# stream ends here. The run is checkpointed, not waiting on the socket.

curl :8000/runs/r1          # {"awaiting_approval":[{"tool":"save_report",…}]}
curl -N -X POST :8000/runs/r1/approve -d '{"approved":true}'
curl -N -X POST :8000/runs/r1/approve -d '{"approved":false,"note":"needs legal review"}'
```

Because the state is in the checkpointer, the run survives the stream closing, a
reviewer taking an hour, and (with a durable checkpointer) a process restart.
Holding approval on the original connection would make a human decision depend on
a socket staying up.

A rejection is not a failed run: the write is marked `cancelled`, the read-only
work already done is kept, and the answer says a write was rejected and why.

### The LangGraph gotcha this surfaced

**After an interrupt, the node re-runs from the top on resume** — everything
above `interrupt()` executes twice. So `act()` emits exactly one thing before
interrupting: a `tool_call` whose id is stable, which the Flutter client renders
as the *same* pending card both times. A thinking bubble was emitted there
originally and appeared twice in the client, because `ThinkingStep` has no id to
dedupe on.

---

## MCP

The same four tools, over stdio, to any MCP client:

```bash
claude mcp add agent-kit-server -- /abs/path/.venv/bin/python /abs/path/mcp_server.py
```

The tool definitions are **not duplicated**. `mcp_server.py` iterates
`ALL_TOOLS` and registers each one with the input schema taken from the same
Pydantic model the LangGraph agent uses. Add a tool to `app/tools/__init__.py`
and it appears over MCP with no edit to that file — the only version of "we
support MCP" that does not rot.

`save_report` is advertised with `readOnlyHint: false`. **MCP has no equivalent
of the graph's approval interrupt** — the host application owns that decision,
and Claude Code/Desktop prompt for permission themselves. The annotation is what
lets a host do that; it is not a gate this process enforces.

---

## Wire compatibility is tested, not asserted

This is the part worth looking at. `agent_kit`'s `AgentEvent.fromJson` accepts
exactly four `type` tags and throws `FormatException` on anything else, with
snake_case field names and specific enum spellings. A typo here is a crash on
someone's phone.

So the contract is tested from the Dart side, against the real published package:

1. `python scripts/capture_events.py` runs the **real agent** — a read-only run,
   plus a write run that pauses and is approved — and dumps all 39 emitted frames
   to `clients/flutter_smoke/test/fixtures/events.jsonl`.
2. `cd clients/flutter_smoke && flutter test` decodes every frame with the real
   `agent_kit` models and asserts: every frame decodes; each decodes to the
   subtype its tag claims; **`fromJson` → `toJson` round-trips losslessly** (which
   catches a field the server sends and the Dart model silently drops); every
   tool-call id reaches a terminal state and never moves backwards out of one;
   every plan step terminates; every citation carries a parseable url.
3. CI runs both jobs, so the two sides cannot drift silently.

A hand-written fixture would only prove the fixture matches the client. Capturing
from the real server is what makes it prove anything.

**It found two real bugs**, both now regression-tested on both sides:

- **`observe()` updated the plan in state but never emitted it.** The checkpoint
  was right and the client was wrong: the last plan step stayed `running` forever
  in the Flutter timeline. Caught by "every plan step reaches a terminal state".
- **Tool-call ids were only unique within a run.** Two runs both start at step
  `s1`, so a client showing more than one run in a view would collide their
  cards. Ids now include the run id. Caught by a test decoding two concatenated
  runs from one fixture.

---

## Two modes: one agent, or a supervisor routing to specialists

```bash
curl -X POST :8000/runs -d '{"question":"...","mode":"supervisor"}'
curl :8000/specialists
```

`single` is the default and is the right answer for most tasks. `supervisor`
routes to specialist agents:

| Specialist | Tools | Budget | Writes |
|---|---|---|---|
| `researcher` | `web_search` | 2 steps | no |
| `analyst` | `query_tickets`, `calculator` | 3 steps | no |
| `scribe` | `save_report` | 1 step | **yes — approval required** |

```
route -> dispatch(specialist) -> [more specialists] -> synthesize
```

### What makes this multi-agent rather than one agent with a label

Three things, and the third is the one usually missing:

1. **Restricted tool sets.** The analyst cannot write; the scribe cannot query. A test
   asserts the sets are strict subsets *and* that they do not overlap — overlap would mean
   two agents can do the same job, so routing stops meaning anything and a failure is
   ambiguous between them.
2. **Per-specialist step budgets.** A shared budget lets one agent starve the others, which
   is the classic multi-agent failure. Budgets live on the specialist, not on the run.
3. **An explicit state contract.** A specialist receives the question and the findings so
   far, and returns `Finding`s attributed to it. It never sees the supervisor's routing
   state and cannot edit another specialist's results. An answer you cannot trace back to
   an agent is a multi-agent system you cannot debug.

Capability becomes a property of *who was dispatched* — decidable before anything runs and
visible in the plan the UI renders — rather than a property of the current step.

### When to use which

**Use `single`.** For three or four tools it is less machinery and strictly easier to
reason about. The supervisor exists here because the tool sets genuinely differ in blast
radius — one of them writes — and because routing lets a policy question skip the SQL path
entirely.

Scaling up the number of agents is not free: every handoff is a place for the state
contract to be violated, and a supervisor that routes badly is harder to debug than one
agent that plans badly, because the failure is now split across two decisions.

### Two bugs this surfaced, both now regression-tested

- **A specialist could not see its own earlier findings.** The analyst's calculator step
  computes from the SQL result it just produced, but LangGraph only commits state when the
  node returns — so mid-dispatch findings were invisible and the calculator failed every
  time. In-flight findings are now passed through explicitly.
- **`Finding` did not satisfy the resolver contract.** The late-binding resolvers are shared
  with the single agent and read `duration_ms`; omitting it raised `KeyError` during
  argument resolution — which happens *before* `interrupt()`, so the scribe's approval gate
  was silently skipped and the write never even asked. A missing key removed a human
  approval step: worth knowing that is the shape such a bug takes.

---

## Tracing

```bash
make trace     # two traced runs, rendered as span trees -> docs/TRACE.md
```

OpenTelemetry, **off by default** (`TRACE_EXPORTER=none`) — the no-op tracer makes every
`with span(...)` free, which is why no node needs an `if tracing_enabled` guard. Set
`file` for a JSONL span log you can render, or `otlp` to ship to any collector.

A rendered tree rather than a screenshot, deliberately: it is diffable, it needs no image
host, and anyone can regenerate it and check it matches. For an agent this tree *is* the
artifact — "why did that run take nine seconds and call search twice" is answered by the
shape of the traversal, not by reading interleaved logs.

```
Run 1 — read-only (search -> SQL -> calculator), no approval needed

agent.plan                 0.42ms   6.2% █                    planner=heuristic steps=3 tools=web_search,query_tickets,calculator
agent.act                  0.02ms   0.3%                      step_id=s1 tool=web_search requires_approval=False outcome=ready
tool.web_search            1.30ms  19.2% ████                 ok=True attempts=1
agent.reflect              0.01ms   0.1%                      steps_taken=1 total_steps=3 failed_steps=0 budget=8
agent.act                  0.01ms   0.1%                      step_id=s2 tool=query_tickets requires_approval=False outcome=ready
tool.query_tickets         0.53ms   7.8% ██                   ok=True attempts=1
agent.reflect              0.01ms   0.1%                      steps_taken=2 total_steps=3 failed_steps=0 budget=8
agent.act                  0.02ms   0.3%                      step_id=s3 tool=calculator requires_approval=False outcome=ready
tool.calculator            0.21ms   3.1% █                    ok=True attempts=1
agent.reflect              0.01ms   0.1%                      steps_taken=3 total_steps=3 failed_steps=0 budget=8
agent.answer               0.01ms   0.1%                      successes=3 failures=0 halt_reason=completed

11 spans, 6.78ms wall clock

Run 2 — write action: pauses for approval, then resumes

agent.plan                 0.02ms   0.3%                      planner=heuristic steps=3 tools=web_search,query_tickets,save_report
agent.act                  0.01ms   0.2%                      step_id=s1 tool=web_search requires_approval=False outcome=ready
tool.web_search            0.48ms   7.7% ██                   ok=True attempts=1
agent.reflect              0.01ms   0.2%                      steps_taken=1 total_steps=3 failed_steps=0 budget=8
agent.act                  0.08ms   1.3%                      step_id=s2 tool=query_tickets requires_approval=False outcome=ready
tool.query_tickets         0.20ms   3.2% █                    ok=True attempts=1
agent.reflect              0.01ms   0.2%                      steps_taken=2 total_steps=3 failed_steps=0 budget=8
agent.act                  0.02ms   0.3%                      step_id=s3 tool=save_report requires_approval=True outcome=awaiting_approval
agent.act                  0.02ms   0.3%                      step_id=s3 tool=save_report requires_approval=True outcome=awaiting_approval
agent.approval             0.01ms   0.2%                      step_id=s3 tool=save_report approved=True
tool.save_report           0.50ms   8.1% ██                   ok=True attempts=1
agent.reflect              0.01ms   0.2%                      steps_taken=3 total_steps=3 failed_steps=0 budget=8
agent.answer               0.01ms   0.2%                      successes=3 failures=0 halt_reason=completed

13 spans, 6.21ms wall clock
```

**Three things worth reading off that output:**

- **The budget is visible on every hop.** `agent.reflect` carries
  `steps_taken=2 total_steps=3 budget=8`, so a run that halted early tells you *why* from
  the trace rather than from a log line you have to go find.
- **Retries are visible after the fact.** `tool.* attempts=1` is the only place a silent
  retry shows up once the request is over. An `attempts=3` on a tool that still returned
  `ok=True` is the signature of a flaky dependency that no error log would catch.
- **The approval pause appears as two `agent.act` spans, both `awaiting_approval`.** That
  is not a bug in the instrumentation — it is the LangGraph semantics being honest.
  `interrupt()` unwinds the stack, so the node genuinely re-executes from the top on
  resume, and the trace shows two executions because there were two. It is the same
  property that forces everything before `interrupt()` to be idempotent.

### On LangSmith

`TRACE_EXPORTER=otlp` pointed at LangSmith's OTLP endpoint with an `x-api-key` header, with
the config in `.env.example`.

The identical exporter path **was verified in the sibling
[`rag-service`](https://github.com/jayu1023/rag-service) repo** — `make langsmith` there
emits a trace and inspects the export result, and it returned `PASS: 2/2 export(s)
accepted (HTTP 2xx)` on 2026-11-20. `app/tracing.py` is the same module in both repos, so
the OTLP wiring is shared and proven.

**What has not been done here:** that check has not been run against *this* service's
spans, so no `agent.*` or `tool.*` span has been seen by a LangSmith collector. The
exporter is the same; the specific run is not. Point `TRACE_EXPORTER=otlp` with a key at
it and the agent traces go to LangSmith — expected to work, not yet demonstrated.

---

## Tests

```
make test        # 132 python tests
make test-dart   #  8 dart wire-compatibility tests
```

What they actually cover, rather than a coverage number:

- **`test_events.py`** — every type tag, enum spelling and snake_case key asserted
  against the shapes read out of `agent_kit`'s source. Including that control
  frames never look like agent events, because a fifth `type` throws in Dart.
- **`test_tools.py`** — the sandboxes: `eval`-style injection, SQL writes,
  statement smuggling, `ATTACH DATABASE`, DoS exponents, path traversal.
- **`test_guards.py`** — transient retried, permanent *not* retried (asserted on
  the call count), timeout honoured, `run_tool` never raises, and four 0.3s tools
  finishing in under 1s to prove they do not serialise on the event loop.
- **`test_planner.py`** — intent routing, the late-binding resolvers, and that a
  hallucinated tool or a schema violation is rejected at planning time.
- **`test_graph.py`** — plan emitted before the first tool call, approval pauses
  *before* executing, rejection keeps prior work, the step budget halts a long
  plan and still answers, a failing step does not abort the run.
- **`test_api.py`** — SSE framing, the two-request approval flow, double-approve
  is a 409.

---

## API

```bash
POST /runs                    {"question": "...", "mode": "single|supervisor"}  → SSE
POST /runs/{id}/approve       {"approved": true, "note": "..."}          → SSE
GET  /runs/{id}               plan, answer, halt_reason, awaiting_approval
GET  /tools                   names, schemas, requires_approval
GET  /specialists             the supervisor roster: tools, budgets, who can write
GET  /health
```

Two SSE event names, on purpose:

```
event: agent     → exactly what AgentEvent.fromJson accepts (4 types)
event: control   → run_started | approval_required | done | error
```

Control frames carry `kind`, not `type`, and ride a different event name,
because the Dart union has four members and a fifth would throw. A client that
ignores unknown SSE event names keeps working either way.

---

## Honest status

A working reference implementation with a tested protocol contract. Specifically
**not** a production deployment:

- **`docker compose up` is written but has not been run** — no Docker on the
  build machine. The Dockerfile is two-stage and runs as a non-root user; it is
  unverified. Everything else below was executed: 132 Python tests, 8 Dart tests,
  the MCP server listing and calling tools, and the full approval flow over real
  HTTP against `uvicorn`.
- **No LLM was used in any verified run.** `PLANNER=llm` is implemented and
  validated against the tool registry, but every number and every test here comes
  from the heuristic planner. The planner is rule-based, not clever — it
  recognises five intents and falls back to search-only.
- **`web_search` searches a bundled corpus by default, not the internet.** The
  Tavily provider is implemented and untested against the live API.
- **The final answer is extractive, not generated.** `answer()` concatenates tool
  results and citations rather than calling an LLM, because this node's job in a
  reference implementation is to prove the results survived the graph intact.
  Swapping in a generation call does not touch the event protocol.
- **`MemorySaver` keeps runs in process memory.** So an approval must reach the
  same worker, a restart loses paused runs, and this scales to exactly one
  replica. The Postgres checkpointer is the fix and is a config change, not a
  redesign.
- **No auth, no rate limiting, no multi-tenancy, no per-user tool permissions.**
- **The supervisor's router is keyword-based**, like the single agent's planner, and for
  the same reason: determinism makes the tests assert exact rosters and lets the demo run
  with no API key. An LLM router drops in behind the same signature. Specialists also run
  sequentially — there is no parallel dispatch, and no specialist can call another.
- **Tracing has only been exercised with the file exporter.** The OTLP path (and therefore
  the LangSmith path) is implemented and unrun — no collector and no key were used.
- **No screen recording of the Flutter UI.** The wire contract is tested against
  the real package, which is the part that can actually break; a GIF of widgets
  rendering is not evidence of anything the tests do not already cover.

## Next

- Postgres checkpointer, so paused runs survive a restart and more than one replica
- Token-level answer streaming (the protocol already has `thinking.is_streaming`)
- Per-run tool permissions, so approval is policy rather than a hardcoded set
- Parallel dispatch for independent specialists, with a shared deadline
- A tool-level eval set: given a question, did it pick the right tool and arguments

## License

BSD-3-Clause.