# veriloop

**Verifier-scored agent runtime.** A minimal ReAct loop you can fully inspect: every step is budgeted, scored by a verifier function, and written to a replayable JSONL trace. The tools are exposed once over MCP so they plug into any framework. No LangChain, no LangGraph, no CrewAI — just the primitives, typed.

[![ci](https://github.com/arvindcr4/veriloop/actions/workflows/ci.yml/badge.svg)](https://github.com/arvindcr4/veriloop/actions/workflows/ci.yml)

## The problem

Agent demos hide their failure modes. A polished screencast shows the run that worked; it does not show the runs that looped, hallucinated a tool name, sent malformed arguments, or burned forty steps on a two-step task. Framework-first builds make this worse: when the loop belongs to someone else's abstraction, you can't see *why* a run went wrong, only *that* it did.

The position of this repo: **the first agent worth building is a minimal loop you fully understand.** Frameworks earn their place later, for durable stateful orchestration — not as a substitute for knowing what your agent actually does at each step.

veriloop makes every decision inspectable and scoreable:

- **Hard step budget** — the loop terminates, provably (there's a test for it).
- **Verified tool calls** — args are schema-checked *before* execution; bad calls are blocked, logged, and fed back so the model can self-correct.
- **A verifier score on every step** — verifier functions return scored judgments (reward-function discipline), logged alongside the step, not bolted on after.
- **Full JSONL trace** — every run is a replayable, diffable artifact. Traces and scores are first-class outputs, not debug noise.
- **Kill switch** — a `threading.Event` stops the run cleanly from outside.
- **Tools over MCP** — the same three-tool registry the loop executes is served over MCP stdio, so any MCP-capable client reuses it unchanged.

## Approach

The loop is the classic ReAct cycle — think → act → observe — kept deliberately small (a few hundred lines of typed Python across `loop.py`, `verifiers.py`, `trace.py`, `tools.py`, `llm.py`):

1. **Think** — an `LLMClient` (protocol; any model plugs in — a deterministic `FakeLLM` ships for tests/CI, an `OpenRouterClient` with per-call token/cost accounting and a hard cost cap ships for live runs) looks at the task and the step history and emits a `Decision`: a thought plus exactly one of `tool_call` / `answer`.
2. **Verify** — pre-act verifiers judge the decision. The built-in `SchemaVerifier` validates tool args against the tool's pydantic schema; a failing judgment blocks execution.
3. **Act** — the tool runs (calculator, sandboxed `file_read`, offline `web_fetch` stub), or the blocked call becomes an error observation.
4. **Observe & score** — post-act verifiers score the step (the built-in `BudgetVerifier` scores remaining headroom). Decision + judgments + observation are appended to the JSONL trace as one step record.
5. **Repeat** until the model answers, the budget is exhausted, the kill switch fires, or too many consecutive failures trip the fallback stop.

Retry policy is budget-honest: a rejected step *consumes* a step and its error is fed back as the observation — there are no free retries, so traces never lie about cost.

Verifiers are the extension point: implement the `Verifier` protocol (a `name`, a `phase`, and `judge(ctx) -> Judgment`) to add task-specific checks, and their scores land in the same trace.

## Evaluation

The eval plan is 30 cases; **10 seed cases are committed** in [`eval/cases.seed.jsonl`](eval/cases.seed.jsonl) (arithmetic, sandboxed file tasks, and recovery/adversarial cases: malformed args, unknown tools, sandbox escapes, budget traps). The remaining 20 follow the same schema: 10 more multi-step arithmetic/file compositions and 10 more adversarial cases.

Metrics, measured by [`eval/harness.py`](eval/harness.py) (scripted) and [`eval/run_live.py`](eval/run_live.py) (live):

| Metric | Definition | Scripted (CI) | Live: `openai/gpt-4o-mini` (2026-07-19) |
|---|---|---|---|
| Task completion rate | runs ending `completed` | 9/10 | 9/10 |
| Expected-outcome pass rate | all of a case's checks pass | 10/10 | 5/10 |
| Mean steps-to-completion | steps used, completed runs only | 2.33 | 1.89 |
| Mean verifier score | all judgments, all steps | 0.883 | 0.945 |
| Verifier-blocked steps | tool calls blocked pre-execution | 2 (scripted by design) | 0 |
| Budget-exhaustion rate | runs ending `budget_exhausted` | 1/10 (by design) | 0/10 |
| Total cost | from OpenRouter usage accounting | $0 | $0.0016 (8,066 in / 666 out tokens) |

**Honesty note:** the live column is one run of the 10-case seed set — single repetition, temperature 0, max 512 tokens/call, ≤6 steps/case, via OpenRouter. The raw artifacts for that exact run — per-case JSONL traces, `summary.json` with per-case token/cost accounting, and a failure analysis — are committed at [`eval/results/live-gpt4omini-2026-07-19/`](eval/results/live-gpt4omini-2026-07-19/). The 5/10 live pass rate is signal, not embarrassment: the misses are adversarial cases where the model behaved *reasonably* (refused a sandbox-escape without calling the tool, declined an infinite-loop task, never produced the malformed calls the recovery cases script for) — dissected case-by-case in [`results.md`](eval/results/live-gpt4omini-2026-07-19/results.md). Cases were not tuned to make the model pass.

Scripted mode exercises the runtime's plumbing deterministically (CI reruns it; output goes to the gitignored `eval/results.md`); live mode measures model capability, and its numbers come only from committed run artifacts.

## Sample trace

**Illustrative format example — not output from a recorded run.** Generate a real one with `uv run python -m veriloop` (written to `traces/demo.jsonl`).

```jsonl
{"type":"run_start","task":"What is 17 * 23?","max_steps":8,"ts":1789700000.01}
{"type":"step","step":0,"decision":{"thought":"Arithmetic; use the calculator.","tool_call":{"tool":"calculator","args":{"expression":"17 * 23"}},"answer":null},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"args match calculator schema"},{"verifier":"budget","phase":"post_act","score":1.0,"passed":true,"reason":"step 1/8; headroom 1.00"}],"observation":{"ok":true,"content":"391"},"ts":1789700000.02}
{"type":"step","step":1,"decision":{"thought":"The observation has the product.","tool_call":null,"answer":"17 * 23 = 391"},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"final answer step; no tool call to validate"},{"verifier":"budget","phase":"post_act","score":0.88,"passed":true,"reason":"step 2/8; headroom 0.88"}],"observation":null,"ts":1789700000.03}
{"type":"run_end","status":"completed","answer":"17 * 23 = 391","steps_used":2,"mean_score":0.97,"error":null,"ts":1789700000.03}
```

## Run it

```bash
uv run python -m veriloop              # scripted demo; prints the trace it wrote
uv run pytest                          # real tests: budget, verifiers, trace round-trip, tools
uv run ruff check .                    # lint
uv run python eval/harness.py          # scripted eval; writes eval/results.md
OPENROUTER_API_KEY=... uv run python eval/run_live.py   # live eval (gpt-4o-mini); writes eval/results/live-*/
```

Docker one-liner:

```bash
docker build -t veriloop . && docker run --rm veriloop
```

### MCP server

The tool registry is served over MCP stdio:

```bash
uv run python -m veriloop.mcp_server
```

Plug it into any MCP client — e.g. Claude Code:

```json
{
  "mcpServers": {
    "veriloop": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/veriloop", "python", "-m", "veriloop.mcp_server"]
    }
  }
}
```

Same tools, same schemas, zero duplication: the loop and the MCP server share one `ToolRegistry`.

## Limitations

- **Scripted CI numbers measure plumbing, not intelligence.** `FakeLLM` replays fixed decision paths. The live column is measured but thin: one model, one run, 10 cases — not a benchmark.
- **Verifiers are heuristic functions**, not learned reward models — dense signal, but only as good as the checks you write.
- **`web_fetch` is an offline stub by default**; enabling live fetches without an allowlist is an SSRF risk (flagged in `tools.py`).
- **Single-threaded, one tool call per step** — no parallel tool fan-out, no streaming.
- **Traces are replayable but the loop is not resumable** — replay reconstructs what happened; it does not restart a run mid-flight.
- **The sandbox fences `file_read` only**; the calculator and fetch stub have their own guards, but there is no process-level isolation.

## Layout

```
src/veriloop/     loop.py  verifiers.py  trace.py  tools.py  llm.py  mcp_server.py
eval/             cases.seed.jsonl  harness.py  sandbox/
tests/            budget, verifier, trace round-trip, tool-safety tests
docs/             DECISIONS.md
ARCHITECTURE.md   state machine, verifier contract, MCP layering
```

MIT — see [LICENSE](LICENSE).