Skip to main content
Glama
veriloop

veriloop

by arvindcr4

veriloop

Verifier-scored agent runtime. A minimal ReAct loop you can fully inspect: every step is budgeted, scored by a verifier function, and written to a replayable JSONL trace. The tools are exposed once over MCP so they plug into any framework. No LangChain, no LangGraph, no CrewAI — just the primitives, typed.

ci

The problem

Agent demos hide their failure modes. A polished screencast shows the run that worked; it does not show the runs that looped, hallucinated a tool name, sent malformed arguments, or burned forty steps on a two-step task. Framework-first builds make this worse: when the loop belongs to someone else's abstraction, you can't see why a run went wrong, only that it did.

The position of this repo: the first agent worth building is a minimal loop you fully understand. Frameworks earn their place later, for durable stateful orchestration — not as a substitute for knowing what your agent actually does at each step.

veriloop makes every decision inspectable and scoreable:

  • Hard step budget — the loop terminates, provably (there's a test for it).

  • Verified tool calls — args are schema-checked before execution; bad calls are blocked, logged, and fed back so the model can self-correct.

  • A verifier score on every step — verifier functions return scored judgments (reward-function discipline), logged alongside the step, not bolted on after.

  • Full JSONL trace — every run is a replayable, diffable artifact. Traces and scores are first-class outputs, not debug noise.

  • Kill switch — a threading.Event stops the run cleanly from outside.

  • Tools over MCP — the same three-tool registry the loop executes is served over MCP stdio, so any MCP-capable client reuses it unchanged.

Related MCP server: MCP Demo

Approach

The loop is the classic ReAct cycle — think → act → observe — kept deliberately small (a few hundred lines of typed Python across loop.py, verifiers.py, trace.py, tools.py, llm.py):

  1. Think — an LLMClient (protocol; any model plugs in — a deterministic FakeLLM ships for tests/CI, an OpenRouterClient with per-call token/cost accounting and a hard cost cap ships for live runs) looks at the task and the step history and emits a Decision: a thought plus exactly one of tool_call / answer.

  2. Verify — pre-act verifiers judge the decision. The built-in SchemaVerifier validates tool args against the tool's pydantic schema; a failing judgment blocks execution.

  3. Act — the tool runs (calculator, sandboxed file_read, offline web_fetch stub), or the blocked call becomes an error observation.

  4. Observe & score — post-act verifiers score the step (the built-in BudgetVerifier scores remaining headroom). Decision + judgments + observation are appended to the JSONL trace as one step record.

  5. Repeat until the model answers, the budget is exhausted, the kill switch fires, or too many consecutive failures trip the fallback stop.

Retry policy is budget-honest: a rejected step consumes a step and its error is fed back as the observation — there are no free retries, so traces never lie about cost.

Verifiers are the extension point: implement the Verifier protocol (a name, a phase, and judge(ctx) -> Judgment) to add task-specific checks, and their scores land in the same trace.

Evaluation

The eval plan is 30 cases; 10 seed cases are committed in eval/cases.seed.jsonl (arithmetic, sandboxed file tasks, and recovery/adversarial cases: malformed args, unknown tools, sandbox escapes, budget traps). The remaining 20 follow the same schema: 10 more multi-step arithmetic/file compositions and 10 more adversarial cases.

Metrics, measured by eval/harness.py (scripted) and eval/run_live.py (live):

Metric

Definition

Scripted (CI)

Live: openai/gpt-4o-mini (2026-07-19)

Task completion rate

runs ending completed

9/10

9/10

Expected-outcome pass rate

all of a case's checks pass

10/10

5/10

Mean steps-to-completion

steps used, completed runs only

2.33

1.89

Mean verifier score

all judgments, all steps

0.883

0.945

Verifier-blocked steps

tool calls blocked pre-execution

2 (scripted by design)

0

Budget-exhaustion rate

runs ending budget_exhausted

1/10 (by design)

0/10

Total cost

from OpenRouter usage accounting

$0

$0.0016 (8,066 in / 666 out tokens)

Honesty note: the live column is one run of the 10-case seed set — single repetition, temperature 0, max 512 tokens/call, ≤6 steps/case, via OpenRouter. The raw artifacts for that exact run — per-case JSONL traces, summary.json with per-case token/cost accounting, and a failure analysis — are committed at eval/results/live-gpt4omini-2026-07-19/. The 5/10 live pass rate is signal, not embarrassment: the misses are adversarial cases where the model behaved reasonably (refused a sandbox-escape without calling the tool, declined an infinite-loop task, never produced the malformed calls the recovery cases script for) — dissected case-by-case in results.md. Cases were not tuned to make the model pass.

Scripted mode exercises the runtime's plumbing deterministically (CI reruns it; output goes to the gitignored eval/results.md); live mode measures model capability, and its numbers come only from committed run artifacts.

Sample trace

Illustrative format example — not output from a recorded run. Generate a real one with uv run python -m veriloop (written to traces/demo.jsonl).

{"type":"run_start","task":"What is 17 * 23?","max_steps":8,"ts":1789700000.01}
{"type":"step","step":0,"decision":{"thought":"Arithmetic; use the calculator.","tool_call":{"tool":"calculator","args":{"expression":"17 * 23"}},"answer":null},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"args match calculator schema"},{"verifier":"budget","phase":"post_act","score":1.0,"passed":true,"reason":"step 1/8; headroom 1.00"}],"observation":{"ok":true,"content":"391"},"ts":1789700000.02}
{"type":"step","step":1,"decision":{"thought":"The observation has the product.","tool_call":null,"answer":"17 * 23 = 391"},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"final answer step; no tool call to validate"},{"verifier":"budget","phase":"post_act","score":0.88,"passed":true,"reason":"step 2/8; headroom 0.88"}],"observation":null,"ts":1789700000.03}
{"type":"run_end","status":"completed","answer":"17 * 23 = 391","steps_used":2,"mean_score":0.97,"error":null,"ts":1789700000.03}

Run it

uv run python -m veriloop              # scripted demo; prints the trace it wrote
uv run pytest                          # real tests: budget, verifiers, trace round-trip, tools
uv run ruff check .                    # lint
uv run python eval/harness.py          # scripted eval; writes eval/results.md
OPENROUTER_API_KEY=... uv run python eval/run_live.py   # live eval (gpt-4o-mini); writes eval/results/live-*/

Docker one-liner:

docker build -t veriloop . && docker run --rm veriloop

MCP server

The tool registry is served over MCP stdio:

uv run python -m veriloop.mcp_server

Plug it into any MCP client — e.g. Claude Code:

{
  "mcpServers": {
    "veriloop": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/veriloop", "python", "-m", "veriloop.mcp_server"]
    }
  }
}

Same tools, same schemas, zero duplication: the loop and the MCP server share one ToolRegistry.

Limitations

  • Scripted CI numbers measure plumbing, not intelligence. FakeLLM replays fixed decision paths. The live column is measured but thin: one model, one run, 10 cases — not a benchmark.

  • Verifiers are heuristic functions, not learned reward models — dense signal, but only as good as the checks you write.

  • web_fetch is an offline stub by default; enabling live fetches without an allowlist is an SSRF risk (flagged in tools.py).

  • Single-threaded, one tool call per step — no parallel tool fan-out, no streaming.

  • Traces are replayable but the loop is not resumable — replay reconstructs what happened; it does not restart a run mid-flight.

  • The sandbox fences file_read only; the calculator and fetch stub have their own guards, but there is no process-level isolation.

Layout

src/veriloop/     loop.py  verifiers.py  trace.py  tools.py  llm.py  mcp_server.py
eval/             cases.seed.jsonl  harness.py  sandbox/
tests/            budget, verifier, trace round-trip, tool-safety tests
docs/             DECISIONS.md
ARCHITECTURE.md   state machine, verifier contract, MCP layering

MIT — see LICENSE.

Available Tools

3 tools
calculatorA

Evaluate a pure-arithmetic expression (+ - * / // % **, parentheses)

Inputs: expression (required). Returns: result

file_readA

Read a UTF-8 text file from inside the sandbox directory

Inputs: path (required). Returns: result

web_fetchA

Fetch a URL over HTTP GET (offline stub by default)

Inputs: url (required). Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool performs a completely different function: arithmetic evaluation, file reading, and web fetching. There is no overlap, so an agent can easily select the right tool.

    Naming Consistency4/5

    Two tools follow verb_noun pattern (file_read, web_fetch), but 'calculator' is a noun, which is a minor deviation. Still, the names are clear and predictable overall.

    Tool Count5/5

    With only 3 tools, the server is well-scoped as a small utility set. Each tool serves a distinct purpose, and the count feels appropriate for the intended lightweight functionality.

    Completeness4/5

    The tools cover basic arithmetic, file reading, and web fetching, but missing complementary operations like file_write or web_post create minor gaps. These are workable for typical sandbox use cases.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Provides math and weather tools accessible via LangGraph agent using MCP protocol with stdio and streamable HTTP transports.
      1
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      A model-agnostic MCP server exposing example tools (add1, multiply2, greet) for learning purposes, working with any LLM through stdio transport.
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      Serves a single set of typed, validated tool definitions to MCP clients over stdio, letting them perform real actions through one controlled executor. The same definitions power a speech-driven OpenAI tool-calling agent, so voice and MCP clients share identical, validated capabilities.
      1
      MIT