Skip to main content
Glama

verdict

https://github.com/user-attachments/assets/62923912-98aa-4246-8bc8-4adf55ae3ff8

CI PyPI Python License

Structured, sandboxed verification feedback for coding agents.

An MCP server that replaces your agent's pytest shell-outs with something built for the agent inner loop: impact-selected tests, run in an isolated environment, returning compact typed verdicts instead of 40,000 tokens of raw runner output — with failure fingerprints that tell the agent whether a failure is its regression or was broken all along.

raw pytest dump:  ~40,000 tokens, unstructured, run un-sandboxed on your machine
verdict:              ~400 tokens, typed JSON, run in a rootless container, with memory

Why

The highest-frequency tool call in agentic coding is verification — and it's the least structured. Agents re-run whole suites when one module changed, burn context parsing ANSI-coded tracebacks, run arbitrary code directly on your machine, and routinely misdiagnose pre-existing breakage as their own regression (then "fix" code that wasn't broken). verdict fixes all four.

Related MCP server: Checkride

Tools

Tool

What it does

verify(scope?, base?)

Selects tests affected by your working-tree diff (static import graph via grimp), runs them via podman/docker with the worktree mounted read-only, returns typed failures with fingerprints and a preexisting flag

explain_failure(check_id)

Full traceback for one failure, on demand — bulk never rides in the summary

history(fingerprint)

First seen / last seen / times seen — regression vs. long-standing breakage

run_checks(["ruff","mypy"])

Lint and type checks, normalized into the same verdict schema

Every failure carries a fingerprint: a stable hash of the normalized failure signature (volatile tokens — addresses, tmp paths, ids, durations — collapsed). Same logical failure, same fingerprint, across runs and refactors. Fingerprints are what give verdict memory.

Quickstart

No install step needed — uvx fetches it on first use. (Or uv tool install verdict-mcp / pip install verdict-mcp for a permanent verdict-mcp command.)

Claude Code — .mcp.json in your project root:

{
  "mcpServers": {
    "verdict": {
      "command": "uvx",
      "args": ["verdict-mcp"],
      "env": { "VERDICT_PROJECT": "." }
    }
  }
}

Cursor — same shape in .cursor/mcp.json.

Optional verdict.toml in your repo root:

[project]
packages = ["your_package"]          # for impact selection (auto-guessed if omitted)

[runner]
image = "ghcr.io/you/yourproj-test"  # prebuilt env with your deps
setup_cmd = "pip install -e .[test]" # or install on the fly (runs with network; tests don't)
# prefer = "local"                   # escape hatch if you have no container runtime

[limits]
max_failures = 10

Try it without an agent:

cd examples/demo_project
VERDICT_PROJECT=. verdict-mcp   # then connect any MCP client, or use the MCP inspector

Sandbox posture (v0.1)

Checks run in an ephemeral container (podman preferred, docker fallback): worktree mounted read-only at /src, copied to a writable /work inside the container, --network=none for the check run. Your host environment is never mutated by a test run. If setup_cmd is configured, that step runs with network before the check; prefer a prebuilt image for a tighter posture. No container runtime → explicit prefer = "local" fallback runs checks against a temp copy of your worktree (still never in place). See SECURITY.md for the full threat model and known limitations.

Troubleshooting: if a verdict says container engine 'podman' could not start the check, run the suggested podman pull <image> by hand — the engine's own error is the answer. One known trap on macOS: a "credsStore": "gcloud" line in ~/.docker/config.json makes podman call the gcloud credential helper for every registry, including docker.io; an expired gcloud login then breaks all pulls. Fix with gcloud auth login or remove that line.

Honest limitations

  • Impact selection uses the static import graph — approximate by design. Dynamic imports, fixture-by-name resolution, and data-driven tests can be missed; verify(scope="all") is always available and verdict says in selection_note whenever it falls back.

  • Python/pytest only today, plus ruff/mypy. The adapter interface is small and documented — vitest and go test -json adapters are the most-wanted contributions (CONTRIBUTING.md).

  • Flake detection and coverage-map-based selection are v0.2 (roadmap).

Roadmap

v0.2: coverage-based impact maps (precise selection), flake detection via fingerprint alternation, devcontainer.json support, result cache keyed on (tree hash, check, image digest). Later: vitest/jest, go test, cargo test adapters; per-repo verdict daemon mode.

License

Apache-2.0

Available Tools

4 tools
explain_failureA

Full traceback and output for one failing check from a previous verify run.

Inputs: check_id (required), run_id. Returns: object

historyA

When was this failure fingerprint first/last seen, and how many times?

Use this to distinguish a regression you introduced (unknown fingerprint) from long-standing breakage (fingerprint seen across many runs/commits).

Inputs: fingerprint (required). Returns: object

run_checksB

Run lint/type checks ('ruff', 'mypy') in the sandbox; same verdict shape as verify.

Inputs: checks. Returns: object

verifyA

Verify the current working-tree changes by running affected pytest tests in a sandbox.

Inputs: base, scope. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 4 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: verify runs unit tests, run_checks runs static checks, explain_failure provides failure details, and history shows recurrence patterns. No overlap or ambiguity exists between tools.

    Naming Consistency4/5

    Most tools follow a verb-first convention (verify, explain_failure, run_checks), but 'history' is a noun used as a query command. The style is otherwise consistent with snake_case and lowercase, making it readable, but the mix prevents a perfect score.

    Tool Count5/5

    With only 4 tools, the server is tightly scoped to verification and failure analysis. Each tool earns its place, covering the necessary actions without bloat or redundancy.

    Completeness5/5

    The tool surface fully covers the core workflow: running tests/lint checks, explaining failures, and checking historical recurrence. There are no obvious missing operations for this domain, and the lifecycle is complete for its stated purpose.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Provides an isolated workspace for testing candidate code, runs tests, and returns deterministic pass/fail verdicts. Enables automated grading of software engineering solutions by ensuring reproducible test runs.
      5
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Enables AI coding agents to run fast, sandboxed pre-flight validation—tests, type-checking, linting, security audits, Git safety scans, and fix suggestions—before committing or pushing code, with instant caching and optional Docker isolation.
      10
      12 npm
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Provides coding agents with visibility into test health through tools for flaky test detection, test quality linting, and LLM evaluation harness, enabling them to triage failures, review test quality, and check prompt changes for regressions.
      9
      Apache 2.0
    • A
      license
      A
      quality
      B
      maintenance
      Evidence-backed, read-only codebase diagnostics for coding agents: secrets, dependency, RLS, and source-impact audits with a verify loop and PR review verdicts. Never writes files or uses the network by default.
      7
      101 npm
      MIT