Comentarios de verificación estructurados y en sandbox para agentes de codificación.
Un servidor MCP que reemplaza las invocaciones a pytest de tu agente con algo construido para el bucle interno del agente: pruebas seleccionadas por impacto, ejecutadas en un entorno aislado, que devuelven veredictos tipados compactos en lugar de 40,000 tokens de salida cruda del ejecutor — con huellas de fallo que le dicen al agente si un fallo es su regresión o si ya estaba roto desde antes.
raw pytest dump: ~40,000 tokens, unstructured, run un-sandboxed on your machine
verdict: ~400 tokens, typed JSON, run in a rootless container, with memoryLa llamada de herramienta de mayor frecuencia en la codificación agéntica es la verificación — y es la menos estructurada. Los agentes vuelven a ejecutar suites completas cuando un módulo cambió, queman contexto analizando tracebacks con códigos ANSI, ejecutan código arbitrario directamente en tu máquina y, con frecuencia, diagnostican erróneamente fallos preexistentes como su propia regresión (y luego "arreglan" código que no estaba roto). verdict soluciona los cuatro problemas.
Related MCP server: Checkride
Herramienta | Qué hace |
| Selecciona las pruebas afectadas por el diff de tu árbol de trabajo (grafo de importación estático mediante grimp), las ejecuta vía podman/docker con el árbol de trabajo montado de solo lectura, devuelve fallos tipados con huellas y una bandera |
| Traceback completo para un fallo, bajo demanda — el volumen nunca viaja en el resumen |
| Primera vez visto / última vez visto / veces visto — regresión vs. fallo de larga duración |
| Comprobaciones de lint y tipos, normalizadas en el mismo esquema de veredicto |
Cada fallo lleva una huella: un hash estable de la firma de fallo normalizada (tokens volátiles — direcciones, rutas tmp, ids, duraciones — colapsados). El mismo fallo lógico, la misma huella, a través de ejecuciones y refactorizaciones. Las huellas son lo que le dan memoria a verdict.
No se necesita paso de instalación — uvx lo obtiene en el primer uso. (O uv tool install verdict-mcp / pip install verdict-mcp para un comando verdict-mcp permanente.)
Claude Code — .mcp.json en la raíz de tu proyecto:
{
"mcpServers": {
"verdict": {
"command": "uvx",
"args": ["verdict-mcp"],
"env": { "VERDICT_PROJECT": "." }
}
}
}Cursor — la misma forma en .cursor/mcp.json.
Opcional verdict.toml en la raíz de tu repositorio:
[project]
packages = ["your_package"] # for impact selection (auto-guessed if omitted)
[runner]
image = "ghcr.io/you/yourproj-test" # prebuilt env with your deps
setup_cmd = "pip install -e .[test]" # or install on the fly (runs with network; tests don't)
# prefer = "local" # escape hatch if you have no container runtime
[limits]
max_failures = 10Pruébalo sin un agente:
cd examples/demo_project
VERDICT_PROJECT=. verdict-mcp # then connect any MCP client, or use the MCP inspectorLas comprobaciones se ejecutan en un contenedor efímero (podman preferido, docker como respaldo): el árbol de trabajo se monta de solo lectura en /src, se copia a un /work escribible dentro del contenedor, --network=none para la ejecución de la comprobación. Tu entorno anfitrión nunca es mutado por una ejecución de prueba. Si setup_cmd está configurado, ese paso se ejecuta con red antes de la comprobación; prefiere una imagen preconstruida para una postura más estricta. Sin runtime de contenedor → el respaldo explícito prefer = "local" ejecuta comprobaciones contra una copia temporal de tu árbol de trabajo (aún nunca en el lugar). Consulta SECURITY.md para el modelo de amenazas completo y las limitaciones conocidas.
Solución de problemas: si un veredicto dice container engine 'podman' could not start the check, ejecuta el sugerido podman pull <image> manualmente — el error del propio motor es la respuesta. Una trampa conocida en macOS: una línea "credsStore": "gcloud" en ~/.docker/config.json hace que podman llame al ayudante de credenciales de gcloud para cada registro, incluido docker.io; un inicio de sesión de gcloud caducado entonces rompe todas las descargas. Solución: gcloud auth login o elimina esa línea.
La selección de impacto utiliza el grafo de importación estático — aproximado por diseño. Las importaciones dinámicas, la resolución de fixtures por nombre y las pruebas basadas en datos pueden pasarse por alto; verify(scope="all") siempre está disponible y verdict lo indica en selection_note cada vez que recurre a ello.
Solo Python/pytest hoy, además de ruff/mypy. La interfaz de adaptadores es pequeña y está documentada — los adaptadores de vitest y go test -json son las contribuciones más solicitadas (CONTRIBUTING.md).
La detección de flake y la selección basada en mapa de cobertura son v0.2 (roadmap).
v0.2: mapas de impacto basados en cobertura (selección precisa), detección de flake mediante alternancia de huellas, soporte de devcontainer.json, caché de resultados clave en (hash de árbol, comprobación, digest de imagen). Más adelante: adaptadores de vitest/jest, go test, cargo test; modo daemon de verdict por repositorio.
Apache-2.0
Full traceback and output for one failing check from a previous verify run.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | No | optional; defaults to the most recent run containing this check. | |
| check_id | Yes | the failure's check_id, e.g. 'pytest::tests/test_x.py::test_y'. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden. It reveals the return content (traceback and output) and the source (previous verify run), which is helpful. Yet it does not explicitly declare read-only behavior or lack of side effects, nor what happens if the check_id is not found. This is a moderate gap given the absence of annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficiently worded sentence that front-loads the core purpose and key constraint (from a previous verify run). There is zero wasted text, making it easy for an agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the output schema is present and parameters are fully described, the description provides sufficient context for the tool's purpose and usage. It mentions the prerequisite of a prior verify run and the focus on a specific check. Minor omissions like error handling or explicit read-only status are not critical for basic invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both run_id and check_id already documented, including the default behavior of run_id and an example for check_id. The description adds no additional semantic value beyond the schema, such as clarifying the format of output or the relationship between the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description precisely states the tool's function: retrieving the full traceback and output for a single failing check from a prior verify run. This clearly distinguishes it from siblings like 'verify' (which runs checks) and 'history' (which lists runs), as it focuses on failure details.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides context by specifying the tool is for a 'previous verify run', implying it should be used after a verification execution. However, it does not explicitly name alternatives or state when not to use it, leaving some room for inference about choosing it over history or re-running verify.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
When was this failure fingerprint first/last seen, and how many times?
Use this to distinguish a regression you introduced (unknown fingerprint) from long-standing breakage (fingerprint seen across many runs/commits).
| Name | Required | Description | Default |
|---|---|---|---|
| fingerprint | Yes |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that the tool returns historical information (first/last seen, count) and implies a read-only query, but it does not explicitly state read-only status, error handling, or any side effects. For a simple query tool, this is acceptable but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with zero filler. The first sentence directly states the query, and the second explains the use case. It is front-loaded with the core purpose and efficiently conveys essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one required parameter, output schema exists) and the description's coverage of purpose and usage, it is largely complete. It could explicitly mention that the tool is read-only, but that omission is minor given the lack of annotations and the straightforward nature of the operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema defines 'fingerprint' as a plain string with no description (0% coverage). The tool description adds meaning by referring to it as a 'failure fingerprint' and using it in context, clarifying that it identifies a specific failure pattern. This compensates for the lack of schema documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states what the tool returns (first/last seen, count) and identifies the resource (failure fingerprint). It also gives a specific use case—distinguishing new regressions from long-standing breakage—which clearly differentiates it from siblings like verify or explain_failure.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear when-to-use: when you need to know if a fingerprint is unknown (possible regression) or has been seen across runs/commits (long-standing breakage). It gives a concrete scenario but does not explicitly name alternatives or state when not to use the tool, so it falls just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run lint/type checks ('ruff', 'mypy') in the sandbox; same verdict shape as verify.
| Name | Required | Description | Default |
|---|---|---|---|
| checks | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the behavioral disclosure burden. It states execution is 'in the sandbox' (an environmental detail) and references the verdict shape of 'verify', but it does not disclose whether the tool is read-only, whether it modifies anything, what side effects exist (if any), or any authorization requirements. For a tool with no annotation coverage, this is a significant gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with the key action, and contains zero filler. Every element ('run lint/type checks', specific tools, sandbox, shape reference) earns its place. Excellent conciseness and structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the tool is simple (one optional param, output schema present), the description leaves critical gaps: it does not explain the 'checks' parameter, relies on the reader knowing 'verify's verdict shape' (which may be defined elsewhere), and does not clarify whether all checks run or a subset can be requested. An agent may not be able to call the tool correctly without additional inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, 'checks', is entirely undocumented in the schema (0% coverage). The description mentions the tools 'ruff' and 'mypy' but never explains that 'checks' likely selects or filters which of these to run, nor does it describe the expected values. The description adds no meaning beyond the schema, leaving the parameter's role ambiguous.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('run lint/type checks') and specifies the exact tools involved ('ruff', 'mypy'), and notes the 'same verdict shape as verify' to differentiate from at least one sibling. It gives a specific, unambiguous purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies a context (running lint/type checks) and points to a related tool ('verify') for shape, but it does not explicitly state when to use this tool over 'verify' or the other siblings. There is no direct 'when not to use' guidance, though the intent is somewhat inferable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Verify the current working-tree changes by running affected pytest tests in a sandbox.
| Name | Required | Description | Default |
|---|---|---|---|
| base | No | git ref to diff against (default: HEAD, i.e. uncommitted changes). | |
| scope | No | None for impact-based selection (default), 'all' for the full suite, or a path like 'tests/test_api.py' to run one file. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that tests run 'in a sandbox', implying isolation, which is useful. However, it does not mention whether the tool modifies files, requires network access, or has any side effects on the workspace. Since it does not contradict annotations (none exist) and provides one meaningful behavioral detail, a middle score is appropriate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence that immediately communicates the action, target, and method. Every word contributes value, with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is relatively simple: two optional params, no required fields, and an output schema exists (so return format need not be explained). The description covers the core behavior adequately. While it could elaborate on how 'base' and 'scope' interplay, the schema already documents those, so the description does not need to repeat them. Minor gap: it does not state whether the tool is read-only, but the sandbox hint mitigates that.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% – both 'base' and 'scope' have descriptions in the schema. The tool description adds no additional parameter semantics beyond what the schema already provides. Per the baseline rule for high schema coverage, a score of 3 is given.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the specific action ('Verify'), the target ('current working-tree changes'), and the method ('running affected pytest tests in a sandbox'). It distinguishes itself from siblings: explain_failure is for diagnosing failures, history is for past actions, and run_checks is a broader term that might overlap, but this tool is explicitly about pytest verification, making its niche unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for verifying changes before committing, but it does not explicitly state when to prefer this over run_checks or provide exclusion criteria. There is no mention of alternatives or conditions that would make this tool inappropriate, leaving the routing decision to the agent's inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0explain_failurehistoryrun_checksverifyScored across 4 tools
Each tool has a clearly distinct purpose: verify runs unit tests, run_checks runs static checks, explain_failure provides failure details, and history shows recurrence patterns. No overlap or ambiguity exists between tools.
Most tools follow a verb-first convention (verify, explain_failure, run_checks), but 'history' is a noun used as a query command. The style is otherwise consistent with snake_case and lowercase, making it readable, but the mix prevents a perfect score.
With only 4 tools, the server is tightly scoped to verification and failure analysis. Each tool earns its place, covering the necessary actions without bloat or redundancy.
The tool surface fully covers the core workflow: running tests/lint checks, explaining failures, and checking historical recurrence. There are no obvious missing operations for this domain, and the lifecycle is complete for its stated purpose.
QA platform for agents: coverage signals, in-repo test plans, verified tests and release governance.
Change-aware CI validation and affected-test guidance for coding agents.
Pre-execution governance for AI agents. Deterministic PASS/FAIL/REVIEW verdicts, replayable proof.
Approved test intent, reviewed Playwright automation and run evidence, inside your editor.
