구조화되고 샌드박스 처리된 코딩 에이전트용 검증 피드백.
에이전트의 pytest 셸 호출을 에이전트 내부 루프에 맞게 설계된 것으로 대체하는 MCP 서버: 영향받는 테스트를 선택하고, 격리된 환경에서 실행하며, 40,000 토큰의 원시 러너 출력 대신 간결한 타입화된 판정(verdict) 을 반환합니다. 또한 실패 지문(fingerprint) 을 제공하여 에이전트가 실패가 자신의 회귀인지 아니면 원래부터 깨져 있었는지 알 수 있게 합니다.
raw pytest dump: ~40,000 tokens, unstructured, run un-sandboxed on your machine
verdict: ~400 tokens, typed JSON, run in a rootless container, with memory에이전트 코딩에서 가장 빈번한 도구 호출은 검증이지만, 가장 구조화되지 않은 작업이기도 합니다. 에이전트는 모듈 하나가 변경되었을 때 전체 스위트를 다시 실행하고, ANSI 코드가 포함된 트레이스백을 파싱하는 데 컨텍스트를 소모하며, 임의의 코드를 사용자 머신에서 직접 실행하고, 기존의 깨진 상태를 자신의 회귀로 잘못 진단한 다음(깨지지 않은 코드를 "수정"하며) 일상적으로 처리합니다. verdict는 이 네 가지 문제를 모두 해결합니다.
Related MCP server: Checkride
도구 | 기능 |
| 작업 트리 diff의 영향을 받는 테스트를 선택하고(rimp를 통한 정적 임포트 그래프), podman/docker로 작업 트리를 읽기 전용으로 마운트하여 실행하며, 지문과 |
| 요청 시 하나의 실패에 대한 전체 트레이스백 — 대량은 요약에 포함되지 않습니다 |
| 처음 발견 / 마지막 발견 / 발견 횟수 — 회귀 vs. 오래된 깨짐 |
| 린트 및 타입 검사를 동일한 verdict 스키마로 정규화합니다 |
모든 실패에는 지문(fingerprint) 이 포함됩니다: 정규화된 실패 시그니처의 안정적인 해시(변동 토큰 — 주소, 임시 경로, ID, 지속 시간 — 은 축약됨). 동일한 논리적 실패는 실행과 리팩터링을 거쳐도 동일한 지문을 갖습니다. 지문이 verdict에 기억력을 부여합니다.
설치 단계가 필요 없습니다 — uvx가 첫 사용 시 가져옵니다. (또는 영구 verdict-mcp 명령을 위해 uv tool install verdict-mcp / pip install verdict-mcp 사용.)
Claude Code — 프로젝트 루트의 .mcp.json:
{
"mcpServers": {
"verdict": {
"command": "uvx",
"args": ["verdict-mcp"],
"env": { "VERDICT_PROJECT": "." }
}
}
}Cursor — .cursor/mcp.json에 동일한 형식.
저장소 루트의 선택적 verdict.toml:
[project]
packages = ["your_package"] # for impact selection (auto-guessed if omitted)
[runner]
image = "ghcr.io/you/yourproj-test" # prebuilt env with your deps
setup_cmd = "pip install -e .[test]" # or install on the fly (runs with network; tests don't)
# prefer = "local" # escape hatch if you have no container runtime
[limits]
max_failures = 10에이전트 없이 시도:
cd examples/demo_project
VERDICT_PROJECT=. verdict-mcp # then connect any MCP client, or use the MCP inspector검사는 임시 컨테이너에서 실행됩니다(podman 우선, docker 대체): 작업 트리는 /src에 읽기 전용으로 마운트되고, 컨테이너 내부의 쓰기 가능한 /work로 복사되며, 검사 실행 중에는 --network=none이 적용됩니다. 호스트 환경은 테스트 실행으로 절대 변경되지 않습니다. setup_cmd가 구성된 경우 해당 단계는 검사 전에 네트워크와 함께 실행됩니다. 더 엄격한 태세를 위해 사전 빌드된 이미지를 선호하십시오. 컨테이너 런타임이 없으면 명시적 prefer = "local" 대체가 작업 트리의 임시 복사본에 대해 검사를 실행합니다(여전히 제자리에서 실행되지 않음). 전체 위협 모델과 알려진 제한 사항은 SECURITY.md를 참조하십시오.
문제 해결: verdict가 container engine 'podman' could not start the check라고 표시하면 제안된 podman pull <image>를 직접 실행하십시오 — 엔진 자체의 오류가 답입니다. macOS에서 알려진 함정 하나: ~/.docker/config.json의 "credsStore": "gcloud" 줄로 인해 podman이 docker.io를 포함한 모든 레지스트리에 대해 gcloud 자격 증명 헬퍼를 호출합니다. 만료된 gcloud 로그인은 모든 풀을 깨뜨립니다. gcloud auth login으로 해결하거나 해당 줄을 제거하십시오.
영향 선택은 정적 임포트 그래프를 사용합니다 — 설계상 근사치입니다. 동적 임포트, 이름 기반 픽스처 해석, 데이터 기반 테스트는 놓칠 수 있습니다. verify(scope="all")은 항상 사용 가능하며, verdict는 대체가 발생할 때마다 selection_note에 명시합니다.
현재는 Python/pytest만 지원하며 ruff/mypy가 추가됩니다. 어댑터 인터페이스는 작고 문서화되어 있습니다 — vitest 및 go test -json 어댑터가 가장 필요한 기여입니다 (CONTRIBUTING.md).
플레이크 감지 및 커버리지 맵 기반 선택은 v0.2입니다 (로드맵).
v0.2: 커버리지 기반 영향 맵(정밀 선택), 지문 교대를 통한 플레이크 감지, devcontainer.json 지원, (트리 해시, 검사, 이미지 다이제스트)를 키로 하는 결과 캐시. 이후: vitest/jest, go test, cargo test 어댑터; 저장소별 verdict 데몬 모드.
Apache-2.0
Full traceback and output for one failing check from a previous verify run.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | No | optional; defaults to the most recent run containing this check. | |
| check_id | Yes | the failure's check_id, e.g. 'pytest::tests/test_x.py::test_y'. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden. It reveals the return content (traceback and output) and the source (previous verify run), which is helpful. Yet it does not explicitly declare read-only behavior or lack of side effects, nor what happens if the check_id is not found. This is a moderate gap given the absence of annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficiently worded sentence that front-loads the core purpose and key constraint (from a previous verify run). There is zero wasted text, making it easy for an agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the output schema is present and parameters are fully described, the description provides sufficient context for the tool's purpose and usage. It mentions the prerequisite of a prior verify run and the focus on a specific check. Minor omissions like error handling or explicit read-only status are not critical for basic invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both run_id and check_id already documented, including the default behavior of run_id and an example for check_id. The description adds no additional semantic value beyond the schema, such as clarifying the format of output or the relationship between the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description precisely states the tool's function: retrieving the full traceback and output for a single failing check from a prior verify run. This clearly distinguishes it from siblings like 'verify' (which runs checks) and 'history' (which lists runs), as it focuses on failure details.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides context by specifying the tool is for a 'previous verify run', implying it should be used after a verification execution. However, it does not explicitly name alternatives or state when not to use it, leaving some room for inference about choosing it over history or re-running verify.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
When was this failure fingerprint first/last seen, and how many times?
Use this to distinguish a regression you introduced (unknown fingerprint) from long-standing breakage (fingerprint seen across many runs/commits).
| Name | Required | Description | Default |
|---|---|---|---|
| fingerprint | Yes |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that the tool returns historical information (first/last seen, count) and implies a read-only query, but it does not explicitly state read-only status, error handling, or any side effects. For a simple query tool, this is acceptable but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with zero filler. The first sentence directly states the query, and the second explains the use case. It is front-loaded with the core purpose and efficiently conveys essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one required parameter, output schema exists) and the description's coverage of purpose and usage, it is largely complete. It could explicitly mention that the tool is read-only, but that omission is minor given the lack of annotations and the straightforward nature of the operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema defines 'fingerprint' as a plain string with no description (0% coverage). The tool description adds meaning by referring to it as a 'failure fingerprint' and using it in context, clarifying that it identifies a specific failure pattern. This compensates for the lack of schema documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states what the tool returns (first/last seen, count) and identifies the resource (failure fingerprint). It also gives a specific use case—distinguishing new regressions from long-standing breakage—which clearly differentiates it from siblings like verify or explain_failure.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear when-to-use: when you need to know if a fingerprint is unknown (possible regression) or has been seen across runs/commits (long-standing breakage). It gives a concrete scenario but does not explicitly name alternatives or state when not to use the tool, so it falls just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run lint/type checks ('ruff', 'mypy') in the sandbox; same verdict shape as verify.
| Name | Required | Description | Default |
|---|---|---|---|
| checks | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the behavioral disclosure burden. It states execution is 'in the sandbox' (an environmental detail) and references the verdict shape of 'verify', but it does not disclose whether the tool is read-only, whether it modifies anything, what side effects exist (if any), or any authorization requirements. For a tool with no annotation coverage, this is a significant gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with the key action, and contains zero filler. Every element ('run lint/type checks', specific tools, sandbox, shape reference) earns its place. Excellent conciseness and structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the tool is simple (one optional param, output schema present), the description leaves critical gaps: it does not explain the 'checks' parameter, relies on the reader knowing 'verify's verdict shape' (which may be defined elsewhere), and does not clarify whether all checks run or a subset can be requested. An agent may not be able to call the tool correctly without additional inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, 'checks', is entirely undocumented in the schema (0% coverage). The description mentions the tools 'ruff' and 'mypy' but never explains that 'checks' likely selects or filters which of these to run, nor does it describe the expected values. The description adds no meaning beyond the schema, leaving the parameter's role ambiguous.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('run lint/type checks') and specifies the exact tools involved ('ruff', 'mypy'), and notes the 'same verdict shape as verify' to differentiate from at least one sibling. It gives a specific, unambiguous purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies a context (running lint/type checks) and points to a related tool ('verify') for shape, but it does not explicitly state when to use this tool over 'verify' or the other siblings. There is no direct 'when not to use' guidance, though the intent is somewhat inferable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Verify the current working-tree changes by running affected pytest tests in a sandbox.
| Name | Required | Description | Default |
|---|---|---|---|
| base | No | git ref to diff against (default: HEAD, i.e. uncommitted changes). | |
| scope | No | None for impact-based selection (default), 'all' for the full suite, or a path like 'tests/test_api.py' to run one file. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that tests run 'in a sandbox', implying isolation, which is useful. However, it does not mention whether the tool modifies files, requires network access, or has any side effects on the workspace. Since it does not contradict annotations (none exist) and provides one meaningful behavioral detail, a middle score is appropriate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence that immediately communicates the action, target, and method. Every word contributes value, with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is relatively simple: two optional params, no required fields, and an output schema exists (so return format need not be explained). The description covers the core behavior adequately. While it could elaborate on how 'base' and 'scope' interplay, the schema already documents those, so the description does not need to repeat them. Minor gap: it does not state whether the tool is read-only, but the sandbox hint mitigates that.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% – both 'base' and 'scope' have descriptions in the schema. The tool description adds no additional parameter semantics beyond what the schema already provides. Per the baseline rule for high schema coverage, a score of 3 is given.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the specific action ('Verify'), the target ('current working-tree changes'), and the method ('running affected pytest tests in a sandbox'). It distinguishes itself from siblings: explain_failure is for diagnosing failures, history is for past actions, and run_checks is a broader term that might overlap, but this tool is explicitly about pytest verification, making its niche unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for verifying changes before committing, but it does not explicitly state when to prefer this over run_checks or provide exclusion criteria. There is no mention of alternatives or conditions that would make this tool inappropriate, leaving the routing decision to the agent's inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0explain_failurehistoryrun_checksverifyScored across 4 tools
Each tool has a clearly distinct purpose: verify runs unit tests, run_checks runs static checks, explain_failure provides failure details, and history shows recurrence patterns. No overlap or ambiguity exists between tools.
Most tools follow a verb-first convention (verify, explain_failure, run_checks), but 'history' is a noun used as a query command. The style is otherwise consistent with snake_case and lowercase, making it readable, but the mix prevents a perfect score.
With only 4 tools, the server is tightly scoped to verification and failure analysis. Each tool earns its place, covering the necessary actions without bloat or redundancy.
The tool surface fully covers the core workflow: running tests/lint checks, explaining failures, and checking historical recurrence. There are no obvious missing operations for this domain, and the lifecycle is complete for its stated purpose.
Build, test and cache insights for Xcode, Gradle and Bazel projects, for coding agents.
QA platform for agents: coverage signals, in-repo test plans, verified tests and release governance.
Change-aware CI validation and affected-test guidance for coding agents.
Pre-execution governance for AI agents. Deterministic PASS/FAIL/REVIEW verdicts, replayable proof.
