This server is an MCP-based deterministic verification engine for API integrations: it audits your code against known provider failure classes, reproduces failures against realistic sandboxes, proposes fixes, and proves fixes with measured before/after evidence and shareable receipts.
Guided triage: guide routes a symptom (payments, webhooks, auth, email, etc.) to a spec, workflow, and failure scenario; coach provides multi-turn conversational help for building integrations.
Bug discovery: find_bugs audits a local project against real-world API integration failure classes without needing a git remote.
Fix proposals: fix_bug returns a minimal git diff for a specific finding without modifying your files.
Measured proof: prove_fix runs the same failure against your buggy code before/after applying the diff, green only on a verified flip, and produces a receipt URL.
Sandbox execution: quickrun runs bundled provider workflows (Stripe, Twilio, etc.) in one call; run_workflow, run_all_workflows, list_workflows, and list_runs execute and inspect sandbox runs.
Reference verification: verify_behavior demonstrates a failure class on buggy vs fixed reference handlers, proving a known fix pattern holds.
Custom specs: import_spec ingests OpenAPI 3.x specs by URL or pasted content to create callable sandboxes; list_specs browses the available catalog.
Public receipts: submit_proof attaches real before/after request-response evidence from your own app to a shareable receipt page.
Privacy controls: excludes credential-shaped files before upload, refuses archives containing live credentials, and supports telemetry opt-out via FETCHSANDBOX_TELEMETRY=0.
Auth options: works without an account, supports browser sign-in, or FETCHSANDBOX_API_KEY for CI/headless usage.
Provides curated workflows for issue lifecycle management (create, comment, close, reopen).
Allows importing the Linear OpenAPI spec and running workflows.
Allows importing the Notion OpenAPI spec and running workflows.
Allows importing the Paddle Billing OpenAPI spec and running workflows.
Provides curated workflows for accepting payments, including creating customers, PaymentIntents, and verifying webhooks.
Provides curated workflows for sending SMS and verifying messages with realistic Twilio-formatted responses.
Also on Smithery, npm and the official MCP registry.
A deterministic verification engine for agents, as an MCP server for FetchSandbox.
Your agent writes an integration. This checks whether it actually works — against a sandbox that behaves like the real provider, including the failures: retried webhooks, declined cards, rate limits, auth errors.
When it finds a bug, it can propose a fix and then prove it: the same failure is run against your code before and after the diff. Green only if it reproduced first and stopped after. You get a receipt URL either way.
Same stdio command everywhere. npx fetches the current version, so there's
nothing to install.
{
"mcpServers": {
"fetchsandbox": {
"command": "npx",
"args": ["-y", "fetchsandbox-mcp@latest"]
}
}
}Client | File |
Claude Code |
|
Claude Desktop |
|
Cursor |
|
Zed |
|
Codex |
|
Restart the client afterwards. Anything else that speaks MCP takes the same command and args.
Related MCP server: agent-validator-mcp-server
Describe the problem the way you'd describe it to a colleague. You don't need to name a tool.
Customers are reporting more seats than they bought after a Paddle payment. Can you find out why?
The agent works through: route the symptom, reproduce it against the provider sandbox, read your code, get a fix, prove the fix on your code. Each step hands back what the next one needs.
One thing worth knowing, because it's easy to get backwards: prove_fix needs
the unfixed tree. Run it before you write the diff to disk, or there's no
bug left to reproduce and no proof to be had.
You don't need one to start. Install it, ask a question, and everything runs.
The first time a run produces something worth keeping — a receipt, or a set of findings — you'll get a short code and a link. Signing in takes about twenty seconds and does two things: the evidence behind your receipts stops being archived after 15 days, and the runs from that machine collect in one place. You'll be asked at most once a day, and never once you're signed in.
For CI, or anywhere a browser isn't available, set a key instead:
FETCHSANDBOX_API_KEY=fsk_...The key is written to ~/.fetchsandbox/credentials.json when you sign in from
an editor; the environment variable always wins.
Start with guide. It picks the right ones for what you asked.
Finding and fixing
Tool | What it does | Arguments |
| Routes a symptom to a spec, workflow and known failure class |
|
| Audits your project against known integration failure classes. No git remote needed — it reads the directory you point it at |
|
| Returns a |
|
| Runs the failure against your code before and after the diff. Green only on a measured flip |
|
Running the sandbox
Tool | What it does | Arguments |
| Runs a workflow against a bundled spec in one call. Returns |
|
| Shows a failure class on reference handlers — buggy vs fixed |
|
| Runs one workflow on a sandbox you already have |
|
| Runs several in one call |
|
| Workflows available for a spec |
|
| Past runs for a sandbox |
|
Bringing your own spec
Tool | What it does | Arguments |
| Specs already available |
|
| Ingests an OpenAPI 3.x spec by URL or pasted content. Returns a callable sandbox |
|
| Publishes a receipt for a run |
|
| Multi-turn help building an integration |
|
* = required.
find_bugs, fix_bug and prove_fix package the directory you point them at
and upload it for analysis. Worth saying plainly, because the previous wording
here implied the opposite.
Excluded before packing: .git, node_modules and build output, agent
instruction files, and anything credential-shaped — .env*, *.pem, *.key,
id_rsa*, *.tfstate, .npmrc, .aws, .ssh and more.
Then the archive is read back and refused if it still contains something
shaped like a live credential, wherever it lives and whatever it is called. A
key in config/local.yml stops the upload and names the file. Patterns only
cover what someone thought of; the scan is there for the rest.
If you would rather nothing left at all, the analysis needs the source today. That is the honest state.
submit_proof attaches the real requests and responses from your app's
before/after run to the receipt page, so the receipt shows your code's own
behaviour. That page is served without a login — that is the point of it, you
drop the link in a PR — which means the bodies on it are readable by anyone who
has the link.
The probes run against the FetchSandbox twin, not your provider, so the data is sandbox data. But the request bodies are the ones your app built, and those can carry values from your config. Look at a receipt before you share it.
Env var | Default | Purpose |
| none | Sign in without a browser. Overrides the stored credentials |
|
| Point at a different backend |
| on | Set to |
Telemetry records an opaque per-machine id (a random UUID in
~/.fetchsandbox/session.json), the tool name, latency, and whether the call
succeeded. Not spec content, not request bodies, not credentials. It's how we
count sessions and see which APIs people bring.
Once you sign in, calls are also attributed to your account — that is the point of signing in, and it is what lets your runs appear in one place.
FETCHSANDBOX_TELEMETRY=0 stops the per-machine id being sent, so calls are no
longer linked to your machine. It does not make a call invisible: the server
still records that a tool ran, because it is the thing running it. And if you
are signed in, your key identifies you regardless — that is what a key is. To
be unattributed, don't sign in.
MIT — see LICENSE.
Conversational integration coach for FetchSandbox. Server-side orchestrator that walks the user through adding an API integration (payments / email / auth / etc.) — intake the goal, elicit domain-aware discovery questions from the spec's brain.yaml, route to the right workflow, prove the contract via FetchSandbox, surface compliance notes. Call this BEFORE any other FetchSandbox tool when the user has an open-ended 'help me add X', 'integrate X', 'test my X integration' ask. BEHAVIOR — strict, do exactly this each turn: (1) Say message_for_user to the user (verbatim or lightly paraphrased to fit your voice — but don't add new content). (2) If next_action=wait_for_user AND options is present + non-empty: USE YOUR CLIENT'S NATIVE QUESTION-PICKER TOOL (in Cursor / Claude Code this is AskUserQuestion) to render the lettered picker with the question text, the options[].label as rows, default_option as default, and an 'Other...' freeform row when allow_freeform=true. When the user picks or types, call coach again with {session_id, user_response: <picked value or freeform text>}. (3) If next_action=wait_for_user AND no options: just wait for free text. (4) If next_action=call_tool: invoke tool_call.tool with tool_call.args, then call coach again with the result in context. (5) If next_action=done: end the session. The state machine is server-side — DON'T try to predict the next step or skip ahead; let the server drive.
| Name | Required | Description | Default |
|---|---|---|---|
| intent | No | User's free-form integration ask, on the FIRST call only. Pass through verbatim — the server's intent router benefits from the full phrasing. | |
| context | No | Optional context the LLM brings to the turn — e.g. a summary of the user's repo (if you ran an introspect step), or the result of a previously-instructed `tool_call`. | |
| session_id | No | Returned by a previous coach call. Required on every call after the first. | |
| user_response | No | The user's reply to the previous coach turn's question. Required when the previous turn returned `next_action: wait_for_user`. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate read-only, idempotent, non-destructive. The description adds substantial behavioral detail: it's a server-side state machine that requires the agent to strictly follow each turn's instructions, not predict ahead, and to use native question-picker tools. It discloses the exact expected behavior each turn, exceeding what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured: a clear purpose statement, then a numbered list of behavioral rules. Every sentence serves a purpose, though it could be tightened. The front-loading of the purpose and the explicit behavioral protocol justify the length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity, the description covers the necessary flow: how to initiate, how to handle each next_action type, how to use the picker, and when to end. It lacks explicit description of the output schema, but since there is none provided, it's acceptable. The description is thorough enough for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers all 4 parameters with descriptions. The description adds crucial context: intent is only on first call, session_id required after first, user_response needed when waiting. This clarifies parameter usage beyond the schema definitions, especially the conditional requirements.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's role as a conversational integration coach that orchestrates API integration setup. It specifies the resource (FetchSandbox integrations) and the action (walks the user through). It also differentiates itself by instructing to call this BEFORE any other FetchSandbox tool for open-ended integration asks, distinguishing it from siblings like find_bugs or run_workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: for open-ended 'help me add X', 'integrate X', 'test my X integration' asks, and instructs to call it before any other FetchSandbox tool. It also provides a detailed protocol for each turn, including when to use a question-picker tool and when to call other tools, which is essentially usage guidance for the interaction flow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
FIRST STEP for any API-integration bug — webhooks, payments, auth, email, SMS, subscriptions. Reach for this the moment a user reports a symptom in an app that talks to a provider: 'customers charged twice', 'the webhook fires but nothing happens', 'their balance is wrong', 'we keep emailing a bounced address', 'paused but they still have access'. Packages the current project — local, uncommitted, private, no git URL needed — and audits it against a corpus of real, harvested API-integration failures (idempotency, signature verification, event ordering, pagination, rate limits, state drift) that a read of the code alone does not surface. THIS IS A FINDING, NOT A PROOF. Every finding is a hypothesis until it is measured on the real code. Do NOT stop here, and do NOT write your own test to confirm one — call fix_bug for a proposed diff, then prove_fix to measure it. The response carries next_actions telling you exactly what to call next; follow it. Runs for up to a few minutes.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | OPTIONAL. Absolute path to the project to analyze. Defaults to the current working directory. Pass the repo root for a full audit. | |
| spec | No | OPTIONAL. Grounding hint for which integration to focus on (e.g. 'stripe', 'paddle', 'twilio'). Narrows the analysis to that API's known failure classes. | |
| timeout_s | No | OPTIONAL. Analysis budget in seconds (default 300, max 600). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark it read-only and non-destructive, and the description adds valuable behavior: it packages local uncommitted code without a git URL, runs for minutes, and produces hypotheses rather than confirmed proofs. This goes well beyond the safety profile already conveyed by annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The most important guidance is front-loaded: FIRST STEP, examples, and the finding-not-proof caveat. The longer symptom list and repeated emphasis on the next actions are purposeful and reinforce the critical workflow rather than padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description still tells the agent what the response will contain (next_actions) and to follow it. It covers runtime, scope, optional path behavior, and the limitation that findings are hypotheses, making the definition complete for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three optional parameters are already documented with meaning. The description adds contextual color about the project being analyzed, but no parameter-level semantics beyond what the schema provides, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States an explicit purpose: it is the FIRST STEP for API-integration bug discovery, auditing the current project against a corpus of known failure classes. It clearly distinguishes itself from siblings by calling itself a finding, not a proof, and by being the audit step before fix_bug and prove_fix.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use context with concrete symptom examples, and explicitly says not to stop here or write a test, directing the agent to fix_bug and prove_fix. It even tells the agent to follow next_actions, making the routing unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
FetchSandbox remediation: get a proposed fix for a specific bug in YOUR code. The 'fix' step after find_bugs. Ships the project to the FetchSandbox runtime, which authors a MINIMAL fix grounded in the known remediation for that failure class and returns a git diff — it does NOT modify your files, so review the diff and apply it yourself. IMPORTANT: the diff is a proposal, not a proof. After applying it, prove the fix (submit_proof / verify_behavior) so the buggy→fixed flip is measured, not assumed. Runs for up to a few minutes.
| Name | Required | Description | Default |
|---|---|---|---|
| bug | Yes | The specific bug to fix — ideally 'file:line — description', taken from a find_bugs finding. | |
| path | No | OPTIONAL. Absolute path to the project, as it is. Defaults to the current working directory. Do NOT copy the project first: fix_bug returns a diff and never writes to your tree. | |
| spec | No | OPTIONAL. Grounding hint for the integration (e.g. 'stripe'). | |
| timeout_s | No | OPTIONAL. Fix budget in seconds (default 300, max 600). | |
| fix_pattern | No | OPTIONAL. The known remediation pattern for this failure class, if you have it (e.g. from guide's matched_bug_pattern). Improves fix quality; the proof still certifies. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, and the description consistently reinforces this by stating 'it does NOT modify your files'. It adds valuable behavior beyond the annotations: the project is shipped to FetchSandbox runtime, the tool authors a minimal fix, the result is a git diff, and runtime can take up to a few minutes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences carry a lot of precisely relevant information: what the tool does, what it does not do, how long it runs, and what the user should do next. There is no filler, and the most important caveat is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 5 parameters, no output schema, and annotations that already cover read-only/destructive behavior, the description is complete enough for correct invocation. It tells the agent the input source (find_bugs), the output (git diff), the non-mutating behavior, the runtime bound, and the follow-up proof step.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the input schema already documents each parameter in detail. The description adds workflow-level context (e.g., bug should come from a find_bugs finding, and fix_pattern relates to the guide's matched_bug_pattern), but it does not materially expand parameter semantics beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb+resource: 'get a proposed fix for a specific bug in YOUR code', and explicitly names its role as 'The fix step after find_bugs'. It differentiates from siblings by emphasizing it returns a git diff and does not modify files, so an agent can distinguish it from prove_fix, submit_proof, and verify_behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit usage context: use it after find_bugs, and after applying the diff prove the fix using submit_proof or verify_behavior. It also warns that the diff is a proposal not a proof, which is a clear when-to-use/when-not-to-use boundary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ROUTE a symptom to the provider behaviour that explains it. Use this when the user names a provider or a domain (payments, email, auth, SMS, subscriptions) and you need to know what that provider ACTUALLY does — not what its docs say, and not what can be inferred from reading the integration code. Reading the code tells you what your app does with a field. It cannot tell you what the field MEANS at the provider — whether a line item is a seat count, whether a 200 body carries ok:false, whether an event can arrive out of order. That is the class of bug this routes. Returns {spec, workflow, scenario, confidence, reasoning} plus next_actions: a typed list of what to call next, with arguments pre-filled. Follow it rather than improvising the next step.
| Name | Required | Description | Default |
|---|---|---|---|
| hints | No | Optional caller-supplied overrides. Each field short-circuits the corresponding detection step. | |
| intent | Yes | The developer's free-form prompt as they typed it. Don't pre-process or shorten — the router benefits from the full phrasing (capture timing, geo cues, failure mode language). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds value by explaining the routing logic and the return shape ('Returns {spec, workflow, scenario, confidence, reasoning} plus next_actions'), which is not redundant with annotations. It does not contradict any annotation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is longer than average but every sentence serves a purpose: it defines the tool's unique value, explains the failure class it addresses, and outlines the output. The key action ('ROUTE a symptom...') is front-loaded, and the subsequent examples are illustrative, not filler. It could be tightened slightly but is not wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (routing across providers and domains) and the absence of an output schema, the description compensates by explicitly listing the return fields and next_actions. It also explains the reasoning behind its use, making it complete for an agent to decide when to call it and what to expect. No critical information is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds meaningful guidance beyond the schema, particularly for the intent parameter: 'Don't pre-process or shorten — the router benefits from the full phrasing (capture timing, geo cues, failure mode language)'. This is concrete, behavioral guidance that improves correct usage, justifying a 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'ROUTE a symptom to the provider behaviour that explains it', stating a specific verb and resource, and clearly distinguishes itself from code-reading and documentation tools by explaining what it does NOT do ('not what its docs say, and not what can be inferred from reading the integration code'). This makes its purpose unmistakable and separates it from siblings like find_bugs or fix_bug.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly provides a use condition ('Use this when the user names a provider or a domain...') and explains why alternatives fail (code cannot reveal provider meaning). It also instructs the agent to 'Follow it rather than improvising the next step', giving clear post-call guidance. This is strong, actionable usage direction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Ingest an OpenAPI spec and get a working sandbox you can call immediately. REQUIRED: pass EITHER url OR content — never just name alone (name is a display label, not a lookup). If the user mentions a popular API by name (Stripe, GitHub, Twilio, Notion, OpenAI, etc.), FIRST call list_specs with a filter to confirm it's in the catalog, then call import_spec with that vendor's public OpenAPI URL (e.g. Stripe: https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json). The backend content-hashes the spec and auto-matches to the bundled sandbox when applicable. Returns sandbox_id, workflows_preview, and a base_url that proxies schema-validated responses. Private URLs (localhost, 10.x, 192.168.x) are rejected by the backend — use content to paste those inline.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Public URL to an OpenAPI 3.x file (JSON or YAML). Use this for any publicly reachable spec — GitHub raw links, docs portals, vendor SDKs. | |
| name | No | Optional DISPLAY label only — NOT a lookup key. Defaults to info.title from the spec. To resolve a known API by name ("Stripe", "GitHub"), call list_specs first. | |
| content | No | Pasted OpenAPI spec content (JSON or YAML). Use this when the user pastes the spec inline or has it on disk. Provide the raw text exactly. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are thin (readOnlyHint=false, openWorldHint=true), so the description correctly shoulders the behavioral burden. It discloses real operational traits: backend content-hashing with auto-match to a bundled sandbox, private-URL rejection, and the proxying behavior of base_url for schema-validated responses. It also spells out return fields (sandbox_id, workflows_preview, base_url). No rate-limit or malformed-spec error detail, but for an ingest tool this is strong coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense paragraph, but every sentence is functional — the required-param rule, the sibling routing, the matching behavior, the return contract, and the private-URL edge case all earn their place. The critical constraint (never name alone) is bolded and front-loaded. Slightly long, but free of filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description must state the return contract, and it does (sandbox_id, workflows_preview, base_url). It covers the subtle constraints (name is not a key, private URL rejection) that an agent would otherwise trip on. Minor omissions like malformed-spec or size-limit error behavior prevent a 5, but for a 3-parameter ingest tool it is essentially complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds genuine value above the schema: it warns that name is a display label, NOT a lookup key (a non-obvious trap), encodes the url-XOR-content constraint, and gives a concrete URL example (Stripe raw GitHub link). This meaningfully compensates beyond the structured field docs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource ('Ingest an OpenAPI spec') and a concrete outcome ('get a working sandbox you can call immediately'). It clearly differentiates from siblings: list_specs is for catalog lookup, run_workflow for execution, and import_spec is the sandbox-creation tool. An agent can distinguish this from every sibling without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Exceptionally explicit. It states the hard rule ('REQUIRED: pass EITHER url OR content — never just name alone'), gives the exact branching condition for the alternate tool ('If the user mentions a popular API by name... FIRST call list_specs... then call import_spec'), and dictates parameter selection by scenario (private URLs → use content). Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List recent workflow runs (and ad-hoc traffic) for a sandbox, newest first. Use when the user asks 'what did I run', 'show me recent validation runs', 'did the stripe test pass earlier', or wants to find a previous run to share or re-inspect. Each run includes its shareable timeline URL (fetchsandbox.com/runs/) so the user can open the visual trace in a browser or drop it in Slack / a PR comment. Requires sandbox_id (from import_spec).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max number of runs to return (default 20, server may cap). | |
| sandbox_id | Yes | The sandbox_id returned by import_spec. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe read. The description adds behavioral context beyond annotations by noting that each run includes a shareable timeline URL (fetchsandbox.com/runs/<sandbox_id>) and that the server may cap the limit. This gives the agent a preview of the return payload and server-side behavior, which is useful. No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: it opens with the tool's core function and ordering, then immediately gives usage triggers, then explains the output benefit (shareable URLs). Every sentence earns its place—there is no filler or repetition. The length is appropriate for the tool's complexity, and the logical flow (what → when → output) helps an agent parse it quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description carries the burden of explaining what the tool returns. It mentions that runs include a shareable timeline URL, which is the most important output detail for sharing/re-inspection. It also clarifies that it covers ad-hoc traffic, and notes the limit default/cap. However, it does not enumerate other fields of each run (e.g., run ID, status, timestamps) that might be expected from a list endpoint. Given the tool's simplicity and the described use cases, this is a minor gap; the description is still largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already covers both parameters (limit and sandbox_id) with descriptions, achieving 100% coverage. The description adds extra semantic value by clarifying that sandbox_id comes specifically from import_spec, and that limit may be capped by the server, which the schema does not mention. This goes beyond the baseline expectation for high schema coverage, so a 4 is warranted.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the tool lists recent workflow runs and ad-hoc traffic for a sandbox, newest first. The verb is specific ('list'), the resource is specific ('recent workflow runs'), and it distinguishes from siblings like list_workflows by focusing on runs rather than defined workflows. The description also gives concrete example queries, making the purpose unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly enumerates when to use the tool: 'Use when the user asks "what did I run", "show me recent validation runs", "did the stripe test pass earlier", or wants to find a previous run to share or re-inspect.' It also states the required prerequisite (sandbox_id from import_spec), leaving no ambiguity about invocation conditions. While it doesn't state when not to use it, the positive triggers are clear and the tool is distinct from siblings, so this is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the FAILURE SCENARIOS this sandbox can inject on demand — the specific ways the real provider misbehaves in production. Call this BEFORE claiming an integration works. A checkout that passes on the happy path tells you almost nothing; the bugs that reach customers are declined cards, webhooks delivered twice or out of order, expired tokens, rate limits and slow networks. Typical names: payment_declined, insufficient_funds, webhook_retries, flaky_webhooks, replayed_old_signed_event, signed_event_with_mutated_body, auth_failure, rate_limited, slow_network — but they differ per provider, which is why you list them instead of guessing. Returns each scenario's name and what it does. Arm one with set_scenario, then re-run the same request and check the app still behaves correctly.
| Name | Required | Description | Default |
|---|---|---|---|
| sandbox_id | Yes | The sandbox_id returned by quickrun, run_workflow or import_spec. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds valuable behavioral context beyond annotations: it explains that the tool returns each scenario's name and what it does, and that scenario names differ per provider. It also implies the tool is a read-only discovery operation, consistent with annotations. Minor gap: it doesn't specify pagination or ordering, but that's not critical for a list operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is moderately long but every sentence earns its place: it defines the resource, gives usage timing, explains why the tool matters, provides example scenario names, notes provider variability, and routes to the next step. It is front-loaded with the core purpose ('List the FAILURE SCENARIOS') and the critical usage instruction ('Call this BEFORE claiming an integration works'). Slightly verbose in the middle with the example list, but that list adds concrete value for an agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with one parameter, 100% schema coverage, and annotations covering safety, the description is nearly complete. It explains what the return value contains (each scenario's name and what it does), which is useful since there is no output schema. It also provides the workflow context (list then set_scenario). It doesn't mention error cases or empty results, but those are minor for a discovery tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents the single parameter sandbox_id. The description mentions sandbox_id indirectly by referencing 'this sandbox' and the sibling set_scenario, but doesn't add new semantic detail about the parameter beyond what the schema provides. Baseline 3 is appropriate since the schema carries the parameter documentation burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('List') and resource ('FAILURE SCENARIOS this sandbox can inject on demand'), and explains what these scenarios are (specific ways the real provider misbehodes in production). It distinguishes itself from siblings by framing the tool as the discovery step before using set_scenario, and by noting that scenario names differ per provider, so listing is necessary rather than guessing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance: 'Call this BEFORE claiming an integration works.' It explains why the happy path is insufficient and that the tool should be used to discover provider-specific failure scenarios. It also names the sibling set_scenario as the next step ('Arm one with set_scenario'), providing clear routing between related tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Browse the FetchSandbox spec catalog — every API (Stripe, GitHub, Twilio, Notion, OpenAI, Polar, GitLab, and 40+ more) that has a ready-to-use sandbox with curated workflows. Use when the user asks 'what APIs do you support?', 'what specs are available?', 'show me the catalog', 'do you have ?', or wants to explore before committing to one. Returns each spec's slug (use as name arg to import_spec), description, endpoint count, and tags. Pass filter to narrow by substring (e.g., filter='pay' returns Stripe, Paddle, Polar).
| Name | Required | Description | Default |
|---|---|---|---|
| filter | No | Optional case-insensitive substring filter. Matches against spec name, slug, description, and tags. Omit to return the full catalog. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already carry the safety profile (readOnlyHint=true, idempotentHint=true, destructiveHint=false), and the description adds meaningful context beyond that: the return structure (slug, description, endpoint count, tags) and how the slug feeds into import_spec. OpenWorldHint ('40+ more') is honored without contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Resource is front-loaded first, then usage triggers, then return format, then filter example — a logical, scannable order. Slightly long, but each sentence contributes (concrete API names, query phrases, downstream slug usage, filter example), so no waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
A low-complexity tool (one optional param) with no output schema, so the description rightly takes on the return-format burden, which it meets. The only minor gap is no mention of result limits or pagination, but that's immaterial for a simple catalog browse tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% — the schema already fully documents the filter param (case-insensitive substring, matches name/slug/description/tags, omit for full catalog). The description's filter example (filter='pay') is a nice illustration but adds little beyond what the schema states, so the high-coverage baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a precise verb ('Browse') and a specific resource ('FetchSandbox spec catalog') with concrete examples of covered APIs (Stripe, GitHub, Twilio, etc.). It implicitly differentiates from siblings like list_workflows (workflows, not specs) and import_spec (the consumer of the returned slug).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit query triggers ('what APIs do you support?', 'do you have <X>?') and the decision context 'before committing to one', which routes the agent to import_spec afterward. This is as explicit as it gets for a browse tool without listing every sibling as a negative case.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the named, runnable workflows for a previously-imported spec. Workflows are realistic multi-step API journeys (e.g. 'create customer → attach payment method → create subscription'). Use this after import_spec for exploration ("what can I do?", "show me the flows") OR before run_all_workflows when the user wants a SCOPED validation: list, filter by user intent ("checkout", "webhooks"), then pass the matching ids as workflow_names to run_all_workflows. Returns: id, name, description, and step count per workflow.
| Name | Required | Description | Default |
|---|---|---|---|
| spec_id | Yes | The spec_id returned by import_spec. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint=false, covering safety. The description adds return format (id, name, description, step count) and the filtering implication (named, runnable workflows only). This goes beyond annotations by explaining what the agent receives and what 'runnable' means.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose, followed by usage guidance and return format. It is slightly verbose in the usage sentence but every part earns its place, providing actionable examples. It is well-structured without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple 1-parameter tool with no output schema, the description covers all essential aspects: what it returns, when to use it, and how it fits with siblings. The relationship to import_spec and run_all_workflows is explicit, and the return fields are listed, making it complete for an agent to decide and invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already describes spec_id as 'The spec_id returned by import_spec.' with 100% coverage. The description reinforces this by mentioning 'previously-imported spec' but adds no new meaning about the parameter itself. Baseline 3 is appropriate since the schema carries the full semantic load.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists named, runnable workflows for a previously imported spec, using a specific verb and resource. It differentiates from siblings like list_specs and run_all_workflows by specifying the object type (workflows) and the prerequisite (imported spec). The example journey also clarifies the concept.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use this tool: after import_spec for exploration, or before run_all_workflows for scoped validation. It even gives a workflow pattern: list, filter by intent, then pass matching ids to run_all_workflows. This provides clear context and alternatives, leaving no ambiguity about appropriate invocation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
THE PROOF STEP. The only way to establish that a fix actually works, and the only thing that produces a shareable receipt. Ship the STILL-BUGGY project plus your proposed diff: FetchSandbox applies the diff to a copy and runs the SAME failure probe against the buggy and the fixed tree, against real provider behaviour. Green ONLY on a measured flip — the bug demonstrably present before, demonstrably gone after. CALL THIS BEFORE YOU WRITE THE FIX TO DISK. prove_fix needs the broken tree to reproduce against; applying the diff first makes a measured proof impossible. DO NOT SUBSTITUTE YOUR OWN TEST. A harness you write runs only the FIXED code, so it can show a fix passing while proving nothing about whether the bug was ever reproduced or is genuinely gone — and an agent-authored check must never be reported to a user as 'proven' or 'verified'. If you have already written one, still call this: your test and this proof answer different questions.
| Name | Required | Description | Default |
|---|---|---|---|
| bug | No | The finding being fixed — used to select FetchSandbox's scenario. | |
| diff | Yes | The unified diff returned by fix_bug. | |
| path | No | OPTIONAL. The project root, as it is. Defaults to cwd. Do NOT copy the project first: prove_fix builds its own before/after copies and never writes to your tree. A path outside the workspace is refused. | |
| scenario | No | OPTIONAL. Explicit scenario id, if known. | |
| timeout_s | No | OPTIONAL. Budget in seconds (default 300). | |
| sandbox_id | No | OPTIONAL. A twin from quickrun/run_workflow. Usually omit — the last twin this session used is attached automatically. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already say readOnlyHint=false and destructiveHint=false, and the description adds valuable behavior beyond that: the tool builds its own before/after copies, never writes to the user's tree, applies the diff to a copy, requires a measured before/after flip, and refuses paths outside the workspace. It does not fully disclose failure-mode or pass-vs-error semantics, which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense and imperative, with front-loaded purpose and clear ordering constraint. It is longer than average and includes some repetition and all-caps emphasis, but nearly every sentence carries operational significance; a small amount of rhetorical repetition ('only way', 'DO NOT SUBSTITUTE', 'never') costs it one point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a six-parameter tool with no output schema, the description explains the core behavior (applies diff to a copy, runs same probe, measured flip), security/order constraints, and parameter idioms. It does not describe the result receipt shape, error cases, or what happens when the flip is not observed, which an agent would need to interpret the tool's output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already covers 100% of parameters with descriptions, so the baseline is 3. The description goes above the schema by explaining the significance of 'diff' (must come from fix_bug), 'path' (do not copy the project; outside paths refused), and 'sandbox_id' (usually omit; last twin auto-attached). This is genuine added value beyond the schema text.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('prove'), a precise mechanism (FetchSandbox applies the diff to a copy and runs the same failure probe against buggy and fixed trees), and a measurable success condition (green only on a measured flip). This makes it clearly distinct from siblings like verify_behavior, fix_bug, or submit_proof, and from an agent's own ad-hoc test.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit timing ('CALL THIS BEFORE YOU WRITE THE FIX TO DISK'), explains why ordering matters (the broken tree is needed for reproduction), and explicitly warns against substituting a custom harness with the rule that such a check must never be reported as proven or verified. This is strong when-vs-when-not guidance with clear user-facing consequences.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a curated proof workflow against a KNOWN, bundled spec (stripe, clerk, descope, resend, twilio, and 50+ others) in ONE call — it spins up the sandbox by slug, so you do NOT need import_spec or a sandbox_id first. This is the normal path when the user is testing an integration with a well-known provider: call guide on their prompt, then call quickrun with the returned spec + workflow. To reproduce a failure, pass the scenario from guide's matched_bug_pattern.reproduce_with.scenario (e.g. webhook_retries, payment_declined). Returns sandbox_id + flow_run_id — pass BOTH to verify_behavior to prove the fix — plus a receipt URL. Use run_workflow instead ONLY when you already hold a sandbox_id from import_spec of a custom/private spec.
| Name | Required | Description | Default |
|---|---|---|---|
| scenario | No | OPTIONAL failure scenario to reproduce (e.g. webhook_retries, payment_declined). Take it from guide's matched_bug_pattern.reproduce_with.scenario. Omit for the happy path. | |
| spec_slug | Yes | The bundled spec slug from guide (e.g. 'stripe'). Lowercase. | |
| workflow_name | Yes | The workflow id from guide (e.g. 'accept_payment'). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false, openWorldHint=true, and destructiveHint=false for the safety profile; the description adds meaningful behavioral context beyond that — it spins up a sandbox as a side effect, requires no prior sandbox_id, and discloses the exact return contract (sandbox_id + flow_run_id + receipt URL) with chaining guidance to verify_behavior. This exceeds baseline transparency for a state-changing tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four dense sentences, front-loaded with the core purpose and key differentiator (no import_spec needed). Every subsequent sentence earns its place: the normal call path, scenario reproduction, return contract, and the run_workflow exclusion. Slightly long, but no wasted words given the routing and chaining information it must carry.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a tool with no output schema: it discloses return values (sandbox_id, flow_run_id, receipt URL), preconditions (no import_spec needed), the failure-reproduction parameter path, and the surrounding workflow (guide → quickrun → verify_behavior). An agent has everything needed to invoke it correctly in context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3 and the schema already documents all three parameters, including where to source them ('from guide') and the scenario semantics. The description reinforces these sources with concrete examples (stripe, accept_payment, webhook_retries) but adds little meaning the schema doesn't already provide.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Run a curated proof workflow against a KNOWN, bundled spec') and immediately distinguishes itself from siblings: it does NOT need import_spec or a sandbox_id, and it is explicitly differentiated from run_workflow at the end. An agent can tell exactly what this tool is for and how it differs from its peers.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use guidance: 'This is the normal path when the user is testing an integration with a well-known provider: call guide... then call quickrun.' It also names the alternative and its condition: 'Use run_workflow instead ONLY when you already hold a sandbox_id from import_spec of a custom/private spec.' Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute EVERY workflow (or a scoped subset) for a sandbox in ONE call. Use this — NOT a loop of run_workflow — for any validation-style request: "validate this integration", "run all workflows", "check coverage", "test stripe checkout", "fs validate". IDEs (Cursor, Claude Code) approve each MCP call individually, so 18 workflows via run_workflow = 18 clicks. This tool = 1 click, total. Scope via workflow_names: pass an array of workflow ids to run a subset (e.g., user says "validate stripe CHECKOUT" → pass ["create_checkout_session", "checkout_complete"]). Names are case-insensitive; dashes and underscores interchangeable. Returns: summary (pass/fail counts, totals) + full step trace per workflow. After running, the user can visit fetchsandbox.com/runs/ for a shareable visual timeline.
| Name | Required | Description | Default |
|---|---|---|---|
| sandbox_id | Yes | The sandbox_id returned by import_spec. | |
| workflow_names | No | Optional list of workflow ids/names to run. Case-insensitive; dashes and underscores are interchangeable. Omit to run every workflow for the spec. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only provide flags (readOnlyHint=false, destructiveHint=false), so the description must fill in behavior. It discloses that this is a write/mutating operation (runs workflows), reduces UI approvals, and returns a summary plus full step trace. It also mentions the shareable timeline link. Slightly less than 5 because it doesn't detail failure behavior, but it covers key points.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense and front-loads the primary use case early with 'Execute EVERY workflow... in ONE call.' It then explains usage, scoping, and returns concisely. Slight padding with the IDE click explanation could be trimmed but serves a purpose for user context. Overall, it's structured effectively.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (batch execution), the description provides sufficient usage context, scoping options, and return information. It lacks a detailed list of possible failure modes or timeouts, but since there's no output schema and annotations are minimal, it does well. A 4 is justified as it could be more complete with error handling, but it's solidly above average.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, covering sandbox_id and workflow_names. The description adds value by explaining how workflow_names scopes the subset with examples, and clarifies case-insensitivity and character interchangeability, beyond the schema's brief notes. It reinforces the semantics, which is helpful. A 4 is appropriate since it's not adding entirely new meaning but contextualizes the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool executes every workflow (or a subset) for a sandbox in one call, and distinguishes it from run_workflow. It provides a strong verb (execute), specific resource (workflows for a sandbox), and concrete use cases (validate, run all workflows), making its purpose unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly tells users to use this instead of a loop of run_workflow, and gives concrete examples of when to use it ('validate this integration', 'run all workflows', etc.). It also explains how to scope with workflow_names, including case-insensitivity and dash/underscore handling, so an agent knows exactly when and how to invoke it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute ONE specific workflow by name and return its step-by-step trace PLUS a share_url — a public, replayable proof URL that renders the full timeline (every request, response, webhook event) for this run. The share_url is the canonical 'here's what happened' artifact: surface it verbatim in any reply that needs evidence (PR comments, Slack threads, blog posts, X replies). Do NOT substitute a docs URL or any other link as the proof — the share_url is the only valid receipt. Use ONLY when the user explicitly names a single workflow to run (e.g., "run accept_payment", "just check the refund workflow"). For ANY validation-style request — "validate stripe", "check coverage", "run all workflows", "test this integration", or even "validate stripe checkout" (multiple workflows match "checkout") — use run_all_workflows instead. The batch tool collapses N approvals to 1 and supports a workflow_names filter for scope. Calling this in a loop is an anti-pattern.
| Name | Required | Description | Default |
|---|---|---|---|
| scenario | No | OPTIONAL failure scenario to exercise (e.g. payment_declined, insufficient_funds, fraud_hold). Toggles the sandbox engine's scenario for the duration of the run, then restores. Use this for 'test with declined card' / 'simulate failure X' intents. Omit for the happy path. | |
| sandbox_id | Yes | The sandbox_id returned by import_spec. | |
| workflow_name | Yes | Workflow id or name from list_workflows. Case-insensitive; dashes and underscores are interchangeable. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false, openWorldHint=true, idempotentHint=false, and destructiveHint=false, and the description is consistent with all of them (execution implies side effects; webhook events align with openWorld). The description adds genuine value beyond annotations: the scenario toggling behavior (set for the run's duration then restored), the canonical status of share_url, and the instruction not to substitute a docs URL as proof. No contradictions found.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is longer than average but every sentence earns its place — primary action first, then output semantics, then usage rules and alternatives. It is front-loaded with the core action before the routing guidance. Slightly verbose in the exclusion list, but dense with actionable information rather than filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema present, the description correctly takes on the burden of explaining the return value (trace + share_url) and its role as the proof artifact. The tool is simple (3 params, 2 required, no enums, no nesting) and the description fully covers invocation, output, and routing — an agent has everything needed to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with each parameter (scenario, sandbox_id, workflow_name) already well documented including case-insensitivity and scenario-restore semantics. The description reinforces the workflow_name routing and scenario intent but does not add parameter details beyond what the schema provides, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('Execute ONE specific workflow by name') plus the precise output contract (step-by-step trace and a share_url proof artifact). It clearly distinguishes itself from run_all_workflows by contrasting single-workflow execution against batch validation, so an agent can tell them apart without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Extremely explicit about when to use it ('Use ONLY when the user explicitly names a single workflow') and when not to, naming the alternative (run_all_workflows) with concrete examples of ambiguous intents like 'validate stripe checkout' where multiple workflows match. Also flags calling this in a loop as an anti-pattern. Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ARM a failure scenario on a sandbox, so the next requests hit the provider misbehaving instead of the happy path. This is how you find out whether the integration you just wrote actually survives production. Use it like this: run the flow once on 'default' and confirm it works, call set_scenario with a failure from list_scenarios, run the SAME flow again, then check the end state is still correct — not that the call merely returned. A declined payment must not produce a confirmed order; a webhook delivered twice must not charge twice or send two emails. Set scenario to 'default' to return the sandbox to normal. The scenario stays armed until you change it, and it affects every caller of that sandbox — so put it back when you are done.
| Name | Required | Description | Default |
|---|---|---|---|
| scenario | Yes | A scenario name from list_scenarios, or 'default' to restore normal behaviour. Guessing a name that does not exist fails — list first. | |
| sandbox_id | Yes | The sandbox to arm. From quickrun, run_workflow or import_spec. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate non-read-only, open-world, idempotent, and non-destructive. The description adds that the scenario persists until changed and affects all callers of the sandbox, which is critical behavioral context beyond annotations. It also emphasizes the importance of resetting, fully disclosing side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single paragraph of about 150 words, covering purpose, usage, restoration, and warnings. While not as tight as a two-sentence description, every sentence contributes to correct usage, so it is appropriately sized and front-loaded with the core action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with two parameters and no output schema, the description covers the intended workflow, restoration, and side effects. It is complete for an agent to know when and how to invoke it correctly, including the need to verify end state and reset after testing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides thorough descriptions for both parameters, including valid values for scenario and failure behavior on unknown names. The description adds usage context but no new parameter-specific semantics, so the baseline of 3 is appropriate given 100% schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb (ARM) and resource (failure scenario on a sandbox), clearly distinguishing it from list_scenarios. It explains the purpose of testing integration resilience, leaving no ambiguity about what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit step-by-step workflow: run on 'default', arm a failure from list_scenarios, re-run, verify end state, and restore with 'default'. It also warns about side effects and the need to reset, making when and how to use it unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Receipts are readable by anyone with the link — do not attach bodies the user would not paste into a public PR. Attach REAL before/after evidence from the USER's actual app to the run receipt. Use this AFTER you've fixed the bug and re-run the app: it makes the receipt show their own code's behavior (before your fix vs after), which is far stronger proof than a reference simulation. Provide one proofs entry per probe you fired — the request you sent, the response BEFORE the fix (bug reproduces), and the response AFTER the fix (bug gone) — keyed to the run's sandbox_id + flow_run_id (from run_workflow's result). This is SELF-REPORTED evidence: it's shown on the receipt but never counts as a proven (green) result. For a MEASURED green proof — FetchSandbox reproduces the bug on your real code and verifies your fix flips it — use prove_fix instead.
| Name | Required | Description | Default |
|---|---|---|---|
| proofs | Yes | one per probe: { name, request:{method,path,body}, before:{status,body}, after:{status,body} }. before = pre-fix (bug reproduces), after = post-fix (bug gone). | |
| summary | No | optional one-line description of what you ran against the app | |
| sandbox_id | Yes | sandbox_id from the run_workflow result | |
| flow_run_id | Yes | flow_run_id from the run_workflow result | |
| bug_pattern_id | Yes | the bug_pattern id you reproduced (e.g. webhook_duplicate_side_effect) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false and destructiveHint=false, but the description adds critical behavioral context: this is self-reported evidence that 'never counts as a proven (green) result'. It also discloses that receipts are public and the submission is non-authoritative. This goes well beyond annotations and prevents misinterpretation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is detailed but every sentence earns its place: it front-loads the privacy warning, then gives usage timing, then explains the self-reported nature and contrast with prove_fix, then parameter guidance. It is structured logically and avoids redundancy with the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 5 parameters, nested objects, and no output schema, the description covers everything an agent needs: when to use it, what to attach, how to structure proofs, keying to run IDs, and its non-green status. The presence of an alternative (prove_fix) is clearly stated, so the agent can make an informed choice.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are already documented. However, the description adds operational semantics by instructing 'Provide one proofs entry per probe you fired' and explaining the expected structure (request, before, after) and keying to sandbox_id + flow_run_id. This adds value beyond the schema's field definitions, though it could have gone further with concrete examples.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the tool's purpose: to submit self-reported proof of a bug fix by attaching real before/after evidence from the user's app to the run receipt. It clearly distinguishes itself from the sibling tool 'prove_fix' by contrasting self-reported vs measured green proof, so an agent can tell them apart without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit when-to-use guidance: 'Use this AFTER you've fixed the bug and re-run the app'. It also names the alternative (prove_fix) for measured green proof and explains the condition (when you want FetchSandbox verification). Additionally, it warns about public visibility of receipts, which is an exclusion condition for sensitive content.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Prove a known fix survives a bug — the 'prove' half of reproduce→prove. The backend spawns a buggy AND a fixed reference handler and fires the bug_pattern's probes at both, returning the side-by-side diff (e.g. the buggy handler double-charges on a duplicate webhook, the fixed handler dedupes). Call this AFTER run_workflow reproduces a failure, when the matched bug_pattern has a simulation block, to show the fix actually holds — not just that the failure reproduced. Pass sandbox_id + flow_run_id from the run so the diff is saved onto that run's receipt URL. bug_pattern_id comes from guide's matched_bug_pattern. The buggy/fixed handlers are FetchSandbox reference implementations, NOT the user's code — apply the brain's fix_pattern to inherit the proven behavior.
| Name | Required | Description | Default |
|---|---|---|---|
| prompt | No | OPTIONAL. The user's own description of the symptom. For patterns that can originate in the handler OR the provider, this classifies which side to simulate. Omit to run both. | |
| sandbox_id | No | OPTIONAL. The sandbox from the run. Pass with flow_run_id to save the diff onto that run's receipt. | |
| flow_run_id | No | OPTIONAL. The flow_run_id returned by run_workflow. Pass with sandbox_id so the receipt URL renders the diff alongside the steps. | |
| bug_pattern_id | Yes | The bug_pattern to prove (e.g. webhook_duplicate_side_effect). Comes from guide's matched_bug_pattern.id or the spec's brain. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations (readOnlyHint=false, openWorldHint=true) suggest the tool may have side effects, and the description adds clarity by explaining it spawns two handlers, fires probes, saves the diff to a receipt URL when sandbox_id and flow_run_id are provided, and that it operates on reference implementations. This goes beyond the annotations' basic flags, though it doesn't enumerate potential side effects like cost or resource limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single paragraph but packs in the purpose, usage condition, parameter roles, and a critical caveat about reference implementations. It's front-loaded with the core purpose and flows logically. Slightly dense but not padded with fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with four parameters and no output schema, the description covers the invoke sequence (after run_workflow), the required inputs and their source (bug_pattern_id from guide), the optional inputs and their effect (saving diff to receipt), and even the return type (side-by-side diff). It also warns about the reference implementation limitation, leaving no critical gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaningful context for each parameter: prompt classifies which side to simulate, sandbox_id/flow_run_id are tied to the receipt URL, and bug_pattern_id comes from the guide's matched_bug_pattern. This goes beyond the schema's simple field descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Prove a known fix survives a bug'. It immediately distinguishes this from run_workflow by calling it the 'prove' half of the reproduce→prove sequence, and clarifies it returns a side-by-side diff. This is a concrete, non-tautological purpose that an agent can act on.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to call it: 'AFTER run_workflow reproduces a failure, when the matched bug_pattern has a simulation block'. It also conditions the necessity on having a matched bug_pattern and notes the handlers are reference implementations, not user code, which prevents misuse. This gives clear, actionable guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.5.7list_scenariosset_scenariov0.5.6fix_bug1 field changedInput schema / properties / path / descriptionPrevious value: -"OPTIONAL. Absolute path to the project. Defaults to the current working directory."New value: +"OPTIONAL. Absolute path to the project, as it is. Defaults to the current working directory. Do NOT copy the project first: fix_bug returns a diff and never writes to your tree."prove_fix1 field changedInput schema / properties / path / descriptionPrevious value: -"OPTIONAL. Project dir. Defaults to cwd."New value: +"OPTIONAL. The project root, as it is. Defaults to cwd. Do NOT copy the project first: prove_fix builds its own before/after copies and never writes to your tree. A path outside the workspace is refused."v0.5.1coachfind_bugsfix_bugguideimport_spec1 field changedInput schema / properties / name / descriptionPrevious value: -"Optional friendly name for the spec. Defaults to info.title from the spec, or the URL hostname."New value: +"Optional DISPLAY label only — NOT a lookup key. Defaults to info.title from the spec. To resolve a known API by name (\"Stripe\", \"GitHub\"), call list_specs first."list_runslist_specsprove_fixquickrunrun_all_workflowsrun_workflow1 field changedInput schema / properties / scenarioAdded value: +{
+ "description": "OPTIONAL failure scenario to exercise (e.g. payment_declined, insufficient_funds, fraud_hold). Toggles the sandbox engine's scenario for the duration of the run, then restores. Use this for 'test with declined card' / 'simulate failure X' intents. Omit for the happy path.",
+ "type": "string"
+}submit_proofverify_behaviorv0.1.1import_speclist_workflowsrun_workflowScored across 16 tools
Each tool has a clearly defined role in the sandbox workflow, and the descriptions thoroughly differentiate them. The only potential confusion is between prove_fix and verify_behavior, both of which prove fixes but against different codebases, and between quickrun and run_workflow, though the descriptions address this.
Most tools follow a verb_noun pattern (list_scenarios, set_scenario, import_spec, run_workflow, prove_fix), but a few single-word verbs (coach, guide) and 'quickrun' deviate. The pattern is still predictable enough for an agent to infer tool purpose.
At 16 tools, the set covers the full testing workflow but is slightly heavy. Each tool is justified, though some (prove_fix vs verify_behavior) could potentially be consolidated.
The lifecycle of import → run → inject failure → find → fix → prove is well covered, with no dead ends. Minor gaps exist (no explicit run-retrieval or sandbox-reset tool beyond list_runs/set_scenario), but they are workaroundable.
End-to-end API testing — generate and run tests from OpenAPI, curl, Postman, or real user traffic.
AI-callable tools for API mocking, testing, monitoring, security, and automation.
Discover, compare, and monitor 1,400+ APIs directly from your AI coding agent.
Build, validate, deploy — HTTP APIs, cron jobs, webhooks and MCP tools — from your AI client.
