Skip to main content
Glama
fetchsandbox-mcp

fetchsandbox-mcp

Official

fetchsandbox-mcp

Also on Smithery, npm and the official MCP registry.

A deterministic verification engine for agents, as an MCP server for FetchSandbox.

Your agent writes an integration. This checks whether it actually works — against a sandbox that behaves like the real provider, including the failures: retried webhooks, declined cards, rate limits, auth errors.

When it finds a bug, it can propose a fix and then prove it: the same failure is run against your code before and after the diff. Green only if it reproduced first and stopped after. You get a receipt URL either way.

Install

The MCP server runs locally through Node.js. Clients can resolve npx differently; Cursor on Mac may need the full path from command -v npx.

{
  "mcpServers": {
    "fetchsandbox": {
      "command": "npx",
      "args": ["-y", "fetchsandbox-mcp@latest"],
      "env": { "FETCHSANDBOX_API_KEY": "fsk_your_key_here" }
    }
  }
}

Discovery and reference tools are available without a key. Application planning, verification and managed local tunnelling require sign-in or an API key from https://fetchsandbox.com/keys. See Accounts.

Client

File

Claude Code

.mcp.json in your project root, or claude mcp add

Claude Desktop

~/Library/Application Support/Claude/claude_desktop_config.json

Cursor

~/.cursor/mcp.json, or .cursor/mcp.json in the repo

Zed

~/.config/zed/settings.json, under context_servers

Codex

~/.codex/config.toml, as [mcp_servers.fetchsandbox]

Cursor on Mac: tested setup

Get a real account key from https://fetchsandbox.com/keys. Run command -v npx and use that full path in ~/.cursor/mcp.json. This example is for Homebrew on Apple Silicon; replace the path if yours differs. On Windows, use where.exe npx to locate your installed command.

{
  "mcpServers": {
    "fetchsandbox": {
      "command": "/opt/homebrew/bin/npx",
      "args": ["-y", "fetchsandbox-mcp@0.7.24"],
      "env": { "FETCHSANDBOX_API_KEY": "fsk_your_key_here" }
    }
  }
}

Replace fsk_your_key_here privately with the real raw key, without a Bearer prefix. The placeholder cannot authenticate. Check for a second FetchSandbox entry in .cursor/mcp.json in your project. Fully quit and reopen Cursor after saving; a window reload may retain the old MCP process. Enable the server if needed and confirm its tools load. A green connection alone does not prove that an application verification call is authenticated.

Restart the client afterwards. Anything else that speaks MCP takes the same command and args.

No terminal? Use the hosted endpoint

Browser-based builders — Lovable, Bolt, v0 — can't run npx. They connect straight to the hosted streamable-HTTP transport instead:

https://fetchsandbox.com/mcp/v1

Authenticate with a Bearer token, not "None" — paste the key from https://fetchsandbox.com/keys.

Same twins, same scenarios, seven tools: validate_integration, quickrun, run_workflow, verify_behavior, list_workflows, guide, coach.

The rest are not hidden — they cannot work there. find_bugs, fix_bug and prove_fix read your project from disk and a hosted server has no copy of it, so advertising them only produced sessions that dead-ended. To analyse your own code, run npx fetchsandbox-mcp locally in the project you want looked at.

Start with validate_integration. A green quickrun proves the provider behaves as documented; it does not touch your checkout endpoint, your webhook handler, or your email send. validate_integration hands you a fresh twin per provider and then reports what your code actually called.

Related MCP server: @sandboxapis/mcp

Using it

Describe the problem the way you'd describe it to a colleague. You don't need to name a tool.

Customers are reporting more seats than they bought after a Paddle payment. Can you find out why?

The agent works through: route the symptom, reproduce it against the provider sandbox, read your code, get a fix, prove the fix on your code. Each step hands back what the next one needs.

One thing worth knowing, because it's easy to get backwards: prove_fix needs the unfixed tree. Run it before you write the diff to disk, or there's no bug left to reproduce and no proof to be had.

Accounts

You don't need one to start. Install it, ask a question, and everything runs.

The first time a run produces something worth keeping — a receipt, or a set of findings — you'll get a short code and a link. Signing in takes about twenty seconds and does two things: the evidence behind your receipts stops being archived after 15 days, and the runs from that machine collect in one place. You'll be asked at most once a day, and never once you're signed in.

For CI, the hosted endpoint, or anywhere a browser isn't available, set a key instead. Create one at https://fetchsandbox.com/keys — sign in, then "Create a key". It looks like fsk_....

In an MCP client config, it goes in env:

"env": { "FETCHSANDBOX_API_KEY": "fsk_your_key_here" }

In a shell or CI:

export FETCHSANDBOX_API_KEY=fsk_your_key_here

On the hosted endpoint, send it as Authorization: Bearer fsk_....

The key is also written to ~/.fetchsandbox/credentials.json when you sign in from an editor; the environment variable always wins.

Tools

Start with guide. It picks the right ones for what you asked.

Finding and fixing

Tool

What it does

Arguments

guide

Routes a symptom to a spec, workflow and known failure class

intent*, hints

find_bugs

Audits your project against known integration failure classes. No git remote needed — it reads the directory you point it at

path, spec, timeout_s

fix_bug

Returns a git diff for one finding. Doesn't touch your files

bug*, fix_pattern, path, spec, timeout_s

get_job

Retrieve the same running job without starting another analysis or proof

job_id*

prove_fix

Runs the failure against your code before and after the diff. Green only on a measured flip

diff*, bug, scenario, sandbox_id, path, timeout_s

Verifying an integration

Tool

What it does

Arguments

validate_integration

Proves your own code against a provider, not that our twin works. Hands back a fresh twin per provider, then reads our request log to report what your app actually called

providers, app_base_url, session_id

quickrun

Runs one of OUR curated workflows against a bundled spec. Green here says the provider behaves as documented — it says nothing about your integration

spec_slug*, workflow_name*, scenario

run_workflow

The same, on a sandbox you already have — this is where you re-run with a scenario armed

sandbox_id*, workflow_name*, scenario

verify_behavior

Shows a failure class on reference handlers — buggy vs fixed. Not your code

bug_pattern_id*, prompt, sandbox_id, flow_run_id

list_workflows

Workflows AND the failure scenarios a spec can inject, in one answer

spec_slug or spec_id

guide

Routes a symptom to the provider behaviour that explains it, and returns the next call with its arguments filled in

intent*, hints

coach

Multi-turn help building an integration

intent, session_id, user_response, context

Those seven are the whole of what a browser-based builder sees, and they are the only ones that need no filesystem. Everything below requires a local project.

On a machine with your code

Tool

What it does

Arguments

find_bugs

Audits your project against known integration failure classes

path, spec, timeout_s

fix_bug

Returns a git diff for one finding. Doesn't touch your files

bug*, fix_pattern, path, spec, timeout_s

get_job

Retrieve the same running job without starting another analysis or proof

job_id*

prove_fix

Runs the failure against your code before and after the diff. Green only on a measured flip

diff*, bug, scenario, sandbox_id, path, timeout_s

run_all_workflows

Runs several workflows in one call

sandbox_id*, workflow_names

list_scenarios

The failures a spec can inject

sandbox_id*

set_scenario

Arms a failure on a sandbox

sandbox_id*, scenario*

list_runs

Past runs for a sandbox

sandbox_id*, limit

list_specs

Specs already available

filter

submit_proof

Publishes a receipt for a run

sandbox_id, flow_run_id, bug_pattern_id, summary, proofs

* = required.

What leaves your machine

find_bugs, fix_bug and prove_fix package the directory you point them at and upload it for analysis. Worth saying plainly, because the previous wording here implied the opposite.

Excluded before packing: .git, node_modules and build output, agent instruction files, and anything credential-shaped — .env*, *.pem, *.key, id_rsa*, *.tfstate, .npmrc, .aws, .ssh, .mcp.json, .mcp.*.json, mcp.json, and more.

Then the archive is read back and refused if it still contains something shaped like a live credential, wherever it lives and whatever it is called. A key in config/local.yml stops the upload and names the file. Patterns only cover what someone thought of; the scan is there for the rest.

If you would rather nothing left at all, the analysis needs the source today. That is the honest state.

Long-running work

The MCP response waits briefly, then returns status: running, its job_id, and a next_tool_call for get_job. Follow that call until status: done or error. It reads the existing backend job; it does not upload the project again or start another proof. Keep the original tree unchanged while the proof runs. A running job does not establish a pass.

This applies to discovery, proposed fixes, application proofs, and async reference verification. Direct library calls keep their blocking behavior. Job retention and ownership still belong to the existing backend. This change does not make its in-memory job tracker survive a backend restart.

submit_proof attaches the real requests and responses from your app's before/after run to the receipt page, so the receipt shows your code's own behaviour. That page is served without a login — that is the point of it, you drop the link in a PR — which means the bodies on it are readable by anyone who has the link.

Provider execution can use a FetchSandbox twin or a recording test double; read the receipt's execution scope. A recording double does not establish twin or live-provider fidelity. But the request bodies are the ones your app built, and those can carry values from your config. Look at a receipt before you share it.

Configuration

Env var

Default

Purpose

FETCHSANDBOX_API_KEY

none

Sign in without a browser. Overrides the stored credentials

FETCHSANDBOX_BASE_URL

https://fetchsandbox.com

Point at a different backend

FETCHSANDBOX_TELEMETRY

on

Set to 0 to turn off

Telemetry records an opaque per-machine id (a random UUID in ~/.fetchsandbox/session.json), the tool name, latency, and whether the call succeeded. Not spec content, not request bodies, not credentials. It's how we count sessions and see which APIs people bring.

Once you sign in, calls are also attributed to your account — that is the point of signing in, and it is what lets your runs appear in one place.

FETCHSANDBOX_TELEMETRY=0 stops the per-machine id being sent, so calls are no longer linked to your machine. It does not make a call invisible: the server still records that a tool ran, because it is the thing running it. And if you are signed in, your key identifies you regardless — that is what a key is. To be unattributed, don't sign in.

License

MIT — see LICENSE.

Verify a local app without ngrok

Version 0.7.13 adds manage_tunnel to the local MCP package. Start your actual isolated app, open its loopback port, and give the returned app_base_url to validate_integration. Declare the existing app webhook path in application_config.event_destinations. FetchSandbox signs provider-native events and checks independently saved state through the existing application runner. Configure test secrets via the protected handoff, preflight, execute once and retrieve the same run receipt. Close the connection and remove temporary settings afterward. Connectivity alone is not a passing test.

Requires outbound HTTPS to fetchsandbox.com, a running local MCP process and a real local app. No inbound firewall rule or separate tunnel install is needed. Restricted networks may require their approved proxy/CA configuration. Node 22.21+ or 24.5+ supports NODE_USE_ENV_PROXY=1 with HTTPS_PROXY and NO_PROXY; include 127.0.0.1/localhost in NO_PROXY and use NODE_EXTRA_CA_CERTS for an approved corporate CA. Older Node versions do not automatically support this proxy setup. Do not disable TLS certificate validation. Corporate firewalls can still block the connection. Leases expire after 30 minutes or server restart; uncertain writes are not automatically repeated.

Private credentials for local app verification (0.7.14)

After configuring an owned application run, Cursor, Claude Code and other local MCP clients call install_test_credentials with its session and run IDs. The existing FetchSandbox account key authorizes retrieval. Temporary provider keys, webhook secrets and the verifier token are written to a new private dotenv file under ~/.fetchsandbox/handoffs/ (directory 0700, file 0600), outside the project. Only its path and variable names enter the tool result. env_mapping explicitly maps returned names to the real app's environment settings.

Load that file with the disposable application's dotenv loader without printing it. Configure the returned twin base URLs, restart the app, preflight, execute once and retrieve the same receipt. Remove the temporary file after testing. Files do not delete themselves and are not production credentials. Never paste values into chat, source, logs or command arguments. Replit/Lovable keep their existing owner browser handoff and native secret managers. This change does not route prove_fix through the local relay.

Available Tools

19 tools
coachA
Read-onlyIdempotent
Inspect

Conversational integration coach for FetchSandbox. Server-side orchestrator that walks the user through adding an API integration (payments / email / auth / etc.) — intake the goal, elicit domain-aware discovery questions from the spec's brain.yaml, route to the right workflow, prove the contract via FetchSandbox, surface compliance notes. Call this BEFORE any other FetchSandbox tool when the user has an open-ended 'help me add X', 'integrate X', 'test my X integration' ask. BEHAVIOR — strict, do exactly this each turn: If next_action=act_in_app, carry out next_actions with your platform's code, configuration and test tools, then return the results to the same validation session. Do not ask the human to choose routine fault tests, supply missing fixture data you can generate, or copy tool output. Ask only for genuine access, secret-entry constraints or ambiguous business rules. Never claim unsupported checks passed. (1) Say message_for_user to the user (verbatim or lightly paraphrased to fit your voice — but don't add new content). (2) If next_action=wait_for_user AND options is present + non-empty: USE YOUR CLIENT'S NATIVE QUESTION-PICKER TOOL (in Cursor / Claude Code this is AskUserQuestion) to render the lettered picker with the question text, the options[].label as rows, default_option as default, and an 'Other...' freeform row when allow_freeform=true. When the user picks or types, call coach again with {session_id, user_response: <picked value or freeform text>}. (3) If next_action=wait_for_user AND no options: just wait for free text. (4) If next_action=call_tool: invoke tool_call.tool with tool_call.args, then call coach again with the result in context. (5) If next_action=done: end the session. For a hosted app-builder request to build or test an integration, keep the user's simple goal intact and let the server route all named providers into one validation session. When FetchSandbox returns twin bindings, treat those as temporary TEST credentials: configure them server-side in the app-builder project, never ask the user for live Paddle/Resend keys, and never claim they were applied until the builder reports the change and the verifier observes app traffic. Infer app configuration details from the project when available; ask the user only for a fact the builder cannot discover. The deterministic verifier, not this conversation, owns the verdict. The state machine is server-side — DON'T try to predict the next step or skip ahead; let the server drive.

NameRequiredDescriptionDefault
intentNoUser's free-form integration ask, on the FIRST call only. Pass through verbatim — the server's intent router benefits from the full phrasing.
contextNoOptional context the LLM brings to the turn — e.g. a summary of the user's repo (if you ran an introspect step), or the result of a previously-instructed `tool_call`.
session_idNoReturned by a previous coach call. Required on every call after the first.
user_responseNoThe user's reply to the previous coach turn's question. Required when the previous turn returned `next_action: wait_for_user`.

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover the safety profile (readOnly/idempotent/openWorld), and the description goes well beyond them: server-side state machine ownership, twin bindings as temporary TEST credentials, never asking for live Paddle/Resend keys, never claiming unverified checks passed, and the rule to ask only for genuine access/secret/ambiguous-business-rule facts. That is rich operational context an agent cannot infer from annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is correctly front-loaded, but the body is roughly 450 words of dense prose mixing routing rules, safety policy, and credential guidance, with some redundancy ('the state machine is server-side' / 'let the server drive' / 'don't predict the next step'). The numbered turn protocol helps, yet the whole would benefit from tighter structure.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates by explaining the return contract in prose (the next_action states and how to handle each) along with credential, verification, and routing semantics. For a multi-turn orchestrator with four optional parameters, nothing essential to correct invocation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and each of the four parameters is already documented (intent first-call-only, session_id after the first call, user_response after wait_for_user, context). The description restates the same call protocol rather than adding new syntax, formats, or constraints, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific role and resource ('conversational integration coach', 'server-side orchestrator that walks the user through adding an API integration') and enumerates its actual functions: intake, discovery questions from brain.yaml, workflow routing, FetchSandbox proof, compliance notes. It also differentiates itself from siblings by stating it must be called BEFORE any other FetchSandbox tool for open-ended integration asks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('open-ended help me add X / integrate X / test my X integration') and explicit when-not ('DON'T try to predict the next step or skip ahead'), plus the per-turn routing rules for act_in_app, wait_for_user, call_tool, and done. It also names the alternative path indirectly by insisting the server, not the caller, selects workflows and the verifier owns the verdict.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

find_bugsA
Read-only
Inspect

Long work returns status:running and next_tool_call:get_job within a short response. Follow that SAME job until done; never repeat this tool to poll. FIRST STEP for any API-integration bug — webhooks, payments, auth, email, SMS, subscriptions. Reach for this the moment a user reports a symptom in an app that talks to a provider: 'customers charged twice', 'the webhook fires but nothing happens', 'their balance is wrong', 'we keep emailing a bounced address', 'paused but they still have access'. Packages the current project — local, uncommitted, private, no git URL needed — and audits it against a corpus of real, harvested API-integration failures (idempotency, signature verification, event ordering, pagination, rate limits, state drift) that a read of the code alone does not surface. THIS IS A FINDING, NOT A PROOF. Every finding is a hypothesis until it is measured on the real code. Do NOT stop here, and do NOT write your own test to confirm one — call fix_bug for a proposed diff, then prove_fix to measure it. The response carries next_actions telling you exactly what to call next; follow it. Runs for up to a few minutes.

NameRequiredDescriptionDefault
pathNoOPTIONAL. Absolute path to the project to analyze. Defaults to the current working directory. Pass the repo root for a full audit.
specNoOPTIONAL. Grounding hint for which integration to focus on (e.g. 'stripe', 'paddle', 'twilio'). Narrows the analysis to that API's known failure classes.
timeout_sNoOPTIONAL. Analysis budget in seconds (default 300, max 600).

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover the safety profile (readOnlyHint, openWorldHint, destructiveHint false), but the description adds substantial behavior they cannot convey: it is long-running (up to a few minutes), it returns status:running/next_tool_call rather than a blocking result, it packages local/uncommitted/private code with no git URL required, and its output is a hypothesis rather than a proof. These are non-obvious operational facts an agent must know before invoking.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the single most important operational fact (do not poll, follow the job) and otherwise dense with useful workflow content. It is long and the 'do not stop here' / 'follow next_actions' points are restated in adjacent sentences, costing a little efficiency, but nearly every sentence carries weight for a multi-step workflow tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, yet the description still tells the agent what comes back (status:running, next_tool_call, next_actions), how long it takes, and what to do with the result. Combined with the 100% covered input schema, nothing an agent needs to call this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all three optional parameters (path, spec, timeout_s) are already fully documented, including defaults and the max on timeout_s. The description adds no format or semantic detail beyond the schema, so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (packages the project and audits it against a corpus of harvested API-integration failure classes) and enumerates the exact symptom classes it targets (webhooks, payments, auth, idempotency, signature verification). An agent can distinguish it from fix_bug and prove_fix without opening any schema, since the description names those siblings and their roles.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use ('FIRST STEP for any API-integration bug', 'the moment a user reports a symptom'), explicit when-not ('never repeat this tool to poll', use get_job instead), and explicit alternatives in the workflow ('call fix_bug for a proposed diff, then prove_fix to measure it'). This is about as complete a routing instruction as a description can carry.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fix_bugA
Read-only
Inspect

Long work returns status:running and next_tool_call:get_job within a short response. Follow that SAME job until done; never repeat this tool to poll. FetchSandbox remediation: get a proposed fix for a specific bug in YOUR code. The 'fix' step after find_bugs. Ships the project to the FetchSandbox runtime, which authors a MINIMAL fix grounded in the known remediation for that failure class and returns a git diff — it does NOT modify your files. Review the diff, then call prove_fix with the diff and the STILL-BUGGY project BEFORE applying it. Apply the patch only after inspecting that result. The diff is a proposal, not a proof; quickrun and verify_behavior on reference handlers do not prove this change on your application. Runs for up to a few minutes.

NameRequiredDescriptionDefault
bugYesThe specific bug to fix — ideally 'file:line — description', taken from a find_bugs finding.
pathNoOPTIONAL. Absolute path to the project, as it is. Defaults to the current working directory. Do NOT copy the project first: fix_bug returns a diff and never writes to your tree.
specNoOPTIONAL. Grounding hint for the integration (e.g. 'stripe').
timeout_sNoOPTIONAL. Fix budget in seconds (default 300, max 600).
fix_patternNoOPTIONAL. The known remediation pattern for this failure class, if you have it (e.g. from guide's matched_bug_pattern). Improves fix quality; the proof still certifies.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare a safe read (readOnlyHint=true, destructiveHint=false); the description reinforces this with 'it does NOT modify your files' and adds context the annotations do not carry: async job semantics (status:running, next_tool_call:get_job), runtime up to a few minutes, and the caveat that the diff is a proposal, not a proof. No contradiction with the readOnly annotation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the highest-risk operational fact (async job / never poll), then purpose, then the review-before-apply workflow. Dense but each sentence carries instruction or constraint; slightly long and the async aside competes with the core purpose at the top.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex, multi-step, asynchronous tool with no output schema, the description supplies the return shape (git diff, status:running, next_tool_call), the duration expectation, and the full downstream handoff to prove_fix. An agent has everything needed to call and chain it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning: the 'bug' value should be a find_bugs finding ('file:line — description'), fix_pattern should come from guide's matched_bug_pattern, and it stresses not copying the project because no writes occur. These tie parameters to the surrounding workflow rather than restating types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('get a proposed fix for a specific bug in YOUR code') and positions it in the workflow as 'the fix step after find_bugs'. Explicitly distinguishes it from siblings prove_fix and find_bugs, and clarifies it authoring a diff rather than editing files.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('after find_bugs'), the required next step ('Review the diff, then call prove_fix ... BEFORE applying it'), the gating condition ('Apply the patch only after inspecting that result'), and async handling ('Follow that SAME job ... never repeat this tool to poll'). Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_jobA
Read-onlyIdempotent
Inspect

Retrieve the SAME running find_bugs, fix_bug, prove_fix, verify_behavior or validate_integration job (including requirements planning). Follow next_tool_call until status is done or error. Never restart the original tool to poll. A proposed plan is not execution proof; a running response is not proof. Only the completed measured verdict and receipt can establish a pass.

NameRequiredDescriptionDefault
job_idYesExact job_id returned by the original call.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and destructiveHint=false, so safety is covered. The description adds genuinely new behavioral context: the tool returns a next_tool_call to be followed iteratively, and a running response is explicitly not proof of success. It does not cover auth, rate limits, or return shape, but the added polling semantics are substantial.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and resource, then the procedural rule. The final two sentences are somewhat aphoristic ('a proposed plan is not execution proof'), but they do carry the meaningful warning that only a completed verdict establishes a pass, so they earn most of their space.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a polling tool with no output schema, the description supplies the essential loop behavior and the success/error termination condition. It is nearly complete, though it never describes what next_tool_call or the 'receipt' actually contain, leaving the agent to discover that at runtime.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter (job_id) with 100% schema description coverage ('Exact job_id returned by the original call'). The description reinforces that it must be the SAME job but adds no new syntax or format detail beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Retrieve) plus the exact resource (the SAME running job) and enumerates the originating tools whose jobs it polls (find_bugs, fix_bug, prove_fix, verify_behavior, validate_integration). An agent can immediately tell this apart from the starter tools that create those jobs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('Retrieve the SAME running ... job') and an explicit when-not ('Never restart the original tool to poll'), plus the operating loop ('Follow next_tool_call until status is done or error'). The anti-pattern is named directly, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

guideA
Read-onlyIdempotent
Inspect

ROUTE a symptom to the provider behaviour that explains it. Use this when the user names a provider or a domain (payments, email, auth, SMS, subscriptions) and you need to know what that provider ACTUALLY does — not what its docs say, and not what can be inferred from reading the integration code. Reading the code tells you what your app does with a field. It cannot tell you what the field MEANS at the provider — whether a line item is a seat count, whether a 200 body carries ok:false, whether an event can arrive out of order. That is the class of bug this routes. Returns {spec, workflow, scenario, confidence, reasoning} plus next_actions: a typed list of what to call next, with arguments pre-filled. Follow it rather than improvising the next step.

NameRequiredDescriptionDefault
hintsNoOptional caller-supplied overrides. Each field short-circuits the corresponding detection step.
intentYesThe developer's free-form prompt as they typed it. Don't pre-process or shorten — the router benefits from the full phrasing (capture timing, geo cues, failure mode language).

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds value by explaining the routing logic and the return shape ('Returns {spec, workflow, scenario, confidence, reasoning} plus next_actions'), which is not redundant with annotations. It does not contradict any annotation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is longer than average but every sentence serves a purpose: it defines the tool's unique value, explains the failure class it addresses, and outlines the output. The key action ('ROUTE a symptom...') is front-loaded, and the subsequent examples are illustrative, not filler. It could be tightened slightly but is not wasteful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (routing across providers and domains) and the absence of an output schema, the description compensates by explicitly listing the return fields and next_actions. It also explains the reasoning behind its use, making it complete for an agent to decide when to call it and what to expect. No critical information is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds meaningful guidance beyond the schema, particularly for the intent parameter: 'Don't pre-process or shorten — the router benefits from the full phrasing (capture timing, geo cues, failure mode language)'. This is concrete, behavioral guidance that improves correct usage, justifying a 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'ROUTE a symptom to the provider behaviour that explains it', stating a specific verb and resource, and clearly distinguishes itself from code-reading and documentation tools by explaining what it does NOT do ('not what its docs say, and not what can be inferred from reading the integration code'). This makes its purpose unmistakable and separates it from siblings like find_bugs or fix_bug.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly provides a use condition ('Use this when the user names a provider or a domain...') and explains why alternatives fail (code cannot reveal provider meaning). It also instructs the agent to 'Follow it rather than improvising the next step', giving clear post-call guidance. This is strong, actionable usage direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

install_test_credentialsAInspect

Local coding agents only: after validate_integration configures an owned run, install its temporary twin API keys, webhook signing secrets and verifier token privately. Uses your existing FetchSandbox MCP account key; no browser sign-in or secret values in chat. Returns a private dotenv file path and variable names ONLY. Supply env_mapping from returned required_secret_names to the inspected app setting names, e.g. FETCHSANDBOX_STRIPE_WEBHOOK_SECRET:STRIPE_WEBHOOK_SECRET. Never invent secrets or remove signature checks. Start/restart ONLY the disposable test app with this file using its dotenv loader; never cat/read/print it or paste values into a prompt, source, logs or shell arguments. Twin base URLs come from validate_integration and must also be configured. Preserve existing production configuration. Then preflight and execute once, retrieve the same run receipt, and delete this temporary file after testing. Hosted Replit/Lovable use their existing protected browser handoff and native secret manager.

NameRequiredDescriptionDefault
run_idYes
session_idYes
env_mappingNoTemporary handoff variable name to actual app environment variable name; unspecified names are preserved.

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover the safety quadruple (readOnly=false, openWorld=true, idempotent=false, destructive=false). The description adds substantial behavioral context beyond that: the account key used, no browser sign-in, secrets never appear in chat, return is limited to a private dotenv path and variable names, the file must never be read/printed/catted, existing production configuration must be preserved, and the file must be deleted after testing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and guardrails are front-loaded, but the body is a dense run-on of imperatives that partially repeats itself ('no secret values in chat' vs 'never cat/read/print it or paste values into a prompt'). Most content is safety-critical, so little is truly wasted, but the packaging hurts scanability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a multi-step credential-installation tool with no output schema, the description covers the full lifecycle: preconditions, env_mapping usage, twin base URL configuration, preflight/execute, receipt retrieval, and cleanup. Nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 33%, so the description must carry the load. It does well on env_mapping, giving the exact mapping semantics and an example ('FETCHSANDBOX_STRIPE_WEBHOOK_SECRET:STRIPE_WEBHOOK_SECRET') and noting unspecified names are preserved, but session_id and run_id are never explained beyond the surrounding workflow narrative.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a precise verb and resource: install temporary twin API keys, webhook signing secrets and verifier token for an owned run configured by validate_integration. It names the sibling (validate_integration) and the operational scope (local coding agents only), so an agent can distinguish it from other workflow tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly scopes who may use it ('Local coding agents only'), when to use it (after validate_integration configures an owned run), what to feed it (env_mapping from returned required_secret_names), and when NOT to use it (hosted Replit/Lovable use their protected browser handoff instead). This is a full when/when-not/alternative routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_runsA
Read-onlyIdempotent
Inspect

List recent workflow runs (and ad-hoc traffic) for a sandbox, newest first. Use when the user asks 'what did I run', 'show me recent validation runs', 'did the stripe test pass earlier', or wants to find a previous run to share or re-inspect. Each run includes its shareable timeline URL (fetchsandbox.com/runs/<sandbox_id>) so the user can open the visual trace in a browser or drop it in Slack / a PR comment. Requires sandbox_id (from import_spec).

NameRequiredDescriptionDefault
limitNoMax number of runs to return (default 20, server may cap).
sandbox_idYesThe sandbox_id returned by import_spec.

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe read. The description adds behavioral context beyond annotations by noting that each run includes a shareable timeline URL (fetchsandbox.com/runs/<sandbox_id>) and that the server may cap the limit. This gives the agent a preview of the return payload and server-side behavior, which is useful. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: it opens with the tool's core function and ordering, then immediately gives usage triggers, then explains the output benefit (shareable URLs). Every sentence earns its place—there is no filler or repetition. The length is appropriate for the tool's complexity, and the logical flow (what → when → output) helps an agent parse it quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description carries the burden of explaining what the tool returns. It mentions that runs include a shareable timeline URL, which is the most important output detail for sharing/re-inspection. It also clarifies that it covers ad-hoc traffic, and notes the limit default/cap. However, it does not enumerate other fields of each run (e.g., run ID, status, timestamps) that might be expected from a list endpoint. Given the tool's simplicity and the described use cases, this is a minor gap; the description is still largely complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already covers both parameters (limit and sandbox_id) with descriptions, achieving 100% coverage. The description adds extra semantic value by clarifying that sandbox_id comes specifically from import_spec, and that limit may be capped by the server, which the schema does not mention. This goes beyond the baseline expectation for high schema coverage, so a 4 is warranted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the tool lists recent workflow runs and ad-hoc traffic for a sandbox, newest first. The verb is specific ('list'), the resource is specific ('recent workflow runs'), and it distinguishes from siblings like list_workflows by focusing on runs rather than defined workflows. The description also gives concrete example queries, making the purpose unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly enumerates when to use the tool: 'Use when the user asks "what did I run", "show me recent validation runs", "did the stripe test pass earlier", or wants to find a previous run to share or re-inspect.' It also states the required prerequisite (sandbox_id from import_spec), leaving no ambiguity about invocation conditions. While it doesn't state when not to use it, the positive triggers are clear and the tool is distinct from siblings, so this is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_scenariosA
Read-onlyIdempotent
Inspect

List the FAILURE SCENARIOS this sandbox can inject on demand — the specific ways the real provider misbehaves in production. Call this BEFORE claiming an integration works. A checkout that passes on the happy path tells you almost nothing; the bugs that reach customers are declined cards, webhooks delivered twice or out of order, expired tokens, rate limits and slow networks. Typical names: payment_declined, insufficient_funds, webhook_retries, flaky_webhooks, replayed_old_signed_event, signed_event_with_mutated_body, auth_failure, rate_limited, slow_network — but they differ per provider, which is why you list them instead of guessing. Returns each scenario's name and what it does. Arm one with set_scenario, then re-run the same request and check the app still behaves correctly.

NameRequiredDescriptionDefault
sandbox_idYesThe sandbox_id returned by quickrun, run_workflow or import_spec.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds valuable behavioral context beyond annotations: it explains that the tool returns each scenario's name and what it does, and that scenario names differ per provider. It also implies the tool is a read-only discovery operation, consistent with annotations. Minor gap: it doesn't specify pagination or ordering, but that's not critical for a list operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is moderately long but every sentence earns its place: it defines the resource, gives usage timing, explains why the tool matters, provides example scenario names, notes provider variability, and routes to the next step. It is front-loaded with the core purpose ('List the FAILURE SCENARIOS') and the critical usage instruction ('Call this BEFORE claiming an integration works'). Slightly verbose in the middle with the example list, but that list adds concrete value for an agent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with one parameter, 100% schema coverage, and annotations covering safety, the description is nearly complete. It explains what the return value contains (each scenario's name and what it does), which is useful since there is no output schema. It also provides the workflow context (list then set_scenario). It doesn't mention error cases or empty results, but those are minor for a discovery tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents the single parameter sandbox_id. The description mentions sandbox_id indirectly by referencing 'this sandbox' and the sibling set_scenario, but doesn't add new semantic detail about the parameter beyond what the schema provides. Baseline 3 is appropriate since the schema carries the parameter documentation burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('List') and resource ('FAILURE SCENARIOS this sandbox can inject on demand'), and explains what these scenarios are (specific ways the real provider misbehodes in production). It distinguishes itself from siblings by framing the tool as the discovery step before using set_scenario, and by noting that scenario names differ per provider, so listing is necessary rather than guessing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: 'Call this BEFORE claiming an integration works.' It explains why the happy path is insufficient and that the tool should be used to discover provider-specific failure scenarios. It also names the sibling set_scenario as the next step ('Arm one with set_scenario'), providing clear routing between related tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_specsA
Read-onlyIdempotent
Inspect

Browse the FetchSandbox spec catalog — every API (Stripe, GitHub, Twilio, Notion, OpenAI, Polar, GitLab, and 40+ more) that has a ready-to-use sandbox with curated workflows. Use when the user asks 'what APIs do you support?', 'what specs are available?', 'show me the catalog', 'do you have ?', or wants to explore before committing to one. Returns each spec's slug (use as name arg to import_spec), description, endpoint count, and tags. Pass filter to narrow by substring (e.g., filter='pay' returns Stripe, Paddle, Polar).

NameRequiredDescriptionDefault
filterNoOptional case-insensitive substring filter. Matches against spec name, slug, description, and tags. Omit to return the full catalog.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already carry the safety profile (readOnlyHint=true, idempotentHint=true, destructiveHint=false), and the description adds meaningful context beyond that: the return structure (slug, description, endpoint count, tags) and how the slug feeds into import_spec. OpenWorldHint ('40+ more') is honored without contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Resource is front-loaded first, then usage triggers, then return format, then filter example — a logical, scannable order. Slightly long, but each sentence contributes (concrete API names, query phrases, downstream slug usage, filter example), so no waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A low-complexity tool (one optional param) with no output schema, so the description rightly takes on the return-format burden, which it meets. The only minor gap is no mention of result limits or pagination, but that's immaterial for a simple catalog browse tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% — the schema already fully documents the filter param (case-insensitive substring, matches name/slug/description/tags, omit for full catalog). The description's filter example (filter='pay') is a nice illustration but adds little beyond what the schema states, so the high-coverage baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a precise verb ('Browse') and a specific resource ('FetchSandbox spec catalog') with concrete examples of covered APIs (Stripe, GitHub, Twilio, etc.). It implicitly differentiates from siblings like list_workflows (workflows, not specs) and import_spec (the consumer of the returned slug).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit query triggers ('what APIs do you support?', 'do you have <X>?') and the decision context 'before committing to one', which routes the agent to import_spec afterward. This is as explicit as it gets for a browse tool without listing every sibling as a negative case.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_workflowsA
Read-onlyIdempotent
Inspect

List the named, runnable workflows for a known provider slug or imported spec. Workflows are realistic multi-step API journeys (e.g. 'create customer → attach payment method → create subscription'). Known provider slugs need no import. Use this for exploration ("what can I do?", "show me the flows") OR before run_all_workflows when the user wants a SCOPED validation: list, filter by user intent ("checkout", "webhooks"), then pass the matching ids as workflow_names to run_all_workflows. Default results are summaries. Pass workflow_name with an exact returned id or name to inspect that workflow's full definition, including request bodies, assertions, outcomes and invariants; this does not execute it. Returns the workflows AND the failure scenarios this spec can simulate — each with a plain description and the exact call to run it. When cross_service_available is present, use each exact run_with invocation: cross-provider workflows use the returned namespace, not this provider's slug. These curated workflows exercise provider behavior; they do not verify the user's app. Choose relevant scenarios from the inspected workflow and user goal; ask only about ambiguous business requirements. Do not guess scenario names.

NameRequiredDescriptionDefault
spec_idNoThe spec_id returned by import_spec.
spec_slugNoA human name like 'resend' or 'stripe'. Use this when you do not have a spec_id — you do not need to look one up first.
workflow_nameNoExact workflow id or name returned by this catalog. Returns its full definition, including steps and outcome rules, without executing it; omit for summaries.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description goes beyond them: inspecting via workflow_name does not execute, default results are summaries only, output includes runnable failure scenarios, and cross-provider workflows must use the returned namespace rather than the provider slug. Auth requirements and any result limits/pagination are not mentioned, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and the primary usage modes are front-loaded, and most sentences carry operational information. The closing sentences about choosing scenarios and asking about ambiguous business requirements drift toward agent procedure rather than tool behavior, making the block longer than strictly necessary.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must describe returns, and it does: summaries by default, full definitions (request bodies, assertions, outcomes, invariants) when workflow_name is passed, plus the failure-scenario catalog with exact invocation instructions. For a zero-required-param, read-only listing tool this is complete enough to call correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds genuine meaning: 'Known provider slugs need no import' tells the agent it can skip import_spec, and the cross-provider note clarifies that spec_slug must be replaced by the returned namespace in that case. The workflow_name explanation largely restates the schema text, which caps the score.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource ('List the named, runnable workflows') and defines what a workflow is with a concrete example journey. It clarifies scope by noting it does not execute anything, but it never names or contrasts against the actual siblings (run_workflow, quickrun, validate_integration), so differentiation is implied rather than explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use conditions ('exploration' vs 'scoped validation before run_all_workflows') and a concrete sequencing recipe: list, filter by user intent, then pass matching ids as workflow_names. It also states the condition for the detail mode (pass an exact returned id/name) and warns not to guess scenario names.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

manage_tunnelAInspect

Connect the actual local disposable test app to the FetchSandbox verifier, without installing ngrok or sharing an account token. Local stdio coding agents only; hosted builders use their published app origin. Steps: (1) inspect the real app, its provider clients, existing webhook handler route, signature verification and saved-state read-back; never invent a substitute webhook route or remove signature checks. (2) Start that app on a loopback port and call action=open,port. (3) Call validate_integration with providers and the returned app_base_url; retain session_id. (4) Inspect coverage_policy, propose application checks, prepare/read back fixtures, and configure suite=application_workflow_v1. Register the exact inspected webhook path in application_config.event_destinations, e.g. stripe:/stripe-webhook ONLY if that is the real handler. For Stripe purchase verification, set application_config.event_delivery=provider_lifecycle. FetchSandbox registers this exact tunnel route using the isolated Stripe twin POST /v1/webhook_endpoints, returns registration metadata, and supplies its signing secret through the protected handoff. Drive the real app order/payment entry points and confirm/capture its actual PaymentIntent; do not replace the purchase with synthetic success events. Assert saved order and email effects. Other provider lifecycle registrations remain explicit gaps; existing hosted receipt suites keep their current path. Direct signed-event mode remains for targeted event tests and is not full purchase lifecycle proof. Use provider-native event types/payloads and exact event-ID replay. (5) Use the native protected handoff: call install_test_credentials with session_id/run_id and inspected env_mapping; load its private dotenv file into the disposable app without reading or printing it. Configure temporary provider base URLs; preserve production configuration. The relay credential is internal and never goes to the app. (6) Preflight, execute ONCE, then poll the same session_id/run_id and retrieve its receipt. Independently assert order/state and receipt effects across replay, concurrent/stale/delayed events, invalid signatures and relevant provider failures, response loss and restart. HTTP 200 or connectivity alone is not proof; retain unsupported/unmeasured checks. Compare original buggy and fixed app versions under the same contract. (7) Close the tunnel and clean temporary settings. If local startup/connectivity fails, retain the blocker and use the existing Phalanx prove_fix fallback. This temporary address is verifier-only, not a public website. Direct browser/curl requests deliberately return 404 even when the relay is healthy. Do not treat that 404 as failed connectivity, inspect FetchSandbox internals, expose relay credentials or switch verification paths because of it. After configuration and credential installation, use validate_integration preflight to test actual app connectivity; manage_tunnel status checks only the transport. A disconnect or backend restart ends the lease; do not automatically replay uncertain writes.

NameRequiredDescriptionDefault
portNo
actionYes

TDQS

A3.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite annotations covering safety hints, the description adds substantial behavioral context that structured fields cannot: the deliberate 404 for browser/curl, that status only checks transport while validate_integration tests real connectivity, that a disconnect or backend restart ends the lease, and that the relay credential is internal and must never be exposed. These are exactly the operational traits an agent needs and are not derivable from the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

This is a ~700-word playbook for a two-parameter tool, not a tool description. While the numbered steps (1)-(7) give some shape, large stretches (fixture read-back, replay edge cases, buggy-vs-fixed comparison) belong in workflow documentation, and the sheer volume obscures the front-loaded purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex orchestration tool with no output schema, the description is thorough: it names configuration fields (application_config.event_destinations, event_delivery), the execution/polling cycle, receipt retrieval, and cleanup. It is arguably over-complete, but nothing an agent needs to call the tool correctly appears missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for both parameters, so the description carries the load. It does mention action=open with a port and distinguishes status ('checks only the transport') and close ('Close the tunnel'), covering the enum values implicitly, but it never explains port constraints (1024-65535), what open returns, or the exact effect of status/close, leaving gaps the schema cannot fill.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb+resource+goal: connect a real local disposable test app to the FetchSandbox verifier without ngrok or an account token. It contrasts with siblings like install_test_credentials and validate_integration implicitly, but the actual purpose is then buried under hundreds of words of workflow steps, so an agent must dig to extract it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Rich when/when-not guidance: 'Local stdio coding agents only; hosted builders use their published app origin,' the prove_fix fallback when local startup fails, and explicit warnings not to treat deliberate 404s as failed connectivity. Alternatives are named (validate_integration preflight, direct signed-event mode, hosted receipt suites), though they are tangled into numbered steps rather than stated as selection rules.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

prove_fixAInspect

Long work returns status:running and next_tool_call:get_job within a short response. Follow that SAME job until done; never repeat this tool to poll. THE PROOF STEP. The only way to establish that a fix actually works, and produces an own-code proof receipt. Ship the STILL-BUGGY project plus your proposed diff: FetchSandbox applies the diff to a copy and runs the SAME failure probe against the buggy and the fixed tree. Inspect the receipt's provider modes: a recording double, a twin and an original provider are different evidence scopes. Green ONLY on a measured flip — the bug demonstrably present before, demonstrably gone after. CALL THIS BEFORE YOU WRITE THE FIX TO DISK. prove_fix needs the broken tree to reproduce against; applying the diff first makes a measured proof impossible. DO NOT SUBSTITUTE YOUR OWN TEST. A harness you write runs only the FIXED code, so it can show a fix passing while proving nothing about whether the bug was ever reproduced or is genuinely gone — and an agent-authored check must never be reported to a user as 'proven' or 'verified'. If you have already written one, still call this: your test and this proof answer different questions.

NameRequiredDescriptionDefault
bugNoThe finding being fixed AND the user's acceptance criteria: required app operations, expected state, ownership boundaries, wire types/units and readback checks. Preserve the requested scope; do not reduce a workflow request to a single symptom. Used to select/generate the probe.
diffYesThe unified diff returned by fix_bug.
pathNoOPTIONAL. The project root, as it is. Defaults to cwd. Do NOT copy the project first: prove_fix builds its own before/after copies and never writes to your tree. A path outside the workspace is refused.
scenarioNoOPTIONAL. Explicit scenario id, if known.
timeout_sNoOPTIONAL. Budget in seconds (default 300).
sandbox_idNoOPTIONAL. A twin from quickrun/run_workflow. Usually omit — the last twin this session used is attached automatically.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: long-running calls return status:running plus next_tool_call:get_job and must be followed rather than re-polled, prove_fix builds its own before/after copies and never writes to the caller's tree, paths outside the workspace are refused, and green only occurs on a measured flip. The readOnlyHint=false/destructiveHint=false pairing is consistent with the 'runs in its own sandbox copy, does not touch your tree' behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the async-job handling rule, then the proof semantics, then the prohibitions, in a logical order. It is long and caps-heavy with some repetitive emphasis ('DO NOT SUBSTITUTE YOUR OWN TEST' restated in the following sentence), which costs a point.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex, open-world, six-parameter tool with no output schema, the description covers the execution model, the diff-application flow, the receipt's provider-mode evidence scopes, the preconditions for an honest proof, and the async continuation rule. Nothing material to calling it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all six parameters; baseline would be 3. The description nevertheless adds real meaning: the bug field carries the acceptance criteria and scope, diff is the fix_bug output, path must not be a copied project, and sandbox_id is usually omitted because the last twin is auto-attached.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: it proves a fix by applying the diff to a copy and running the same failure probe before/after, emitting an own-code proof receipt. It is clearly distinguished from siblings like fix_bug (which writes the fix), submit_proof and verify_behavior.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use: call before writing the fix to disk, with the still-buggy tree plus the diff. Explicit when-not and alternatives: do not substitute an agent-authored test, and call this anyway if one exists, because the two answer different questions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

quickrunAInspect

Run a curated reference workflow against a KNOWN, bundled spec (stripe, clerk, descope, resend, twilio, and 50+ others) in ONE call — it spins up the sandbox by slug, so you do NOT need import_spec or a sandbox_id first. Use this when exploring the bundled provider contract: call guide on their prompt, then call quickrun with the returned spec + workflow. To reproduce a failure, pass the scenario from guide's matched_bug_pattern.reproduce_with.scenario (e.g. webhook_retries, payment_declined). Returns sandbox_id + flow_run_id — pass BOTH to verify_behavior to inspect the reference behavior — plus a receipt URL. This does not execute or prove the caller's application. For a bug in their project, use find_bugs, fix_bug and prove_fix on that project; do not substitute this reference receipt for an own-code proof. Use run_workflow instead ONLY when you already hold a sandbox_id from import_spec of a custom/private spec.

NameRequiredDescriptionDefault
scenarioNoOPTIONAL failure scenario to reproduce (e.g. webhook_retries, payment_declined). Take it from guide's matched_bug_pattern.reproduce_with.scenario. Omit for the happy path.
spec_slugYesThe bundled spec slug from guide (e.g. 'stripe'). Lowercase.
workflow_nameYesThe workflow id from guide (e.g. 'accept_payment').

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations by disclosing the side effect (it spins up the sandbox by slug, so no import_spec/sandbox_id is needed) and the return payload (sandbox_id + flow_run_id + receipt URL). It also draws a hard scope boundary: 'This does not execute or prove the caller's application.' It stops short of mentioning auth, cost, or sandbox lifetime, so it is strong but not exhaustive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action and its key advantage ('in ONE call'), then flows through prerequisites, reproduction, return values, and exclusions in a logical order. It is dense and slightly long, but nearly every sentence carries routing or behavioral information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema, the description names what is returned and what to do with it (pass both ids to verify_behavior), covers the prerequisite step (guide) and the alternative tool (run_workflow), and states the tool's scope limits. An agent has everything needed to select and call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description does add relational context (values come from guide's output) and a concrete scenario example, but the schema descriptions already state 'from guide', so the added value is marginal.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Run a curated reference workflow against a KNOWN, bundled spec ... in ONE call') and immediately scopes it to bundled specs (stripe, clerk, descope, etc.). It explicitly distinguishes itself from sibling tools run_workflow, import_spec, find_bugs, fix_bug and prove_fix, so an agent can route without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit ordered flow ('call guide on their prompt, then call quickrun with the returned spec + workflow'), names the failure-repro path via guide's matched_bug_pattern.reproduce_with.scenario, and states the one condition under which run_workflow is the correct alternative (already holding a sandbox_id from import_spec). It also states when NOT to use it (proving a bug in the caller's own project).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_all_workflowsAInspect

Execute EVERY workflow (or a scoped subset) for a sandbox in ONE call. Use this — NOT a loop of run_workflow — for any validation-style request: "validate this integration", "run all workflows", "check coverage", "test stripe checkout", "fs validate". IDEs (Cursor, Claude Code) approve each MCP call individually, so 18 workflows via run_workflow = 18 clicks. This tool = 1 click, total. Scope via workflow_names: pass an array of workflow ids to run a subset (e.g., user says "validate stripe CHECKOUT" → pass ["create_checkout_session", "checkout_complete"]). Names are case-insensitive; dashes and underscores interchangeable. Returns: summary (pass/fail counts, totals) + full step trace per workflow. After running, the user can visit fetchsandbox.com/runs/<sandbox_id> for a shareable visual timeline.

NameRequiredDescriptionDefault
sandbox_idYesThe sandbox_id returned by import_spec.
workflow_namesNoOptional list of workflow ids/names to run. Case-insensitive; dashes and underscores are interchangeable. Omit to run every workflow for the spec.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only provide flags (readOnlyHint=false, destructiveHint=false), so the description must fill in behavior. It discloses that this is a write/mutating operation (runs workflows), reduces UI approvals, and returns a summary plus full step trace. It also mentions the shareable timeline link. Slightly less than 5 because it doesn't detail failure behavior, but it covers key points.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is information-dense and front-loads the primary use case early with 'Execute EVERY workflow... in ONE call.' It then explains usage, scoping, and returns concisely. Slight padding with the IDE click explanation could be trimmed but serves a purpose for user context. Overall, it's structured effectively.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (batch execution), the description provides sufficient usage context, scoping options, and return information. It lacks a detailed list of possible failure modes or timeouts, but since there's no output schema and annotations are minimal, it does well. A 4 is justified as it could be more complete with error handling, but it's solidly above average.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, covering sandbox_id and workflow_names. The description adds value by explaining how workflow_names scopes the subset with examples, and clarifies case-insensitivity and character interchangeability, beyond the schema's brief notes. It reinforces the semantics, which is helpful. A 4 is appropriate since it's not adding entirely new meaning but contextualizes the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool executes every workflow (or a subset) for a sandbox in one call, and distinguishes it from run_workflow. It provides a strong verb (execute), specific resource (workflows for a sandbox), and concrete use cases (validate, run all workflows), making its purpose unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly tells users to use this instead of a loop of run_workflow, and gives concrete examples of when to use it ('validate this integration', 'run all workflows', etc.). It also explains how to scope with workflow_names, including case-insensitivity and dash/underscore handling, so an agent knows exactly when and how to invoke it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_workflowAInspect

Execute ONE specific workflow by name and return its step-by-step trace PLUS a share_url — a public, replayable proof URL that renders the full timeline (every request, response, webhook event) for this run. The share_url is the canonical 'here's what happened' artifact: surface it verbatim in any reply that needs evidence (PR comments, Slack threads, blog posts, X replies). Do NOT substitute a docs URL or any other link as the proof — the share_url is the only valid receipt. Use ONLY when the user explicitly names a single workflow to run (e.g., "run accept_payment", "just check the refund workflow"). For ANY validation-style request — "validate stripe", "check coverage", "run all workflows", "test this integration", or even "validate stripe checkout" (multiple workflows match "checkout") — use run_all_workflows instead. The batch tool collapses N approvals to 1 and supports a workflow_names filter for scope. Calling this in a loop is an anti-pattern.

NameRequiredDescriptionDefault
scenarioNoOPTIONAL failure scenario to exercise (e.g. payment_declined, insufficient_funds, fraud_hold). Toggles the sandbox engine's scenario for the duration of the run, then restores. Use this for 'test with declined card' / 'simulate failure X' intents. Omit for the happy path.
sandbox_idYesThe sandbox_id returned by import_spec.
workflow_nameYesWorkflow id or name from list_workflows. Case-insensitive; dashes and underscores are interchangeable.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, openWorldHint=true, idempotentHint=false, and destructiveHint=false, and the description is consistent with all of them (execution implies side effects; webhook events align with openWorld). The description adds genuine value beyond annotations: the scenario toggling behavior (set for the run's duration then restored), the canonical status of share_url, and the instruction not to substitute a docs URL as proof. No contradictions found.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is longer than average but every sentence earns its place — primary action first, then output semantics, then usage rules and alternatives. It is front-loaded with the core action before the routing guidance. Slightly verbose in the exclusion list, but dense with actionable information rather than filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema present, the description correctly takes on the burden of explaining the return value (trace + share_url) and its role as the proof artifact. The tool is simple (3 params, 2 required, no enums, no nesting) and the description fully covers invocation, output, and routing — an agent has everything needed to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with each parameter (scenario, sandbox_id, workflow_name) already well documented including case-insensitivity and scenario-restore semantics. The description reinforces the workflow_name routing and scenario intent but does not add parameter details beyond what the schema provides, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('Execute ONE specific workflow by name') plus the precise output contract (step-by-step trace and a share_url proof artifact). It clearly distinguishes itself from run_all_workflows by contrasting single-workflow execution against batch validation, so an agent can tell them apart without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Extremely explicit about when to use it ('Use ONLY when the user explicitly names a single workflow') and when not to, naming the alternative (run_all_workflows) with concrete examples of ambiguous intents like 'validate stripe checkout' where multiple workflows match. Also flags calling this in a loop as an anti-pattern. Nothing is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

set_scenarioA
Idempotent
Inspect

ARM a failure scenario on a sandbox, so the next requests hit the provider misbehaving instead of the happy path. This is how you find out whether the integration you just wrote actually survives production. Use it like this: run the flow once on 'default' and confirm it works, call set_scenario with a failure from list_scenarios, run the SAME flow again, then check the end state is still correct — not that the call merely returned. A declined payment must not produce a confirmed order; a webhook delivered twice must not charge twice or send two emails. Set scenario to 'default' to return the sandbox to normal. The scenario stays armed until you change it, and it affects every caller of that sandbox — so put it back when you are done.

NameRequiredDescriptionDefault
scenarioYesA scenario name from list_scenarios, or 'default' to restore normal behaviour. Guessing a name that does not exist fails — list first.
sandbox_idYesThe sandbox to arm. From quickrun, run_workflow or import_spec.

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate non-read-only, open-world, idempotent, and non-destructive. The description adds that the scenario persists until changed and affects all callers of the sandbox, which is critical behavioral context beyond annotations. It also emphasizes the importance of resetting, fully disclosing side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single paragraph of about 150 words, covering purpose, usage, restoration, and warnings. While not as tight as a two-sentence description, every sentence contributes to correct usage, so it is appropriately sized and front-loaded with the core action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with two parameters and no output schema, the description covers the intended workflow, restoration, and side effects. It is complete for an agent to know when and how to invoke it correctly, including the need to verify end state and reset after testing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already provides thorough descriptions for both parameters, including valid values for scenario and failure behavior on unknown names. The description adds usage context but no new parameter-specific semantics, so the baseline of 3 is appropriate given 100% schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb (ARM) and resource (failure scenario on a sandbox), clearly distinguishing it from list_scenarios. It explains the purpose of testing integration resilience, leaving no ambiguity about what the tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit step-by-step workflow: run on 'default', arm a failure from list_scenarios, re-run, verify end state, and restore with 'default'. It also warns about side effects and the need to reset, making when and how to use it unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_proofAInspect

Receipts are readable by anyone with the link — do not attach bodies the user would not paste into a public PR. Attach REAL before/after evidence from the USER's actual app to the run receipt. Use this AFTER you've fixed the bug and re-run the app: it makes the receipt show their own code's behavior (before your fix vs after), which is far stronger proof than a reference simulation. Provide one proofs entry per probe you fired — the request you sent, the response BEFORE the fix (bug reproduces), and the response AFTER the fix (bug gone) — keyed to the run's sandbox_id + flow_run_id (from run_workflow's result). This is SELF-REPORTED evidence: it's shown on the receipt but never counts as a proven (green) result. For a MEASURED green proof — FetchSandbox reproduces the bug on your real code and verifies your fix flips it — use prove_fix instead.

NameRequiredDescriptionDefault
proofsYesone per probe: { name, request:{method,path,body}, before:{status,body}, after:{status,body} }. before = pre-fix (bug reproduces), after = post-fix (bug gone).
summaryNooptional one-line description of what you ran against the app
sandbox_idYessandbox_id from the run_workflow result
flow_run_idYesflow_run_id from the run_workflow result
bug_pattern_idYesthe bug_pattern id you reproduced (e.g. webhook_duplicate_side_effect)

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnlyHint=false and destructiveHint=false, but the description adds critical behavioral context: this is self-reported evidence that 'never counts as a proven (green) result'. It also discloses that receipts are public and the submission is non-authoritative. This goes well beyond annotations and prevents misinterpretation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is detailed but every sentence earns its place: it front-loads the privacy warning, then gives usage timing, then explains the self-reported nature and contrast with prove_fix, then parameter guidance. It is structured logically and avoids redundancy with the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, nested objects, and no output schema, the description covers everything an agent needs: when to use it, what to attach, how to structure proofs, keying to run IDs, and its non-green status. The presence of an alternative (prove_fix) is clearly stated, so the agent can make an informed choice.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameters are already documented. However, the description adds operational semantics by instructing 'Provide one proofs entry per probe you fired' and explaining the expected structure (request, before, after) and keying to sandbox_id + flow_run_id. This adds value beyond the schema's field definitions, though it could have gone further with concrete examples.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the tool's purpose: to submit self-reported proof of a bug fix by attaching real before/after evidence from the user's app to the run receipt. It clearly distinguishes itself from the sibling tool 'prove_fix' by contrasting self-reported vs measured green proof, so an agent can tell them apart without opening schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides explicit when-to-use guidance: 'Use this AFTER you've fixed the bug and re-run the app'. It also names the alternative (prove_fix) for measured green proof and explains the condition (when you want FetchSandbox verification). Additionally, it warns about public visibility of receipts, which is an exclusion condition for sensitive content.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

validate_integrationAInspect

Completion rule: a request to verify an app includes its applicable failure checks without another user prompt. When verification_continuation.required is true, continue in this agent: inspect pending_checks, preserve every declared check and its assertions, implement supported actions, and rerun a fresh attempt. Stop only with measured results or concrete blockers showing the missing capability and attempted evidence. Not run, not armed, and no observer yet are work to finish, not unsupported features. Do not clean up the session or present a final proof while actionable checks remain. Ask the owner only for genuinely missing business decisions, credentials or actions you cannot perform. For a local app, start the actual app and call manage_tunnel with its port; use the returned app_base_url. No ngrok install is needed. The managed relay origin deliberately rejects direct browser/curl requests with 404; after configuration and credential installation, test connectivity with validate_integration preflight instead. Do not interpret that private-origin 404 as an app failure or expose relay credentials. Hosted builders use their published origin. Start with providers (and optional app_base_url), then reuse the returned session_id for application_context, fixtures and attempts. Do not include application_context in the initial session call. Inspect how the actual app or workflow executes before choosing a verification path. If the builder can run that workflow locally, bind its real provider clients to every returned twin URL and the supplied test authentication. Use the returned provider operation contracts to implement those clients; a missing canned workflow does not mean the provider lacks an API operation. The existing arm/probe path does not require a published app or webhook: select a probe from the returned matrix, call arm with session_id, send its request_headers on EVERY provider request across the session, execute the actual workflow entry point, then call probe with the same session_id and returned run_id. Independently assert saved app state in the host and report that evidence separately. Provider traffic alone remains app_verified:false; it does not establish application execution or state. Do not substitute a standalone simulation, direct twin calls or a curated workflow for the user's implementation. A public origin is required when FetchSandbox must drive or observe an HTTP application suite; the hosted setup instructions below apply to that path. For hosted application_workflow_v1 apps, build and publish the disabled test adapter before requesting short-lived credentials. After installing the existing handoff and republishing, call preflight:true with session_id/run_id. A setup_blocked response is not a failed application test. Its next_tool_call is null: repair its exact host-side blocker first, then call resume_tool_call for the SAME run/preflight without rotating credentials. Do not poll an unchanged setup blocker. Execution has a separate deadline. Execution is already non-blocking: execute:true starts the existing backend job and returns running promptly. Use the SAME builder agent to follow status; no extra background helper or helper approval is needed. Never execute twice; retrieve status with session_id/run_id only. If the browser stops at a security challenge, immediately hand the owner the SAME run/order-verified original TEST checkout URL (livemode=false) while observation continues; never bypass the challenge, silently wait, create a replacement checkout or retry Pay. For an owner-assisted browser step, use ONE native question containing that verified URL, pause the builder for its answer while the backend job continues, and record that answer once. Do not keep working while asking for a plain chat reply that the host queues, or ask again after confirmation is recorded. Browser payments require an available browser or one explicit user action, and independently observed paid state; a click is not proof. After proof, remove only temporary settings, republish and verify disabled externally. When asked to reassess or recontract, set reassess=true and supply application_config.scope_review for every candidate check. Record outside-scope features separately, and keep relevant missing or failed behavior required. Scope is declared by the app-owning agent from its inspected code and requirements, not guessed from a provider name. For every app, inspect capabilities, coverage_policy and application_actions. Review every supplied coverage dimension against inspected app behavior without waiting for the user to enumerate failure tests. Supply application_config.coverage_review with relevance, reason and linked check IDs when the builder authors the plan. Missing dimensions stay unreviewed. A dimension review is not execution proof; never claim all possible invariants are known. Prepare missing test products, prices and related records before failures: author fixtures using documented provider operations, include a final GET with an exact identity assertion, then configure the app with returned fixture IDs. The server uses the existing twin workflow runner and independently checks read-back. Fixture readiness is not app proof. Unsupported application checks must be reported explicitly, not replaced by a separate script or quickrun. Use the returned fixture_authoring contract, not a web search for assertion syntax. Prepare fixtures before configuring any attempt. Fixture assertions are GET-only and contain select, equals (string), required:true, count:1. Use separate asserted GET steps for each additional value; do not use application all_of or where predicates in fixture assertions. Paddle selectors include the response envelope, e.g. data.id; Stripe uses id. Read the provided Paddle one-time-price example and adapt its product and amount to the app. If fixed receipt-suite transactions cannot match the app's product, price and pending order, do not execute them as app proof or a pointless rejection diagnostic. Report that compatibility gap, cancel any unexecuted mismatched attempt, then use application_workflow_v1 for supported checks while preserving untested receipt rules. After session creation, send session_id and application_context with the original goal and inspected workflow to obtain model-proposed rules, missing observations and fixture requirements. This planning action requires sign-in. Proposals are not executed checks. The app-owning agent authors the executable contract from inspected code and business scope. Server suggestions are advisory. Review every coverage dimension; declare relevant unsupported behavior explicitly and explain genuinely unrelated features as not_applicable. Omitted suggestions are retained separately and cannot add required checks. If server_requirements_model_available is false or planning returns requirements_model_unavailable, inspect the app and author the candidate contract with your own model; label it builder-proposed, not a FetchSandbox model result. Never wait for the human to write routine test requirements. For a Stripe app on a managed tunnel or published origin, use application_config.event_delivery=provider_lifecycle with the exact Stripe handler in event_destinations and require_verifier_token=true. Configuration creates a real subscription on this isolated Stripe twin via POST /v1/webhook_endpoints; the protected handoff supplies its returned signing secret. Select a returned catalog lifecycle profile with application_config.provider_lifecycle_workflows, e.g. {stripe:'create_and_send_invoice'} for invoice.sent or {stripe:'accept_payment'} for payment_intent.succeeded. The server freezes the workflow's subscription events, witness types and provider readback; missing delivery or mismatched provider state cannot pass. Drive that workflow through the inspected app's actual entry point. Do not manufacture a payment for an invoice or another non-payment workflow. Without a selection, the legacy payment/email witnesses remain unchanged. For the payment profile, confirm/capture the app's returned PaymentIntent and observe saved order plus email effects. Keep payment and duplicate replay in separate checks: first confirm payment and poll independently saved paid order and receipt state; then GET the actual received success-event ID from app state and bind replay_of to that GET response. Never assume the payment response contains event_id or use the PaymentIntent ID as an event ID. If the app cannot expose a received event identity, retain replay as unmeasured and explain the missing observation. Synthetic event type/payload actions are refused in this mode; exact replay of a delivered or explicitly withheld run-correlated generated event is supported. Event actions accept concurrency from 1 to 8 for real overlapping deliveries. Use the returned provider_control and fault.evidence contracts to drive recorded fault recovery. Use fault.evidence=accepted_response_retry when the app recovers a lost accepted response inside its HTTP request or a successful webhook callback. It requires an observed client disconnect before a matching provider retry and the same accepted effect ID; simultaneous sends and a slow response cannot pass. accepted_response_loss remains the failed-callback plus redelivery contract. For a declared client_lifecycle restart, follow client_checkpoint by restarting the real app process with the same durable storage and bindings, then retrieve the same run without another execute. Inspect webhook_subscriptions and recorded callback responses; missing delivery is inconclusive, never a pass. Resend lifecycle can be configured alongside Stripe using its exact existing handler in event_destinations. Configuration calls the isolated Resend POST /webhooks with endpoint/events and supplies the returned Svix signing secret through FETCHSANDBOX_RESEND_WEBHOOK_SECRET; map it to the inspected app setting. The real email send emits email.sent, email.delivered or the armed native email failure event. Inspect the signed callback acknowledgement and independently saved email state; a send ID or HTTP 200 alone does not prove application handling. Other providers remain explicit lifecycle gaps; existing hosted suites are unchanged. Local coding agents should call install_test_credentials after configuration to securely install temporary secrets using their MCP account key; do not call the browser receipt_secrets endpoint with an API key or print secret values. Hosted builders keep the existing owner browser handoff. For a normal HTTP app workflow with independently readable state, configure application_workflow_v1 and application_config. Declare a positive baseline first using ordinary HTTP app actions, no signed events, GET observations and required exact assertions. A pending order is a valid baseline; prove signed payment and access in a later check without removing those assertions. HTTP actions are {method,path,body?,expect_status?,concurrency?}. concurrency from 1 to 8 sends the same inspected app request concurrently against frozen prior bindings; all statuses are checked. For concurrency greater than 1, the response slot is {results:[...]} in request declaration order: bind each actual identity with {{step1.results.0.id}}, {{step1.results.1.id}}, then independently GET saved state. Same input does not imply the same logical operation; use the app business contract to distinguish independent purchases from duplicate submission. Every concurrent request consumes the existing check/run action budget; signed-event actions are ONLY {event:{provider,type,payload},expect_status?}, or {event:{provider,replay_of},expect_status?} for exact replay. Never combine event with method/path/body/headers or invent method EVENT. Webhook paths belong in event_destinations and MUST be the inspected app handler paths. Provider event names, resource envelopes, signing algorithm, raw request bytes and replay event identity must follow that provider. Do not invent a test webhook handler that replaces the real implementation; a protected setup/state adapter may expose isolated records but may not implement the behavior being tested. Managed relay URLs register verifier connectivity only, not a live-provider subscription. All action, before and observe paths must omit query strings and fragments, even run_id queries: use inspected protected path-segment routes and preserve run scoping. A configuration_rejected result identifies offending field paths; repair those fields in the SAME session before another configuration call. It is not an app test result and does not request credentials. Reuse applicable proposed rule IDs when useful; only app-declared checks are binding. Do not drop a relevant failed check to obtain green. Signed callback app_response contains a sanitized acknowledgement; response remains delivery metadata. Inspect response_capture_status when the body is missing, and independently read state for the verdict. Signed provider event actions use isolated secure-handoff signing secrets; bind payload identity to actual app responses and verify saved state. No browser, arbitrary auth-header or SQL execution is supported by this HTTP contract: record those gaps explicitly. Do not weaken signature or access checks to make a test pass. Start a validation session for the user's integration. quickrun and run_workflow execute OUR curated workflow against a twin — green there says the provider behaves as documented, and says nothing about whether their integration wired to it correctly. This instead hands back a FRESH twin per provider and asks the user to point their app at it, then reads external provider traffic. Direct agent calls also appear there; caller identity is not established. Use it when someone wants their integration verified before launch. Call once with providers, then follow the returned next_action. Inspect the app and supply a complete receipt_config before configuring the suite; the application suite drives its own purchases and webhooks, so do not ask the user to run a manual flow first. The matrix is returned before any traffic: pass arm with a probe ID before exercising the app. Send the returned request_headers on every provider request in that attempt, then pass probe and run_id to evaluate. Use cancel: true with run_id to abort or retry failed cleanup. Request logs alone do not prove application state or production readiness. For a supported payment-to-receipt integration, follow the structured next_action; call next_tool_call only when it is non-null and its arguments are complete. Never execute while host-side setup is pending. Do not substitute guide, quickrun, list_workflows, or reference probes for application verification. Use the widest suite the server recommends. receipt_recovery_v1 includes R1-R4 and adds checks R5-R8 for accepted-send response loss, concurrent redelivery, source/email transient failures and delayed redelivery. Pass suite plus receipt_config with the app's exact webhook URL, test recipients and observable purchase marker. FetchSandbox freezes all four receipt rules for both same-customer and different-customer purchases. You are responsible for completing this setup: treat the target as a disposable test app, not the user's live production service, until its declared suite passes. For a greenfield app, keep live billing disabled and publish a temporary test build first to obtain its stable public origin. Use the published webhook URL, not a private preview URL. Inspect the app and identify the exact webhook route before suite configuration. Before requesting or applying test secrets, send an unsigned POST to that exact URL from outside the workspace; a 400 signature rejection is expected, while a 307/login redirect is a blocker. Keep signature validation enabled. After the successful preflight, configure the suite with its complete receipt_config. Apply the returned twin URLs, X-Flow-Run-Id, and three temporary credentials only to this disposable app; execute only after the host-side setup actions are complete. The credentials are twin-only test values: PADDLE_API_KEY, RESEND_API_KEY, and PADDLE_WEBHOOK_SECRET. Use the platform's supported secret store or secret-writing tool and never use real provider credentials or enable live billing. After every required check is held, remove only the test secrets and twin config this run added, including from the disposable test app's published scope. Never modify a separate live production app or delete pre-existing, shared, or real-provider values. Confirm the temporary names are absent, then hand the app to its owner to configure separate development/staging credentials. If the suite is failing or inconclusive, retain test config for repair and a fresh run. The configure_app response contains public twin URLs/IDs and a secret_handoff_url, never the secret values themselves. If the host exposes no safe way for you to write test secrets, explain that specific limitation and show the human the link to open while signed in to the owning FetchSandbox account; they can copy the three values directly into the platform secret store. Do not ask the human to discover endpoints, call FetchSandbox manually, or paste secret values into chat, project files, or logs. Wait for confirmation only when the human must complete that secure-store action. Then call execute:true with its run_id. FetchSandbox drives signed events and independently observes accepted email records. Keep every check and the observation windows in your report. Restore any temporary host privacy setting after the test. On an inconclusive execute result, inspect and report its execution_diagnostics field directly; do not ask the human to infer a cause from the public receipt page, which intentionally omits private proof. If you cannot configure the app, report that blocker; reference tests cannot replace app execution. A verified receipt suite covers only its declared rules and windows, never all production behavior. A missing or untriggered recovery fault stays unmeasured. This application suite requires sign-in. Paddle checkout is a separate app-owned browser flow: if a transaction's checkout.url points to fetchsandbox.com/checkout, returns 404, or does not show a checkout page, do not imply FetchSandbox hosts the merchant's payment UI. Inspect the exact URL and the Paddle Sandbox default payment link or per-transaction override. Tell the builder to use a reachable app checkout page that loads Paddle.js and opens the _ptxn transaction, or a supported Paddle-hosted flow. Verify browser checkout/payment separately from twin API and webhook tests; transaction creation alone is not payment proof.

NameRequiredDescriptionDefault
armNoProbe ID from the matrix to arm before running the app. Requires session_id.
probeNoProbe ID from the matrix to evaluate after running the app. Requires session_id and run_id.
suiteNoConfigure the declared application receipt suite on this session.
cancelNoAbort the attempt or retry failed cleanup. Requires session_id and run_id; cannot combine with arm or probe.
run_idNoAttempt ID returned by arm or suite configuration. Required with probe, cancel, execute or preflight; prevents stale calls from consuming a newer attempt.
executeNoDrive the configured app and evaluate the frozen receipt suite. Requires session_id/run_id; configuration acknowledgment alone is not proof. Starts the existing backend job asynchronously and returns promptly; follow status in this agent without allocating a background helper.
inspectNoRead fixture inputs through MCP using session_id. Prefer this MCP read to extracting a key for direct HTTP: the edge can reject a generic Python user agent even with valid ownership. This path keeps the existing MCP client authentication and headers. Do not ask for live provider keys. First resource=state returns collection counts; select collection to read real IDs and relationships with pagination. resource=operations lists provider contracts; specify both exact method and documented path to resolve one schema. Read-only, session-owned twins only, not application proof.
fixturesNoPrepare session-owned twin data before app tests. Requires session_id. Stable id makes repeated calls replay-safe. Every step names spec, method and relative path. Use {{step1.id}} for dependencies. End with GET and an exact identity assertion. No live provider calls. An explicitly advertised /__fetchsandbox/fixtures/time step is an owner-only synthetic clock control, not a provider API; it is unavailable during active attempts and requires normal provider GET readback.
reassessNoSet true when the user asks to reassess/recontract an app. Requires suite=application_workflow_v1 and application_config.scope_review for EVERY candidate check. Freeze a fresh contract scoped to inspected promised behavior. Outside-scope proposals are recorded separately; missing protection on relevant behavior remains a requirement. Never edit historical results or remove a failed requirement for the same app version.
preflightNoRead-only public readiness check for a configured application_workflow_v1. Requires session_id/run_id. Checks frozen empty baseline and token enforcement without creating orders, injecting faults or rotating secrets. Retry same run after its exact setup blocker is repaired.
providersNoProvider slugs the app integrates, e.g. ['stripe'] or ['paddle','resend']. For provider-free HTTP workflows, omit providers and pass app_base_url.
session_idNoReturned by the first call. Pass it back after their app has run a flow to see what it did.
app_base_urlNoApp origin when FetchSandbox must drive or observe a hosted HTTP suite. Optional for a provider-backed workflow executed by the builder through arm/probe; that path does not require a published app or webhook.
receipt_configNo
application_configNoFrozen HTTP app drive/read-back contract: 1-30 checks, 1-20 actions per executable check, at most 300 declared actions in total, below 64 KiB. Preserve intermediate reads within these bounds; do not silently remove them. First check id=baseline must establish success. Checks require id, name, before {path,assertion}, ordinary HTTP actions [{method,path,body?,expect_status?}] or signed-event actions [{event:{provider,type,payload},expect_status?}], observe {path}, and assertion using select/equals/count/required or all_of. where and all_of must be lists of assertion objects, not field maps. equals accepts JSON strings, numbers, booleans and null; numeric/boolean literals preserve type, and null must be a present null field. Stripe event.payload is only the resource inside data.object, never a full event or a data.object wrapper: the server constructs and signs the envelope. Optional per-check observation_deadline_seconds (0-180, repeat=1 only) records bounded polling for eventual saved state; it does not drive a browser or turn a timeout into a pass. Every check first reads its declared starting-state precondition. Baseline requires count=0 on the same observed collection, with no where filter. Use {{run_id}} to scope fresh synthetic records. All paths are app-relative without query strings or fragments, including non-secret run_id queries; preserve run scoping through an inspected protected path-segment observer. Baseline permits only ordinary HTTP actions without signed events; prove signed purchase/access in later checks. Do not combine event actions with method/path/body/headers. {{step1.id}} binds a real action response to read-back. assertion.repeat (1-5) and stable_under_repeat support replay. Optional fault {provider,scenario} requires independently recorded provider refusal. For reassess/recontract include scope_review for every candidate check. Outside-scope feature proposals contain id,name,unsupported_reason only and move to separate scope_exclusions; they are not required tests. In-scope missing work uses id,name,blocker {category,reason,evidence,...}, never a free-text unsupported reason. engine_unsupported requires a capability confirmed by current backend capabilities; a missing app route or observer is not an engine gap. Unresolved policy remains unmeasured. Legacy untyped reasons remain unclassified, not engine unsupported. No user-supplied credentials, auth headers or live data. require_verifier_token=true generates an owner-only temporary Bearer token through secure handoff for protected test endpoints.
application_contextNoBounded inspected app summary, not credentials or raw code. The existing requirements model proposes rules; it cannot issue a verdict. Requires sign-in and an existing session_id: start with providers first, then send this context in a separate call using the returned session. Separate from fixtures/execution.

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, the description discloses sign-in requirements, session/twin lifecycle, async non-blocking execution ('execute:true starts the existing backend job and returns running promptly'), credential installation via install_test_credentials, bounded observation windows, and strict cleanup obligations for temporary secrets. No behavioral claim conflicts with readOnlyHint=false, openWorldHint=true, idempotentHint=false, destructiveHint=false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Roughly 2,500 words of undifferentiated prose with no headings or grouping, front-loaded with a 'Completion rule' block rather than the tool's purpose. Individual sentences are often operationally relevant, but the sheer size and lack of structure make it hard to scan and largely unregisterable by an agent in one pass.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 16-parameter, deeply nested tool with no output schema, the description compensates thoroughly: it explains returned session_id/twin URLs/next_action, failure modes (configuration_rejected, setup_blocked, requirements_model_unavailable), suite options, and provider-specific lifecycle behavior. Nothing an agent needs to call it correctly appears missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 94%, so the baseline is 3, but the description adds real orchestration meaning: 'Call once with providers, then follow the returned next_action,' 'Do not include application_context in the initial session call,' and the arm→probe→run_id sequencing. It clarifies how session_id/run_id/app_base_url relate rather than restating the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The operative sentence — 'Start a validation session for the user's integration... hands back a FRESH twin per provider and asks the user to point their app at it, then reads external provider traffic' — is a specific verb+resource and explicitly distinguishes the tool from quickrun/run_workflow. However, the description opens with 'Completion rule: a request to verify an app includes its applicable failure checks...' rather than the purpose, so an agent must dig to find what the tool actually does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Named alternatives and exclusions are explicit: 'Do not substitute guide, quickrun, list_workflows, or reference probes for application verification,' 'Use it when someone wants their integration verified before launch,' 'Never execute while host-side setup is pending.' Conditions selecting the arm/probe path vs the hosted suite path are also spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verify_behaviorAInspect

Prove a known fix survives a bug — the 'prove' half of reproduce→prove. The backend spawns a buggy AND a fixed reference handler and fires the bug_pattern's probes at both, returning the side-by-side diff (e.g. the buggy handler double-charges on a duplicate webhook, the fixed handler dedupes). Call this AFTER run_workflow reproduces a failure, when the matched bug_pattern has a simulation block, to show the fix actually holds — not just that the failure reproduced. Pass sandbox_id + flow_run_id from the run so the diff is saved onto that run's receipt URL. bug_pattern_id comes from guide's matched_bug_pattern. The buggy/fixed handlers are FetchSandbox reference implementations, NOT the user's code. A confirmed reference result does not verify the user's app or show that it is ready to ship. Keep the returned provider and evidence scope with any reported result.

NameRequiredDescriptionDefault
promptNoOPTIONAL. The user's own description of the symptom. For patterns that can originate in the handler OR the provider, this classifies which side to simulate. Omit to run both.
sandbox_idNoOPTIONAL. The sandbox from the run. Pass with flow_run_id to save the diff onto that run's receipt. This binds the pattern to that sandbox's provider; provide it when verifying a provider integration.
flow_run_idNoOPTIONAL. The flow_run_id returned by run_workflow. Pass with sandbox_id so the receipt URL renders the diff alongside the steps.
bug_pattern_idYesThe bug_pattern to prove (e.g. webhook_duplicate_side_effect). Comes from guide's matched_bug_pattern.id or the spec's brain.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare it is not read-only and is open-world, so the bar is lower; the description still adds rich behavioral detail — the backend spawns both a buggy and a fixed reference handler, fires the probes at each, and returns a side-by-side diff. It also discloses that the handlers are FetchSandbox reference implementations, not the user's code, which prevents a serious misinterpretation of the result.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense and multi-clause, but front-loaded with the core action and every sentence carries distinct information (lifecycle position, side effects, parameter wiring, scope caveat). No filler, though it runs longer than strictly necessary.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A multi-step orchestration tool with no output schema; the description explains what is produced (side-by-side diff with a concrete example), where it is persisted (receipt URL), and what the result does and does not prove. Nothing needed to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description adds real meaning beyond the schema: sandbox_id and flow_run_id must be passed together to bind the pattern to a provider and save the diff onto that run's receipt, and bug_pattern_id's provenance (guide's matched_bug_pattern) is spelled out.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Prove a known fix survives a bug') and frames itself as the 'prove' half of a reproduce→prove pair, explicitly distinguishing it from run_workflow. An agent can tell exactly what this does without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit trigger ('Call this AFTER run_workflow reproduces a failure, when the matched bug_pattern has a simulation block') plus a clear limit ('a confirmed reference result does not verify the user's app'). Both when-to-use and when-not are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool updatev0.7.30
    • Changedvalidate_integration1 field changed
      • changedInput schema / properties / application_config / properties / checks / items / oneOf
        Previous value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "blocker": {
        -        "additionalProperties": false,
        -        "properties": {
        -          "capability": {
        -            "enum": [
        -              "provider_lifecycle",
        -              "provider_operation",
        -              "independent_storage_attestation"
        -            ],
        -            "type": "string"
        -          },
        -          "category": {
        -            "enum": [
        -              "setup_missing",
        -              "observation_missing",
        -              "policy_unresolved",
        -              "engine_unsupported"
        -            ],
        -            "type": "string"
        -          },
        -          "evidence": {
        -            "items": {
        -              "maxLength": 500,
        -              "minLength": 1,
        -              "type": "string"
        -            },
        -            "maxItems": 10,
        -            "minItems": 1,
        -            "type": "array"
        -          },
        -          "method": {
        -            "enum": [
        -              "GET",
        -              "POST",
        -              "PUT",
        -              "PATCH",
        -              "DELETE"
        -            ],
        -            "type": "string"
        -          },
        -          "path": {
        -            "type": "string"
        -          },
        -          "provider": {
        -            "type": "string"
        -          },
        -          "reason": {
        -            "maxLength": 1000,
        -            "minLength": 1,
        -            "type": "string"
        -          }
        -        },
        -        "required": [
        -          "category",
        -          "reason",
        -          "evidence"
        -        ],
        -        "type": "object"
        -      },
        -      "id": {
        -        "maxLength": 100,
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "name": {
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "id",
        -      "name",
        -      "blocker"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "id": {
        -        "maxLength": 100,
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "name": {
        -        "type": "string"
        -      },
        -      "unsupported_reason": {
        -        "minLength": 1,
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "id",
        -      "name",
        -      "unsupported_reason"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "actions": {
        -        "items": {
        -          "oneOf": [
        -            {
        -              "additionalProperties": false,
        -              "description": "Qualified external requester input, separate from app execution. Zendesk supports a public comment by that ticket's requester; arbitrary financial writes are refused.",
        -              "properties": {
        -                "expect_status": {
        -                  "enum": [
        -                    200
        -                  ],
        -                  "type": "integer"
        -                },
        -                "provider_input": {
        -                  "additionalProperties": false,
        -                  "properties": {
        -                    "body": {
        -                      "type": "object"
        -                    },
        -                    "method": {
        -                      "type": "string"
        -                    },
        -                    "path": {
        -                      "type": "string"
        -                    },
        -                    "provider": {
        -                      "type": "string"
        -                    }
        -                  },
        -                  "required": [
        -                    "provider",
        -                    "method",
        -                    "path",
        -                    "body"
        -                  ],
        -                  "type": "object"
        -                }
        -              },
        -              "required": [
        -                "provider_input"
        -              ],
        -              "type": "object"
        -            },
        -            {
        -              "additionalProperties": false,
        -              "properties": {
        -                "body": {},
        -                "concurrency": {
        -                  "maximum": 8,
        -                  "minimum": 1,
        -                  "type": "integer"
        -                },
        -                "expect_status": {
        -                  "maximum": 599,
        -                  "minimum": 200,
        -                  "type": "integer"
        -                },
        -                "method": {
        -                  "enum": [
        -                    "GET",
        -                    "POST",
        -                    "PUT",
        -                    "PATCH",
        -                    "DELETE"
        -                  ],
        -                  "type": "string"
        -                },
        -                "path": {
        -                  "type": "string"
        -                }
        -              },
        -              "required": [
        -                "method",
        -                "path"
        -              ],
        -              "type": "object"
        -            },
        -            {
        -              "additionalProperties": false,
        -              "properties": {
        -                "concurrency": {
        -                  "maximum": 8,
        -                  "minimum": 1,
        -                  "type": "integer"
        -                },
        -                "event": {
        -                  "oneOf": [
        -                    {
        -                      "additionalProperties": false,
        -                      "properties": {
        -                        "payload": {
        -                          "type": "object"
        -                        },
        -                        "provider": {
        -                          "type": "string"
        -                        },
        -                        "type": {
        -                          "type": "string"
        -                        }
        -                      },
        -                      "required": [
        -                        "provider",
        -                        "type",
        -                        "payload"
        -                      ],
        -                      "type": "object"
        -                    },
        -                    {
        -                      "additionalProperties": false,
        -                      "properties": {
        -                        "provider": {
        -                          "type": "string"
        -                        },
        -                        "replay_of": {
        -                          "type": "string"
        -                        }
        -                      },
        -                      "required": [
        -                        "provider",
        -                        "replay_of"
        -                      ],
        -                      "type": "object"
        -                    }
        -                  ]
        -                },
        -                "expect_status": {
        -                  "maximum": 599,
        -                  "minimum": 200,
        -                  "type": "integer"
        -                }
        -              },
        -              "required": [
        -                "event"
        -              ],
        -              "type": "object"
        -            },
        -            {
        -              "additionalProperties": false,
        -              "properties": {
        -                "delay_seconds": {
        -                  "exclusiveMinimum": 0,
        -                  "maximum": 5,
        -                  "type": "number"
        -                }
        -              },
        -              "required": [
        -                "delay_seconds"
        -              ],
        -              "type": "object"
        -            },
        -            {
        -              "additionalProperties": false,
        -              "properties": {
        -                "provider_control": {
        -                  "additionalProperties": false,
        -                  "properties": {
        -                    "provider": {
        -                      "type": "string"
        -                    },
        -                    "scenario": {
        -                      "enum": [
        -                        "default"
        -                      ],
        -                      "type": "string"
        -                    }
        -                  },
        -                  "required": [
        -                    "provider",
        -                    "scenario"
        -                  ],
        -                  "type": "object"
        -                }
        -              },
        -              "required": [
        -                "provider_control"
        -              ],
        -              "type": "object"
        -            },
        -            {
        -              "additionalProperties": false,
        -              "properties": {
        -                "client_lifecycle": {
        -                  "additionalProperties": false,
        -                  "properties": {
        -                    "identity_field": {
        -                      "type": "string"
        -                    },
        -                    "observe_path": {
        -                      "type": "string"
        -                    },
        -                    "operation": {
        -                      "enum": [
        -                        "restart"
        -                      ],
        -                      "type": "string"
        -                    },
        -                    "timeout_seconds": {
        -                      "maximum": 180,
        -                      "minimum": 1,
        -                      "type": "number"
        -                    }
        -                  },
        -                  "required": [
        -                    "operation",
        -                    "observe_path",
        -                    "identity_field",
        -                    "timeout_seconds"
        -                  ],
        -                  "type": "object"
        -                }
        -              },
        -              "required": [
        -                "client_lifecycle"
        -              ],
        -              "type": "object"
        -            }
        -          ]
        -        },
        -        "maxItems": 20,
        -        "minItems": 1,
        -        "type": "array"
        -      },
        -      "assertion": {
        -        "$ref": "#/$defs/applicationAssertion"
        -      },
        -      "before": {
        -        "additionalProperties": false,
        -        "properties": {
        -          "assertion": {
        -            "$ref": "#/$defs/applicationAssertion"
        -          },
        -          "path": {
        -            "type": "string"
        -          }
        -        },
        -        "required": [
        -          "path",
        -          "assertion"
        -        ],
        -        "type": "object"
        -      },
        -      "fault": {
        -        "additionalProperties": false,
        -        "properties": {
        -          "event_type": {
        -            "type": "string"
        -          },
        -          "evidence": {
        -            "enum": [
        -              "refusal",
        -              "accepted_response_loss",
        -              "accepted_response_retry",
        -              "webhook_withheld"
        -            ],
        -            "type": "string"
        -          },
        -          "provider": {
        -            "type": "string"
        -          },
        -          "scenario": {
        -            "type": "string"
        -          }
        -        },
        -        "required": [
        -          "provider",
        -          "scenario"
        -        ],
        -        "type": "object"
        -      },
        -      "id": {
        -        "maxLength": 100,
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "name": {
        -        "type": "string"
        -      },
        -      "observation_deadline_seconds": {
        -        "maximum": 180,
        -        "minimum": 0,
        -        "type": "number"
        -      },
        -      "observe": {
        -        "additionalProperties": false,
        -        "properties": {
        -          "path": {
        -            "type": "string"
        -          }
        -        },
        -        "required": [
        -          "path"
        -        ],
        -        "type": "object"
        -      },
        -      "provider_assertions": {
        -        "description": "Additional independent state assertions against this session's provider twins. Select records[*] from a public collection; readbacks are recorded separately from app-reported state. This is simulated provider state, not live-provider delivery.",
        -        "items": {
        -          "additionalProperties": false,
        -          "properties": {
        -            "assertion": {
        -              "type": "object"
        -            },
        -            "collection": {
        -              "type": "string"
        -            },
        -            "provider": {
        -              "type": "string"
        -            }
        -          },
        -          "required": [
        -            "provider",
        -            "collection",
        -            "assertion"
        -          ],
        -          "type": "object"
        -        },
        -        "maxItems": 8,
        -        "type": "array"
        -      }
        -    },
        -    "required": [
        -      "id",
        -      "name",
        -      "actions",
        -      "before",
        -      "observe",
        -      "assertion"
        -    ],
        -    "type": "object"
        -  }
        -]New value: +[
        +  {
        +    "additionalProperties": false,
        +    "properties": {
        +      "blocker": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "capability": {
        +            "enum": [
        +              "provider_lifecycle",
        +              "provider_operation",
        +              "independent_storage_attestation"
        +            ],
        +            "type": "string"
        +          },
        +          "category": {
        +            "enum": [
        +              "setup_missing",
        +              "observation_missing",
        +              "policy_unresolved",
        +              "engine_unsupported"
        +            ],
        +            "type": "string"
        +          },
        +          "evidence": {
        +            "items": {
        +              "maxLength": 500,
        +              "minLength": 1,
        +              "type": "string"
        +            },
        +            "maxItems": 10,
        +            "minItems": 1,
        +            "type": "array"
        +          },
        +          "method": {
        +            "enum": [
        +              "GET",
        +              "POST",
        +              "PUT",
        +              "PATCH",
        +              "DELETE"
        +            ],
        +            "type": "string"
        +          },
        +          "path": {
        +            "type": "string"
        +          },
        +          "provider": {
        +            "type": "string"
        +          },
        +          "reason": {
        +            "maxLength": 1000,
        +            "minLength": 1,
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "category",
        +          "reason",
        +          "evidence"
        +        ],
        +        "type": "object"
        +      },
        +      "id": {
        +        "maxLength": 100,
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "name": {
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "id",
        +      "name",
        +      "blocker"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "additionalProperties": false,
        +    "properties": {
        +      "id": {
        +        "maxLength": 100,
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "name": {
        +        "type": "string"
        +      },
        +      "unsupported_reason": {
        +        "minLength": 1,
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "id",
        +      "name",
        +      "unsupported_reason"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "additionalProperties": false,
        +    "properties": {
        +      "actions": {
        +        "items": {
        +          "oneOf": [
        +            {
        +              "additionalProperties": false,
        +              "description": "Qualified external input, recorded separately from app execution. Requester is the default (Zendesk public reply). Explicit external_operator permits only configured twin operations, such as an independent refund or new subscription between app actions. It cannot replace app execution or count as an app effect.",
        +              "properties": {
        +                "expect_status": {
        +                  "enum": [
        +                    200
        +                  ],
        +                  "type": "integer"
        +                },
        +                "provider_input": {
        +                  "additionalProperties": false,
        +                  "properties": {
        +                    "actor": {
        +                      "enum": [
        +                        "requester",
        +                        "external_operator"
        +                      ],
        +                      "type": "string"
        +                    },
        +                    "body": {
        +                      "type": "object"
        +                    },
        +                    "method": {
        +                      "type": "string"
        +                    },
        +                    "path": {
        +                      "type": "string"
        +                    },
        +                    "provider": {
        +                      "type": "string"
        +                    }
        +                  },
        +                  "required": [
        +                    "provider",
        +                    "method",
        +                    "path",
        +                    "body"
        +                  ],
        +                  "type": "object"
        +                }
        +              },
        +              "required": [
        +                "provider_input"
        +              ],
        +              "type": "object"
        +            },
        +            {
        +              "additionalProperties": false,
        +              "properties": {
        +                "body": {},
        +                "concurrency": {
        +                  "maximum": 8,
        +                  "minimum": 1,
        +                  "type": "integer"
        +                },
        +                "expect_status": {
        +                  "maximum": 599,
        +                  "minimum": 200,
        +                  "type": "integer"
        +                },
        +                "method": {
        +                  "enum": [
        +                    "GET",
        +                    "POST",
        +                    "PUT",
        +                    "PATCH",
        +                    "DELETE"
        +                  ],
        +                  "type": "string"
        +                },
        +                "path": {
        +                  "type": "string"
        +                }
        +              },
        +              "required": [
        +                "method",
        +                "path"
        +              ],
        +              "type": "object"
        +            },
        +            {
        +              "additionalProperties": false,
        +              "properties": {
        +                "concurrency": {
        +                  "maximum": 8,
        +                  "minimum": 1,
        +                  "type": "integer"
        +                },
        +                "event": {
        +                  "oneOf": [
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "payload": {
        +                          "type": "object"
        +                        },
        +                        "provider": {
        +                          "type": "string"
        +                        },
        +                        "type": {
        +                          "type": "string"
        +                        }
        +                      },
        +                      "required": [
        +                        "provider",
        +                        "type",
        +                        "payload"
        +                      ],
        +                      "type": "object"
        +                    },
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "provider": {
        +                          "type": "string"
        +                        },
        +                        "replay_of": {
        +                          "type": "string"
        +                        }
        +                      },
        +                      "required": [
        +                        "provider",
        +                        "replay_of"
        +                      ],
        +                      "type": "object"
        +                    }
        +                  ]
        +                },
        +                "expect_status": {
        +                  "maximum": 599,
        +                  "minimum": 200,
        +                  "type": "integer"
        +                }
        +              },
        +              "required": [
        +                "event"
        +              ],
        +              "type": "object"
        +            },
        +            {
        +              "additionalProperties": false,
        +              "properties": {
        +                "delay_seconds": {
        +                  "exclusiveMinimum": 0,
        +                  "maximum": 5,
        +                  "type": "number"
        +                }
        +              },
        +              "required": [
        +                "delay_seconds"
        +              ],
        +              "type": "object"
        +            },
        +            {
        +              "additionalProperties": false,
        +              "properties": {
        +                "provider_control": {
        +                  "additionalProperties": false,
        +                  "properties": {
        +                    "provider": {
        +                      "type": "string"
        +                    },
        +                    "scenario": {
        +                      "enum": [
        +                        "default"
        +                      ],
        +                      "type": "string"
        +                    }
        +                  },
        +                  "required": [
        +                    "provider",
        +                    "scenario"
        +                  ],
        +                  "type": "object"
        +                }
        +              },
        +              "required": [
        +                "provider_control"
        +              ],
        +              "type": "object"
        +            },
        +            {
        +              "additionalProperties": false,
        +              "properties": {
        +                "client_lifecycle": {
        +                  "additionalProperties": false,
        +                  "properties": {
        +                    "identity_field": {
        +                      "type": "string"
        +                    },
        +                    "observe_path": {
        +                      "type": "string"
        +                    },
        +                    "operation": {
        +                      "enum": [
        +                        "restart"
        +                      ],
        +                      "type": "string"
        +                    },
        +                    "timeout_seconds": {
        +                      "maximum": 180,
        +                      "minimum": 1,
        +                      "type": "number"
        +                    }
        +                  },
        +                  "required": [
        +                    "operation",
        +                    "observe_path",
        +                    "identity_field",
        +                    "timeout_seconds"
        +                  ],
        +                  "type": "object"
        +                }
        +              },
        +              "required": [
        +                "client_lifecycle"
        +              ],
        +              "type": "object"
        +            }
        +          ]
        +        },
        +        "maxItems": 20,
        +        "minItems": 1,
        +        "type": "array"
        +      },
        +      "assertion": {
        +        "$ref": "#/$defs/applicationAssertion"
        +      },
        +      "before": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "assertion": {
        +            "$ref": "#/$defs/applicationAssertion"
        +          },
        +          "path": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "path",
        +          "assertion"
        +        ],
        +        "type": "object"
        +      },
        +      "fault": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "event_type": {
        +            "type": "string"
        +          },
        +          "evidence": {
        +            "enum": [
        +              "refusal",
        +              "accepted_response_loss",
        +              "accepted_response_retry",
        +              "webhook_withheld"
        +            ],
        +            "type": "string"
        +          },
        +          "provider": {
        +            "type": "string"
        +          },
        +          "scenario": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "provider",
        +          "scenario"
        +        ],
        +        "type": "object"
        +      },
        +      "id": {
        +        "maxLength": 100,
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "name": {
        +        "type": "string"
        +      },
        +      "observation_deadline_seconds": {
        +        "maximum": 180,
        +        "minimum": 0,
        +        "type": "number"
        +      },
        +      "observe": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "path": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "path"
        +        ],
        +        "type": "object"
        +      },
        +      "provider_assertions": {
        +        "description": "Additional independent state assertions against this session's provider twins. Select records[*] from a public collection; readbacks are recorded separately from app-reported state. This is simulated provider state, not live-provider delivery.",
        +        "items": {
        +          "additionalProperties": false,
        +          "properties": {
        +            "assertion": {
        +              "type": "object"
        +            },
        +            "collection": {
        +              "type": "string"
        +            },
        +            "provider": {
        +              "type": "string"
        +            }
        +          },
        +          "required": [
        +            "provider",
        +            "collection",
        +            "assertion"
        +          ],
        +          "type": "object"
        +        },
        +        "maxItems": 8,
        +        "type": "array"
        +      }
        +    },
        +    "required": [
        +      "id",
        +      "name",
        +      "actions",
        +      "before",
        +      "observe",
        +      "assertion"
        +    ],
        +    "type": "object"
        +  }
        +]
  2. 6 tool updatesv0.7.29
    • Addedget_job
    • Addedinstall_test_credentials
    • Changedlist_workflows1 field changed
      • addedInput schema / properties / workflow_name
        Added value: +{
        +  "description": "Exact workflow id or name returned by this catalog. Returns its full definition, including steps and outcome rules, without executing it; omit for summaries.",
        +  "type": "string"
        +}
    • Addedmanage_tunnel
    • Changedprove_fix1 field changed
      • changedInput schema / properties / bug / description
        Previous value: -"The finding being fixed — used to select FetchSandbox's scenario."New value: +"The finding being fixed AND the user's acceptance criteria: required app operations, expected state, ownership boundaries, wire types/units and readback checks. Preserve the requested scope; do not reduce a workflow request to a single symptom. Used to select/generate the probe."
    • Changedvalidate_integration12 fields changed
      • addedInput schema / $defs
        Added value: +{
        +  "applicationAssertion": {
        +    "additionalProperties": false,
        +    "properties": {
        +      "absent": {
        +        "type": "boolean"
        +      },
        +      "all_of": {
        +        "items": {
        +          "$ref": "#/$defs/applicationAssertion"
        +        },
        +        "type": "array"
        +      },
        +      "contains": {
        +        "type": "string"
        +      },
        +      "count": {
        +        "minimum": 0,
        +        "type": "integer"
        +      },
        +      "equals": {
        +        "type": [
        +          "string",
        +          "number",
        +          "boolean",
        +          "null"
        +        ]
        +      },
        +      "repeat": {
        +        "maximum": 5,
        +        "minimum": 1,
        +        "type": "integer"
        +      },
        +      "required": {
        +        "type": "boolean"
        +      },
        +      "select": {
        +        "type": "string"
        +      },
        +      "set_equals": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array",
        +        "uniqueItems": true
        +      },
        +      "stable_under_repeat": {
        +        "type": "boolean"
        +      },
        +      "where": {
        +        "items": {
        +          "$ref": "#/$defs/applicationAssertion"
        +        },
        +        "type": "array"
        +      }
        +    },
        +    "required": [],
        +    "type": "object"
        +  }
        +}
      • changedInput schema / properties / app_base_url / description
        Previous value: -"Where their app is deployed, if known. Recorded so a later step can deliver webhooks to it."New value: +"App origin when FetchSandbox must drive or observe a hosted HTTP suite. Optional for a provider-backed workflow executed by the builder through arm/probe; that path does not require a published app or webhook."
      • addedInput schema / properties / application_config
        Added value: +{
        +  "additionalProperties": false,
        +  "description": "Frozen HTTP app drive/read-back contract: 1-30 checks, 1-20 actions per executable check, at most 300 declared actions in total, below 64 KiB. Preserve intermediate reads within these bounds; do not silently remove them. First check id=baseline must establish success. Checks require id, name, before {path,assertion}, ordinary HTTP actions [{method,path,body?,expect_status?}] or signed-event actions [{event:{provider,type,payload},expect_status?}], observe {path}, and assertion using select/equals/count/required or all_of. where and all_of must be lists of assertion objects, not field maps. equals accepts JSON strings, numbers, booleans and null; numeric/boolean literals preserve type, and null must be a present null field. Stripe event.payload is only the resource inside data.object, never a full event or a data.object wrapper: the server constructs and signs the envelope. Optional per-check observation_deadline_seconds (0-180, repeat=1 only) records bounded polling for eventual saved state; it does not drive a browser or turn a timeout into a pass. Every check first reads its declared starting-state precondition. Baseline requires count=0 on the same observed collection, with no where filter. Use {{run_id}} to scope fresh synthetic records. All paths are app-relative without query strings or fragments, including non-secret run_id queries; preserve run scoping through an inspected protected path-segment observer. Baseline permits only ordinary HTTP actions without signed events; prove signed purchase/access in later checks. Do not combine event actions with method/path/body/headers. {{step1.id}} binds a real action response to read-back. assertion.repeat (1-5) and stable_under_repeat support replay. Optional fault {provider,scenario} requires independently recorded provider refusal. For reassess/recontract include scope_review for every candidate check. Outside-scope feature proposals contain id,name,unsupported_reason only and move to separate scope_exclusions; they are not required tests. In-scope missing work uses id,name,blocker {category,reason,evidence,...}, never a free-text unsupported reason. engine_unsupported requires a capability confirmed by current backend capabilities; a missing app route or observer is not an engine gap. Unresolved policy remains unmeasured. Legacy untyped reasons remain unclassified, not engine unsupported. No user-supplied credentials, auth headers or live data. require_verifier_token=true generates an owner-only temporary Bearer token through secure handoff for protected test endpoints.",
        +  "properties": {
        +    "app_version": {
        +      "type": "string"
        +    },
        +    "checks": {
        +      "items": {
        +        "oneOf": [
        +          {
        +            "additionalProperties": false,
        +            "properties": {
        +              "blocker": {
        +                "additionalProperties": false,
        +                "properties": {
        +                  "capability": {
        +                    "enum": [
        +                      "provider_lifecycle",
        +                      "provider_operation",
        +                      "independent_storage_attestation"
        +                    ],
        +                    "type": "string"
        +                  },
        +                  "category": {
        +                    "enum": [
        +                      "setup_missing",
        +                      "observation_missing",
        +                      "policy_unresolved",
        +                      "engine_unsupported"
        +                    ],
        +                    "type": "string"
        +                  },
        +                  "evidence": {
        +                    "items": {
        +                      "maxLength": 500,
        +                      "minLength": 1,
        +                      "type": "string"
        +                    },
        +                    "maxItems": 10,
        +                    "minItems": 1,
        +                    "type": "array"
        +                  },
        +                  "method": {
        +                    "enum": [
        +                      "GET",
        +                      "POST",
        +                      "PUT",
        +                      "PATCH",
        +                      "DELETE"
        +                    ],
        +                    "type": "string"
        +                  },
        +                  "path": {
        +                    "type": "string"
        +                  },
        +                  "provider": {
        +                    "type": "string"
        +                  },
        +                  "reason": {
        +                    "maxLength": 1000,
        +                    "minLength": 1,
        +                    "type": "string"
        +                  }
        +                },
        +                "required": [
        +                  "category",
        +                  "reason",
        +                  "evidence"
        +                ],
        +                "type": "object"
        +              },
        +              "id": {
        +                "maxLength": 100,
        +                "minLength": 1,
        +                "type": "string"
        +              },
        +              "name": {
        +                "type": "string"
        +              }
        +            },
        +            "required": [
        +              "id",
        +              "name",
        +              "blocker"
        +            ],
        +            "type": "object"
        +          },
        +          {
        +            "additionalProperties": false,
        +            "properties": {
        +              "id": {
        +                "maxLength": 100,
        +                "minLength": 1,
        +                "type": "string"
        +              },
        +              "name": {
        +                "type": "string"
        +              },
        +              "unsupported_reason": {
        +                "minLength": 1,
        +                "type": "string"
        +              }
        +            },
        +            "required": [
        +              "id",
        +              "name",
        +              "unsupported_reason"
        +            ],
        +            "type": "object"
        +          },
        +          {
        +            "additionalProperties": false,
        +            "properties": {
        +              "actions": {
        +                "items": {
        +                  "oneOf": [
        +                    {
        +                      "additionalProperties": false,
        +                      "description": "Qualified external requester input, separate from app execution. Zendesk supports a public comment by that ticket's requester; arbitrary financial writes are refused.",
        +                      "properties": {
        +                        "expect_status": {
        +                          "enum": [
        +                            200
        +                          ],
        +                          "type": "integer"
        +                        },
        +                        "provider_input": {
        +                          "additionalProperties": false,
        +                          "properties": {
        +                            "body": {
        +                              "type": "object"
        +                            },
        +                            "method": {
        +                              "type": "string"
        +                            },
        +                            "path": {
        +                              "type": "string"
        +                            },
        +                            "provider": {
        +                              "type": "string"
        +                            }
        +                          },
        +                          "required": [
        +                            "provider",
        +                            "method",
        +                            "path",
        +                            "body"
        +                          ],
        +                          "type": "object"
        +                        }
        +                      },
        +                      "required": [
        +                        "provider_input"
        +                      ],
        +                      "type": "object"
        +                    },
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "body": {},
        +                        "concurrency": {
        +                          "maximum": 8,
        +                          "minimum": 1,
        +                          "type": "integer"
        +                        },
        +                        "expect_status": {
        +                          "maximum": 599,
        +                          "minimum": 200,
        +                          "type": "integer"
        +                        },
        +                        "method": {
        +                          "enum": [
        +                            "GET",
        +                            "POST",
        +                            "PUT",
        +                            "PATCH",
        +                            "DELETE"
        +                          ],
        +                          "type": "string"
        +                        },
        +                        "path": {
        +                          "type": "string"
        +                        }
        +                      },
        +                      "required": [
        +                        "method",
        +                        "path"
        +                      ],
        +                      "type": "object"
        +                    },
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "concurrency": {
        +                          "maximum": 8,
        +                          "minimum": 1,
        +                          "type": "integer"
        +                        },
        +                        "event": {
        +                          "oneOf": [
        +                            {
        +                              "additionalProperties": false,
        +                              "properties": {
        +                                "payload": {
        +                                  "type": "object"
        +                                },
        +                                "provider": {
        +                                  "type": "string"
        +                                },
        +                                "type": {
        +                                  "type": "string"
        +                                }
        +                              },
        +                              "required": [
        +                                "provider",
        +                                "type",
        +                                "payload"
        +                              ],
        +                              "type": "object"
        +                            },
        +                            {
        +                              "additionalProperties": false,
        +                              "properties": {
        +                                "provider": {
        +                                  "type": "string"
        +                                },
        +                                "replay_of": {
        +                                  "type": "string"
        +                                }
        +                              },
        +                              "required": [
        +                                "provider",
        +                                "replay_of"
        +                              ],
        +                              "type": "object"
        +                            }
        +                          ]
        +                        },
        +                        "expect_status": {
        +                          "maximum": 599,
        +                          "minimum": 200,
        +                          "type": "integer"
        +                        }
        +                      },
        +                      "required": [
        +                        "event"
        +                      ],
        +                      "type": "object"
        +                    },
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "delay_seconds": {
        +                          "exclusiveMinimum": 0,
        +                          "maximum": 5,
        +                          "type": "number"
        +                        }
        +                      },
        +                      "required": [
        +                        "delay_seconds"
        +                      ],
        +                      "type": "object"
        +                    },
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "provider_control": {
        +                          "additionalProperties": false,
        +                          "properties": {
        +                            "provider": {
        +                              "type": "string"
        +                            },
        +                            "scenario": {
        +                              "enum": [
        +                                "default"
        +                              ],
        +                              "type": "string"
        +                            }
        +                          },
        +                          "required": [
        +                            "provider",
        +                            "scenario"
        +                          ],
        +                          "type": "object"
        +                        }
        +                      },
        +                      "required": [
        +                        "provider_control"
        +                      ],
        +                      "type": "object"
        +                    },
        +                    {
        +                      "additionalProperties": false,
        +                      "properties": {
        +                        "client_lifecycle": {
        +                          "additionalProperties": false,
        +                          "properties": {
        +                            "identity_field": {
        +                              "type": "string"
        +                            },
        +                            "observe_path": {
        +                              "type": "string"
        +                            },
        +                            "operation": {
        +                              "enum": [
        +                                "restart"
        +                              ],
        +                              "type": "string"
        +                            },
        +                            "timeout_seconds": {
        +                              "maximum": 180,
        +                              "minimum": 1,
        +                              "type": "number"
        +                            }
        +                          },
        +                          "required": [
        +                            "operation",
        +                            "observe_path",
        +                            "identity_field",
        +                            "timeout_seconds"
        +                          ],
        +                          "type": "object"
        +                        }
        +                      },
        +                      "required": [
        +                        "client_lifecycle"
        +                      ],
        +                      "type": "object"
        +                    }
        +                  ]
        +                },
        +                "maxItems": 20,
        +                "minItems": 1,
        +                "type": "array"
        +              },
        +              "assertion": {
        +                "$ref": "#/$defs/applicationAssertion"
        +              },
        +              "before": {
        +                "additionalProperties": false,
        +                "properties": {
        +                  "assertion": {
        +                    "$ref": "#/$defs/applicationAssertion"
        +                  },
        +                  "path": {
        +                    "type": "string"
        +                  }
        +                },
        +                "required": [
        +                  "path",
        +                  "assertion"
        +                ],
        +                "type": "object"
        +              },
        +              "fault": {
        +                "additionalProperties": false,
        +                "properties": {
        +                  "event_type": {
        +                    "type": "string"
        +                  },
        +                  "evidence": {
        +                    "enum": [
        +                      "refusal",
        +                      "accepted_response_loss",
        +                      "accepted_response_retry",
        +                      "webhook_withheld"
        +                    ],
        +                    "type": "string"
        +                  },
        +                  "provider": {
        +                    "type": "string"
        +                  },
        +                  "scenario": {
        +                    "type": "string"
        +                  }
        +                },
        +                "required": [
        +                  "provider",
        +                  "scenario"
        +                ],
        +                "type": "object"
        +              },
        +              "id": {
        +                "maxLength": 100,
        +                "minLength": 1,
        +                "type": "string"
        +              },
        +              "name": {
        +                "type": "string"
        +              },
        +              "observation_deadline_seconds": {
        +                "maximum": 180,
        +                "minimum": 0,
        +                "type": "number"
        +              },
        +              "observe": {
        +                "additionalProperties": false,
        +                "properties": {
        +                  "path": {
        +                    "type": "string"
        +                  }
        +                },
        +                "required": [
        +                  "path"
        +                ],
        +                "type": "object"
        +              },
        +              "provider_assertions": {
        +                "description": "Additional independent state assertions against this session's provider twins. Select records[*] from a public collection; readbacks are recorded separately from app-reported state. This is simulated provider state, not live-provider delivery.",
        +                "items": {
        +                  "additionalProperties": false,
        +                  "properties": {
        +                    "assertion": {
        +                      "type": "object"
        +                    },
        +                    "collection": {
        +                      "type": "string"
        +                    },
        +                    "provider": {
        +                      "type": "string"
        +                    }
        +                  },
        +                  "required": [
        +                    "provider",
        +                    "collection",
        +                    "assertion"
        +                  ],
        +                  "type": "object"
        +                },
        +                "maxItems": 8,
        +                "type": "array"
        +              }
        +            },
        +            "required": [
        +              "id",
        +              "name",
        +              "actions",
        +              "before",
        +              "observe",
        +              "assertion"
        +            ],
        +            "type": "object"
        +          }
        +        ]
        +      },
        +      "maxItems": 30,
        +      "minItems": 1,
        +      "type": "array"
        +    },
        +    "coverage_review": {
        +      "description": "Applicability review against the returned coverage_policy. Missing dimensions remain unreviewed. Link relevant dimensions to declared checks, including explicitly unsupported checks. This is a proposed review, not execution proof.",
        +      "items": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "dimension": {
        +            "type": "string"
        +          },
        +          "reason": {
        +            "type": "string"
        +          },
        +          "relevance": {
        +            "enum": [
        +              "relevant",
        +              "not_applicable",
        +              "unresolved"
        +            ],
        +            "type": "string"
        +          },
        +          "rule_ids": {
        +            "items": {
        +              "type": "string"
        +            },
        +            "type": "array"
        +          }
        +        },
        +        "required": [
        +          "dimension",
        +          "relevance",
        +          "reason",
        +          "rule_ids"
        +        ],
        +        "type": "object"
        +      },
        +      "maxItems": 17,
        +      "type": "array"
        +    },
        +    "disposable_test_app": {
        +      "enum": [
        +        true
        +      ],
        +      "type": "boolean"
        +    },
        +    "event_delivery": {
        +      "description": "provider_lifecycle registers provider-native subscriptions and requires actual app workflow actions. Optional provider_lifecycle_workflows selects catalog-owned witnesses/readback. Omitted selection preserves payment/email witnesses. direct uses verifier-signed events.",
        +      "enum": [
        +        "direct",
        +        "provider_lifecycle"
        +      ],
        +      "type": "string"
        +    },
        +    "event_destinations": {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "description": "Session-provider to same-origin webhook path without query/fragment. Requires require_verifier_token=true and the isolated FETCHSANDBOX_<PROVIDER>_WEBHOOK_SECRET from secure handoff. Complete signed action shape: {event:{provider,type,payload},expect_status:200}; exact replay: {event:{provider,replay_of:'{{step2.event_id}}'},expect_status:200}. No method/path/body/headers on signed actions. Baseline uses ordinary HTTP actions; signed events belong in later checks. Existing signature checks must remain enabled.",
        +      "type": "object"
        +    },
        +    "observation_seconds": {
        +      "maximum": 5,
        +      "minimum": 0.1,
        +      "type": "number"
        +    },
        +    "provider_lifecycle_workflows": {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "description": "Optional destination-provider to returned catalog workflow id, e.g. {stripe: create_and_send_invoice}. Only declared application_lifecycle profiles are accepted. The server freezes expected events and independently reads the event resource in the workflow terminal state. Do not invent event names or substitute provider traffic for app-state assertions.",
        +      "type": "object"
        +    },
        +    "require_verifier_token": {
        +      "type": "boolean"
        +    },
        +    "scope_review": {
        +      "description": "Required when reassessing/recontracting app scope. Review EVERY candidate check from inspected app behavior. Outside-app-scope entries must be non-executable proposals with inspection references; the backend excludes them from required checks and records them separately. Keep missing protection on relevant behavior in scope. A reason is an app-agent declaration, not independent source attestation. coverage_review links must name retained required checks only. Preserve failed requirements for the same app version.",
        +      "items": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "check_id": {
        +            "type": "string"
        +          },
        +          "classification": {
        +            "enum": [
        +              "in_scope",
        +              "outside_app_scope",
        +              "unresolved"
        +            ],
        +            "type": "string"
        +          },
        +          "evidence": {
        +            "items": {
        +              "maxLength": 500,
        +              "minLength": 1,
        +              "type": "string"
        +            },
        +            "maxItems": 10,
        +            "minItems": 1,
        +            "type": "array"
        +          },
        +          "reason": {
        +            "maxLength": 1000,
        +            "minLength": 1,
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "check_id",
        +          "classification",
        +          "reason",
        +          "evidence"
        +        ],
        +        "type": "object"
        +      },
        +      "maxItems": 30,
        +      "minItems": 1,
        +      "type": "array"
        +    }
        +  },
        +  "required": [
        +    "app_version",
        +    "disposable_test_app",
        +    "checks"
        +  ],
        +  "type": "object"
        +}
      • addedInput schema / properties / application_context
        Added value: +{
        +  "additionalProperties": false,
        +  "description": "Bounded inspected app summary, not credentials or raw code. The existing requirements model proposes rules; it cannot issue a verdict. Requires sign-in and an existing session_id: start with providers first, then send this context in a separate call using the returned session. Separate from fixtures/execution.",
        +  "properties": {
        +    "app_version": {
        +      "type": "string"
        +    },
        +    "entry_points": {
        +      "items": {
        +        "type": "string"
        +      },
        +      "type": "array"
        +    },
        +    "fixture_requirements": {
        +      "items": {
        +        "type": "string"
        +      },
        +      "type": "array"
        +    },
        +    "goal": {
        +      "type": "string"
        +    },
        +    "state_store": {
        +      "type": "string"
        +    },
        +    "workflow": {
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "goal"
        +  ],
        +  "type": "object"
        +}
      • changedInput schema / properties / execute / description
        Previous value: -"Drive the configured app and evaluate the frozen receipt suite. Requires session_id/run_id; configuration acknowledgment alone is not proof."New value: +"Drive the configured app and evaluate the frozen receipt suite. Requires session_id/run_id; configuration acknowledgment alone is not proof. Starts the existing backend job asynchronously and returns promptly; follow status in this agent without allocating a background helper."
      • addedInput schema / properties / fixtures
        Added value: +{
        +  "additionalProperties": false,
        +  "description": "Prepare session-owned twin data before app tests. Requires session_id. Stable id makes repeated calls replay-safe. Every step names spec, method and relative path. Use {{step1.id}} for dependencies. End with GET and an exact identity assertion. No live provider calls. An explicitly advertised /__fetchsandbox/fixtures/time step is an owner-only synthetic clock control, not a provider API; it is unavailable during active attempts and requires normal provider GET readback.",
        +  "properties": {
        +    "id": {
        +      "maxLength": 100,
        +      "minLength": 1,
        +      "type": "string"
        +    },
        +    "steps": {
        +      "items": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "assertion": {
        +            "additionalProperties": false,
        +            "description": "GET read-back only. Exactly four fields, e.g. {select: data.id, equals: '{{step2.id}}', required: true, count: 1} for Paddle. Use separate GET assertions for amount and currency; application all_of/where and JSON scalar predicates are not supported here.",
        +            "properties": {
        +              "count": {
        +                "enum": [
        +                  1
        +                ],
        +                "type": "integer"
        +              },
        +              "equals": {
        +                "type": "string"
        +              },
        +              "required": {
        +                "enum": [
        +                  true
        +                ],
        +                "type": "boolean"
        +              },
        +              "select": {
        +                "type": "string"
        +              }
        +            },
        +            "required": [
        +              "select",
        +              "equals",
        +              "required",
        +              "count"
        +            ],
        +            "type": "object"
        +          },
        +          "body": {
        +            "type": "object"
        +          },
        +          "expect_status": {
        +            "maximum": 299,
        +            "minimum": 200,
        +            "type": "integer"
        +          },
        +          "method": {
        +            "enum": [
        +              "GET",
        +              "POST",
        +              "PUT",
        +              "PATCH",
        +              "DELETE"
        +            ],
        +            "type": "string"
        +          },
        +          "name": {
        +            "type": "string"
        +          },
        +          "path": {
        +            "type": "string"
        +          },
        +          "query_params": {
        +            "type": "object"
        +          },
        +          "spec": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "spec",
        +          "method",
        +          "path"
        +        ],
        +        "type": "object"
        +      },
        +      "maxItems": 40,
        +      "minItems": 1,
        +      "type": "array"
        +    }
        +  },
        +  "required": [
        +    "id",
        +    "steps"
        +  ],
        +  "type": "object"
        +}
      • addedInput schema / properties / inspect
        Added value: +{
        +  "additionalProperties": false,
        +  "description": "Read fixture inputs through MCP using session_id. Prefer this MCP read to extracting a key for direct HTTP: the edge can reject a generic Python user agent even with valid ownership. This path keeps the existing MCP client authentication and headers. Do not ask for live provider keys. First resource=state returns collection counts; select collection to read real IDs and relationships with pagination. resource=operations lists provider contracts; specify both exact method and documented path to resolve one schema. Read-only, session-owned twins only, not application proof.",
        +  "properties": {
        +    "collection": {
        +      "type": "string"
        +    },
        +    "limit": {
        +      "maximum": 100,
        +      "minimum": 1,
        +      "type": "integer"
        +    },
        +    "method": {
        +      "type": "string"
        +    },
        +    "offset": {
        +      "minimum": 0,
        +      "type": "integer"
        +    },
        +    "path": {
        +      "type": "string"
        +    },
        +    "provider": {
        +      "type": "string"
        +    },
        +    "resource": {
        +      "enum": [
        +        "state",
        +        "operations"
        +      ],
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "provider",
        +    "resource"
        +  ],
        +  "type": "object"
        +}
      • addedInput schema / properties / preflight
        Added value: +{
        +  "description": "Read-only public readiness check for a configured application_workflow_v1. Requires session_id/run_id. Checks frozen empty baseline and token enforcement without creating orders, injecting faults or rotating secrets. Retry same run after its exact setup blocker is repaired.",
        +  "type": "boolean"
        +}
      • changedInput schema / properties / providers / description
        Previous value: -"Provider slugs the app integrates, e.g. ['stripe'] or ['paddle','resend']. Required to start a session."New value: +"Provider slugs the app integrates, e.g. ['stripe'] or ['paddle','resend']. For provider-free HTTP workflows, omit providers and pass app_base_url."
      • addedInput schema / properties / reassess
        Added value: +{
        +  "description": "Set true when the user asks to reassess/recontract an app. Requires suite=application_workflow_v1 and application_config.scope_review for EVERY candidate check. Freeze a fresh contract scoped to inspected promised behavior. Outside-scope proposals are recorded separately; missing protection on relevant behavior remains a requirement. Never edit historical results or remove a failed requirement for the same app version.",
        +  "type": "boolean"
        +}
      • changedInput schema / properties / run_id / description
        Previous value: -"Attempt ID returned by arm. Required with probe or cancel; prevents stale calls from consuming a newer attempt."New value: +"Attempt ID returned by arm or suite configuration. Required with probe, cancel, execute or preflight; prevents stale calls from consuming a newer attempt."
      • changedInput schema / properties / suite / enum
        Previous value: -[
        -  "receipt_delivery_v1",
        -  "receipt_recovery_v1"
        -]New value: +[
        +  "receipt_delivery_v1",
        +  "receipt_recovery_v1",
        +  "application_workflow_v1"
        +]
  3. 2 tool updatesv0.7.1
    • Changedvalidate_integration7 fields changed
      • addedInput schema / properties / arm
        Added value: +{
        +  "description": "Probe ID from the matrix to arm before running the app. Requires session_id.",
        +  "type": "string"
        +}
      • addedInput schema / properties / cancel
        Added value: +{
        +  "description": "Abort the attempt or retry failed cleanup. Requires session_id and run_id; cannot combine with arm or probe.",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / execute
        Added value: +{
        +  "description": "Drive the configured app and evaluate the frozen receipt suite. Requires session_id/run_id; configuration acknowledgment alone is not proof.",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / probe
        Added value: +{
        +  "description": "Probe ID from the matrix to evaluate after running the app. Requires session_id and run_id.",
        +  "type": "string"
        +}
      • addedInput schema / properties / receipt_config
        Added value: +{
        +  "additionalProperties": false,
        +  "properties": {
        +    "app_version": {
        +      "description": "Declared application build identifier; not code attestation.",
        +      "type": "string"
        +    },
        +    "observation_seconds": {
        +      "description": "Observation window after each checkpoint; default 2 seconds. Verdict covers only measured windows.",
        +      "maximum": 5,
        +      "minimum": 0.1,
        +      "type": "number"
        +    },
        +    "purchase_context": {
        +      "description": "Application routing custom_data, e.g. test workspace ID. Never credentials.",
        +      "type": "object"
        +    },
        +    "purchase_marker": {
        +      "additionalProperties": false,
        +      "description": "Where the normal receipt includes its Paddle transaction ID; same-record recipient/marker matching is required.",
        +      "properties": {
        +        "field": {
        +          "enum": [
        +            "html",
        +            "text",
        +            "subject",
        +            "tags"
        +          ],
        +          "type": "string"
        +        },
        +        "tag_name": {
        +          "type": "string"
        +        }
        +      },
        +      "required": [
        +        "field"
        +      ],
        +      "type": "object"
        +    },
        +    "recipient_a": {
        +      "description": "Expected test recipient for purchase A, fixed before execution.",
        +      "type": "string"
        +    },
        +    "recipient_b": {
        +      "description": "Different test recipient for the different-customer case.",
        +      "type": "string"
        +    },
        +    "source_lookup_required": {
        +      "description": "Whether the app reads the source provider during its normal webhook handler.",
        +      "type": "boolean"
        +    },
        +    "webhook_url": {
        +      "description": "Exact normal application webhook URL, including path.",
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "webhook_url",
        +    "recipient_a",
        +    "recipient_b",
        +    "purchase_marker",
        +    "app_version",
        +    "source_lookup_required"
        +  ],
        +  "type": "object"
        +}
      • addedInput schema / properties / run_id
        Added value: +{
        +  "description": "Attempt ID returned by arm. Required with probe or cancel; prevents stale calls from consuming a newer attempt.",
        +  "type": "string"
        +}
      • addedInput schema / properties / suite
        Added value: +{
        +  "description": "Configure the declared application receipt suite on this session.",
        +  "enum": [
        +    "receipt_delivery_v1",
        +    "receipt_recovery_v1"
        +  ],
        +  "type": "string"
        +}
    • Changedverify_behavior1 field changed
      • changedInput schema / properties / sandbox_id / description
        Previous value: -"OPTIONAL. The sandbox from the run. Pass with flow_run_id to save the diff onto that run's receipt."New value: +"OPTIONAL. The sandbox from the run. Pass with flow_run_id to save the diff onto that run's receipt. This binds the pattern to that sandbox's provider; provide it when verifying a provider integration."
  4. 2 tool updatesv0.7.0
    • Removedimport_spec
    • Addedvalidate_integration
  5. 1 tool updatev0.5.9
    • Changedlist_workflows2 fields changed
      • addedInput schema / properties / spec_slug
        Added value: +{
        +  "description": "A human name like 'resend' or 'stripe'. Use this when you do not have a spec_id — you do not need to look one up first.",
        +  "type": "string"
        +}
      • removedInput schema / required
        Removed value: -[
        -  "spec_id"
        -]
  6. 2 tool updatesv0.5.7
    • Addedlist_scenarios
    • Addedset_scenario
  7. 2 tool updatesv0.5.6
    • Changedfix_bug1 field changed
      • changedInput schema / properties / path / description
        Previous value: -"OPTIONAL. Absolute path to the project. Defaults to the current working directory."New value: +"OPTIONAL. Absolute path to the project, as it is. Defaults to the current working directory. Do NOT copy the project first: fix_bug returns a diff and never writes to your tree."
    • Changedprove_fix1 field changed
      • changedInput schema / properties / path / description
        Previous value: -"OPTIONAL. Project dir. Defaults to cwd."New value: +"OPTIONAL. The project root, as it is. Defaults to cwd. Do NOT copy the project first: prove_fix builds its own before/after copies and never writes to your tree. A path outside the workspace is refused."
  8. 13 tool updatesv0.5.1
    • Addedcoach
    • Addedfind_bugs
    • Addedfix_bug
    • Addedguide
    • Changedimport_spec1 field changed
      • changedInput schema / properties / name / description
        Previous value: -"Optional friendly name for the spec. Defaults to info.title from the spec, or the URL hostname."New value: +"Optional DISPLAY label only — NOT a lookup key. Defaults to info.title from the spec. To resolve a known API by name (\"Stripe\", \"GitHub\"), call list_specs first."
    • Addedlist_runs
    • Addedlist_specs
    • Addedprove_fix
    • Addedquickrun
    • Addedrun_all_workflows
    • Changedrun_workflow1 field changed
      • addedInput schema / properties / scenario
        Added value: +{
        +  "description": "OPTIONAL failure scenario to exercise (e.g. payment_declined, insufficient_funds, fraud_hold). Toggles the sandbox engine's scenario for the duration of the run, then restores. Use this for 'test with declined card' / 'simulate failure X' intents. Omit for the happy path.",
        +  "type": "string"
        +}
    • Addedsubmit_proof
    • Addedverify_behavior
  9. 3 tool updatesv0.1.1
    • First observedimport_spec
    • First observedlist_workflows
    • First observedrun_workflow

TDQS

A3.9/5.0

Scored across 19 tools

Disambiguation2/5

Many tools overlap in purpose, especially guide/list_workflows/list_scenarios for discovery, quickrun/run_workflow/run_all_workflows/validate_integration for execution, and find_bugs/fix_bug/prove_fix/verify_behavior/submit_proof for bug proof. The extremely long cross-referential descriptions try to disambiguate but often reference each other and missing tools, leaving boundaries unclear.

Naming Consistency3/5

Most names follow a verb_noun snake_case pattern (list_specs, run_workflow, set_scenario), but 'coach' is a noun, 'quickrun' is a single word, and 'guide' is a bare verb. The pattern is readable but not fully consistent.

Tool Count3/5

19 tools is borderline heavy for the scope; several clusters (e.g., three run tools, multiple proof tools, overlapping discovery tools) suggest some tools could be consolidated. The count is not extreme, but it feels inflated by overlapping capabilities.

Completeness3/5

The surface covers discovery, setup, execution, bug fixing and proof, but references to an import_spec tool that is absent create a dead end for custom/private specs. No explicit sandbox deletion or import tool exists, leaving notable gaps in the lifecycle.

Maintenance

ActivityNo data
ResponsivenessSyncing

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to call real API endpoints and verify responses against OpenAPI specs, supporting contract testing, auth presets, spec diffing, and health checks.
    8
    62 npm
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables AI agents to read realistic, coherent data—repos, pull requests, issues, commits, users—through API-compatible, read-only replicas of 21 developer platforms including GitHub, Jira, Linear, Slack, Sentry and Salesforce, all preloaded with one simulated company so integrations can be built and tested with no accounts, credentials, rate-limit worry, or cleanup.
    10
    454 npm
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI coding agents to build, debug, and test hosted mock REST APIs by scaffolding persistent CRUD resources, importing OpenAPI specs, simulating requests, auto-healing failing endpoints, and injecting latency or error faults. It also exposes live OpenAPI and TypeScript contract resources for each workspace.
    MIT