This MCP server lets an AI assistant generate and run chaos test harnesses from OpenAPI specs.
Check provider readiness (doctor): verify the configured LLM (Ollama/OpenAI/Anthropic) is reachable and the model is available.
Inspect a spec (inspect_spec): load and validate an OpenAPI spec from a file path or URL and list its operations.
Generate a harness (generate_harness): produce a ready-to-run chaos test harness (WireMock stubs, contract tests, robustness probes, chaos scenarios) into a target directory, with options for offline mode, review, forcing, and revision limits.
List chaos scenarios (list_chaos_scenarios): enumerate the failure scenarios in a generated harness.
Activate a chaos scenario (activate_chaos): turn on one specified scenario in the running WireMock (deactivating others).
Deactivate chaos (deactivate_chaos): turn off all chaos scenarios so stubs behave normally again.
Runs API Shakedown locally with Ollama as the default LLM provider, enabling generation of WireMock stubs, contract tests, and chaos scenarios without an API key and without data leaving the machine.
Allows API Shakedown to use OpenAI cloud models as the LLM provider for generating test harnesses, with bring-your-own-key configuration.
Give it an OpenAPI spec. Get back a ready-to-run test harness: WireMock stubs, contract tests, robustness probes, and chaos scenarios, written by a team of LLM agents that review each other's work.
Runs locally on Ollama by default: no API key, no data leaves your machine. Cloud models (OpenAI, Anthropic) are bring-your-own-key. Available as a CLI, a Docker image, and an MCP server so an AI assistant can drive it.
Status: alpha (0.1.2). Works end to end on the example spec with a local 8B model.
A real run on the bundled example with qwen3:8b on a laptop, sped up: generation
took about 9 minutes. Recording: docs/demo.cast.
Related MCP server: agentify
Teams that consume an API need three things that are tedious to write by hand and go stale quickly:
A realistic mock of the API to develop and test against.
Contract tests that tell you when the real API stops matching its spec.
Failure drills: what does your client do when the API is slow, flaky, rate limited, returns garbage, or goes down?
API Shakedown generates all three from the spec you already have, and keeps them honest:
the LLMs never write code that runs. They propose data (response bodies, test
cases, failure scenarios, test records), and code validates every item against the
spec, feeds problems back to the model, and falls back to deterministic output when a
model cannot get it right. Without any model at all (--offline), every part still works.
Situation | Typical teams | What API Shakedown gives you |
Building before the backend exists | Frontend, mobile, partner integrators | A virtual service generated from the agreed spec, so work starts on day one instead of after the backend ships or after hand-written mocks. |
Depending on APIs you don't control (payments, shipping, identity checks, credit bureaus) | Fintech, e-commerce, logistics, insurance | Vendor sandboxes are often rate-limited, flaky or billed per call. Test locally what your app does when the provider times out, rate-limits you, or returns 503 twice before recovering. |
Proving clients survive failure | Microservice teams, SRE game days | Switch timeouts, rate limits, flaky calls and partial outages on and off while your client runs, and confirm retries, backoff and fallbacks really work. |
Catching spec drift before release | API owners, platform and public-API teams | Contract tests run against a real deployment in CI and fail when the implementation stops matching its spec. |
Hardening input validation | Any API team | Robustness probes check that malformed requests get a 4xx, never a 500. A complement to a security review, not a replacement. |
Regulated or offline environments | Banking, healthcare, government | Everything runs on a local model: the spec never leaves the machine, where many AI tools cannot be used at all. |
Migrating or rewriting a service | Legacy modernisation | Generate contract tests from the existing spec and run them against the new implementation to confirm it behaves the same. |
Working through an AI assistant | Developers using Claude and other MCP clients | The assistant generates the harness and runs failure drills on request. |
It is not a load-testing tool or a security scanner, and the virtual service does not keep state between calls; see Limitations.
Prerequisites: Python 3.11+, Docker (for WireMock), and Ollama with
a model (ollama pull qwen3:8b) unless you use --offline or a cloud model.
pip install api-shakedown
shakedown doctor # is the model ready?
shakedown run openapi.yaml --out my-harness # minutes on a local model
cd my-harness
docker compose up -d # WireMock with your stubs on :8080
pip install -r requirements.txt
SHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.py
python chaos/chaos.py on flaky-start # now point your client at :8080
python chaos/chaos.py offTry it on the bundled example: shakedown run examples/petstore.yaml --out petstore.
A real run on the example spec with qwen3:8b:
Petstore 1.0.0: 5 operations (ollama / qwen3:8b)
Generating test data...
Test data ready (3 Pet).
Generating stubs, contract tests, and chaos scenarios...
stubs: done
contracts: done
chaos: done
review contracts: approved
review stubs: approved
review chaos: approved
Wrote 14 files to harness in 436s: 15 stubs, 11 contract cases, 11 chaos scenarios, 30 probes.
Next: cd harness && docker compose up -d (then see README.md)my-harness/
├── README.md # generated: how to run it, every scenario, review notes
├── docker-compose.yml # WireMock 3 serving the stubs (port: SHAKEDOWN_WIREMOCK_PORT)
├── wiremock/mappings/ # stub mappings
├── tests/
│ ├── test_contract.py # contract tests (cases in contract_cases.json)
│ └── test_robustness.py # malformed-request probes (cases in robustness_cases.json)
├── chaos/
│ ├── chaos.py # switch failure scenarios on/off (standard library only)
│ └── chaos_scenarios.json
├── data/dataset.json # test records and edge-case values per field
├── requirements.txt # pytest, httpx, openapi-schema-validator
└── shakedown.json # manifest: counts, review outcome, generation notesPart | What it does |
Stubs | Every documented response of every operation. Known ids return their own record, unknown ids return the documented 404, and any other response can be requested with |
Contract tests | Happy paths, documented error statuses, and boundary values. Each checks the status code, |
Robustness probes | Invalid bodies and parameters derived from the schemas (missing required fields, wrong types, out-of-range values, broken JSON). A robust server answers 4xx, never 5xx. |
Chaos scenarios | Seven baseline failures every client should survive, plus scenarios the LLM targets at your API. |
Test data | Realistic, consistent records per entity, shared by the stubs and the tests, plus edge-case values for every field. |
Baseline chaos scenarios:
Scenario | What happens | A resilient client should |
| Every endpoint returns 503 | Fail fast, back off, not hammer the API |
| Every response takes 3 s | Stay responsive |
| Every response takes 30 s | Give up on its own and free resources |
| Connections are reset | Retry idempotent calls with backoff |
| First 2 calls fail with 503, then recover | Retry and succeed on the third attempt |
| First 3 calls return 429 | Honour |
| Success bodies are cut off mid-JSON | Report a parse error, never crash |
LLM-targeted scenarios from a real run include createPet fails 3 times with 503 then recovers, listPets is rate limited, and getPet loses required fields.
flowchart LR
spec[/"OpenAPI spec"/] --> load["Load, validate,<br/>normalize, chunk"]
load --> data["Test data agent"]
data --> stubs["Stub agent"]
data --> contracts["Contract agent"]
data --> chaos["Chaos agent"]
stubs --> review{"Reviewer agent"}
contracts --> review
chaos --> review
review -- "blocker: revise with feedback" --> stubs
review -- "blocker" --> contracts
review -- "blocker" --> chaos
review -- "approved, or budget spent" --> coord["Coordinator"]
coord --> harness[/"Harness directory"/]Spec layer: loads YAML/JSON from a path or URL, validates it (problems become
warnings), inlines $refs safely, and splits operations into prompt-sized chunks so
large specs fit small models.
Test data agent runs first: the stubs and tests share its records.
Stub, contract, and chaos agents run in parallel.
Reviewer judges each artifact. Blockers send it back to its author with the feedback, for a bounded number of rounds; suggestions are kept as notes.
Coordinator assembles the harness and its README, including anything the reviewer still rejected when the revision budget ran out.
Every agent follows the same pattern, which is what keeps small local models reliable:
sequenceDiagram
participant A as Agent
participant M as LLM
participant C as Code checks
A->>M: task + JSON schema (constrained decoding on Ollama)
M-->>A: proposal (bodies, cases, scenarios, records)
A->>C: validate against the spec
alt problems found
C-->>A: exact errors, e.g. "listPets 200: 'id' is a required property"
A->>M: fix these problems
M-->>A: corrected proposal
end
A->>A: keep valid items, fall back to deterministic ones for the restThings code guarantees, so the model never has to:
Request matching, URL patterns, priorities, and path-parameter types in the stubs.
Response bodies valid against their schemas.
Test inputs that route (valid path parameters, required query parameters), success cases that use ids that exist, 404 cases that use ids that do not.
Authentication from the spec's security schemes.
Chaos scenarios that are internally consistent (a "flaky" failure must recover, a rate limit is a 429, a partial outage leaves some endpoints up).
Reviewer findings that cite real items, and blockers only on what the author controls.
pip install api-shakedown # local models via Ollama; includes the MCP server
pip install "api-shakedown[cloud]" # plus OpenAI and AnthropicOr use the Docker image, which includes everything: see Docker.
Generated harnesses need Docker (for WireMock) and Python for their tests.
shakedown doctor Check the LLM provider is reachable and ready
shakedown inspect SPEC [--json] Show the operations the agents will see
shakedown run SPEC [options] Generate a harness
shakedown versionshakedown run options:
Option | Default | |
|
| Where to write the harness (refuses a non-empty directory) |
| off | No LLM: deterministic output in seconds |
| review on | Skip the reviewer (about 3x faster on the example) |
|
| Review/revise rounds |
| off | Write into a non-empty directory (only overwrites harness files) |
| from settings | Override the LLM for this run |
Using the harness:
docker compose up -d # stubs on http://localhost:8080
SHAKEDOWN_SELECT_STATUS=1 pytest tests/test_contract.py # against the stubs
python chaos/chaos.py list | on <scenario> | off # failure drills
SHAKEDOWN_TARGET_URL=https://api.example.com/v1 pytest tests # against a real serverPowerShell: $env:SHAKEDOWN_SELECT_STATUS = "1"; pytest tests/test_contract.py.
Settings come from environment variables or a .env file
(see .env.example). Everything is optional.
Variable | Default | |
|
|
|
| per provider |
|
|
| Where Ollama runs |
| Endpoint override, e.g. an OpenAI-compatible server | |
| 600 (Ollama) / 120 | Seconds per request |
|
| Ollama context size |
| model default | Ollama: turn a reasoning model's thinking on/off |
| Per-agent models, e.g. | |
| thinking off for generators | Per-agent thinking, e.g. |
| Bring your own key |
Mixing models works well: generate locally, review with a stronger cloud model:
SHAKEDOWN_AGENT_MODELS='{"reviewer": "anthropic:claude-sonnet-5-5"}' shakedown run openapi.yamlGenerated harnesses read SHAKEDOWN_TARGET_URL, SHAKEDOWN_SELECT_STATUS,
SHAKEDOWN_API_KEY, SHAKEDOWN_BASIC_AUTH, SHAKEDOWN_TIMEOUT, and
SHAKEDOWN_WIREMOCK_PORT / SHAKEDOWN_WIREMOCK_URL.
shakedown-mcp (stdio) lets an MCP client such as Claude Code or Claude Desktop
generate harnesses and run failure drills:
Tool | |
| Is the LLM provider ready? |
| List a spec's operations and warnings |
| Generate a harness, with progress notifications |
| Scenarios in a harness |
| Switch a scenario on/off in a running WireMock |
claude mcp add api-shakedown -- uvx api-shakedown mcp # Claude Code, nothing to install
claude mcp add api-shakedown -- shakedown-mcp # Claude Code, after pip installName in the official MCP Registry:
io.github.anzar-ahsan-commits/api-shakedown.
Other clients (mcpServers in their config):
{
"mcpServers": {
"api-shakedown": {
"command": "uvx",
"args": ["api-shakedown", "mcp"],
"env": { "SHAKEDOWN_PROVIDER": "ollama" }
}
}
}Pass absolute paths. Generating with a local model takes minutes; if your client gives up
on long tool calls, use offline or review: false, or raise its tool timeout.
docker build -t api-shakedown .
docker run --rm -v "$PWD:/work" api-shakedown run /work/openapi.yaml --out /work/harness
docker run -i --rm -v "$PWD:/work" --entrypoint shakedown-mcp api-shakedown # MCP serverThe image reaches Ollama on the host at host.docker.internal. Details for Linux,
PowerShell, and cloud keys: DOCKER.md.
flowchart TB
subgraph app["api-shakedown"]
cli["CLI: shakedown"] --> pipeline["pipeline"]
mcp["MCP server: shakedown-mcp"] --> pipeline
pipeline --> agents["agents: testdata, stubs, contracts,<br/>chaos, reviewer, coordinator"]
agents --> domain["spec, wiremock, contracts,<br/>chaos, testdata (code checks)"]
pipeline --> harness["harness writer + templates"]
end
subgraph core["agentpanel (reusable core)"]
workflow["Workflow: parallel specialists,<br/>cross-review, bounded revision"]
structured["structured output:<br/>validate, feed back, retry"]
llm["providers: Ollama, OpenAI, Anthropic"]
end
agents --> workflow
agents --> structured
structured --> llmThe orchestration core is published separately as agentpanel: a small, domain-agnostic library for "specialists draft, reviewers critique, authors revise, a coordinator decides" workflows on LangGraph. It never imports the application; a test enforces that.
packages/agentpanel/ # the reusable core (its own PyPI package)
src/api_shakedown/ # the application
spec/ # loading, validation, $ref resolution, edge cases, samples
agents/ # the six agents
templates/ # runners copied into every harness
examples/petstore.yaml # example spec
tests/ # offline tests; `-m live` for Ollama + Docker testsSpeed on local models: a full run on the 5-operation example takes about 7 minutes
with qwen3:8b, most of it review; --no-review takes about 2.5 minutes.
Small reviewers are imperfect: an 8B reviewer sometimes raises a false blocker, which costs a revision round. Use a stronger model for the reviewer if you can.
Specs: OpenAPI 3.0-3.2. Swagger 2.0 must be converted first; references to other files are not followed; self-referencing schemas are not validated.
Stubs are static: a create call returns a new record, but stubs do not keep state between calls.
Probes against stubs only cover operations that document a 4xx; they are meant for real servers.
Contract tests assume data exists: against an empty real server, cases that read a record fail until you create it (setup steps are on the roadmap).
Scope: not a load-testing tool or a security scanner.
Large specs are slow on a laptop: the Typesense API (79 operations) takes several
hours end to end with qwen3:8b on a laptop GPU, versus minutes for small specs. A
bigger GPU or a cloud model is much faster; --offline takes seconds. Reports from more
real-world specs are very welcome.
Faster runs on large specs (smaller prompts per batch, progress per batch)
Setup steps for contract tests, so they create the data they read on a real server
Stateful stubs (WireMock response templating, created records visible to later calls)
Auth-failure scenarios (401/403) and "valid but nasty input must not 5xx" probes
Review of the test dataset
More harness targets (k6 load profiles, Postman collections)
git clone https://github.com/anzar-ahsan-commits/api-shakedown.git
cd api-shakedown
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e packages/agentpanel -e ".[dev]"
ruff check . && ruff format --check . && mypy
pytest # offline: LLMs are faked, about 20 s
pytest -m live # also uses your local Ollama and DockerContributions are welcome: see CONTRIBUTING.md. Security issues: see SECURITY.md.
MIT © 2026 Anzar Ahsan
Turn one chaos scenario on in the harness's running WireMock (others go off).
| Name | Required | Description | Default |
|---|---|---|---|
| harness_dir | Yes | ||
| scenario_id | Yes | ||
| wiremock_url | No | http://localhost:8080 |
| Name | Required | Description |
|---|---|---|
| active | Yes | |
| wiremock_url | Yes | |
| mappings_added | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare idempotentHint=true and destructiveHint=false, and the description adds a genuinely important side effect not captured there: activating one scenario switches the others off. It still omits error behavior for unknown scenario_id or unreachable WireMock, and whether the state persists across restarts.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler; the exclusivity constraint is packed into the parenthetical rather than padded out.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Output schema exists so return values need not be explained, and annotations cover the safety profile. The remaining gap is the complete absence of parameter guidance for a 3-param tool with 0% schema coverage, which leaves the agent guessing on harness_dir format and scenario_id values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and the description never mentions harness_dir, scenario_id, or wiremock_url. At best it hints at a harness and a scenario, but gives no format, expected values, or default behavior (wiremock_url default is only in the schema).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (turn on) and resource (one chaos scenario in the harness's running WireMock), which clearly separates it from the sibling deactivate_chaos. The parenthetical clarifies the exclusive activation semantics, though it does not explicitly name the sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: activating one scenario implicitly means using deactivate_chaos (or activating another) to turn it off. There is no explicit when-to-use/when-not or prerequisite guidance (e.g., harness must already be generated/running).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Turn all chaos scenarios off: the stubs behave normally again.
| Name | Required | Description | Default |
|---|---|---|---|
| wiremock_url | No | http://localhost:8080 |
| Name | Required | Description |
|---|---|---|
| active | Yes | |
| wiremock_url | Yes | |
| mappings_added | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare idempotentHint=true and destructiveHint=false, so repeat-safety is covered. The description adds genuine behavioral context beyond that by stating the observable result — all scenarios off and stubs returning to normal — but says nothing about what happens if no scenarios are active or whether it affects a specific WireMock instance.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short, front-loaded sentence with no filler; the scope ('all') and the resulting state are both stated immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values needn't be described, and the tool is simple with no nested objects or enums. The only gap is the unaddressed wiremock_url parameter, which is minor given its default.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single wiremock_url parameter has 0% schema description coverage and is never mentioned in the description, so an agent gets no explanation of which WireMock server is targeted. The default (http://localhost:8080) in the schema mitigates the risk but the description does not compensate for the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (turn off) and resource (all chaos scenarios) with a clear outcome (stubs behave normally again). The contrast with the sibling activate_chaos is implied by the name symmetry and the word 'off', but the description never names the alternative explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance is given: it doesn't say whether this is the global reset versus list_chaos_scenarios or activate_chaos, nor what prerequisites exist (e.g. whether scenarios must already be active). Usage must be inferred entirely from the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check that the configured LLM provider is reachable and the model is available.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| ok | Yes | |
| model | Yes | |
| detail | Yes | |
| provider | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so the safety profile is covered. The description adds the valuable detail that the check covers both connectivity and model availability, but says nothing about whether it makes a live network call, timeout behavior, or what a failure looks like.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence with no filler, front-loading the action and both things being verified. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description need not explain return values, and with annotations covering read-only safety and no parameters to document, the definition is nearly complete. It could go slightly further by stating when to run the check relative to the harness/chaos siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there are no parameter semantics for the description to carry; the baseline for a no-arg tool is 4. Nothing is missing or misrepresented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description pairs a specific verb ("Check") with a precise object: LLM provider reachability and model availability. It is clearly distinct from the sibling tools (inspect_spec, generate_harness, chaos controls), though it does not explicitly name any of them as alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The diagnostic framing implies when to use it (verifying setup before running a harness or generating one), but the description never states this condition or points to any sibling. No exclusions or prerequisites are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate a ready-to-run chaos test harness for an OpenAPI spec into out_dir.
offline=true skips the LLM (deterministic, seconds); otherwise this takes minutes on a local model, and review=false roughly halves that. Refuses a non-empty out_dir unless force=true (which overwrites only the harness's own files).
| Name | Required | Description | Default |
|---|---|---|---|
| spec | Yes | ||
| force | No | ||
| review | No | ||
| offline | No | ||
| out_dir | Yes | ||
| max_revisions | No |
| Name | Required | Description |
|---|---|---|
| files | Yes | |
| notes | Yes | |
| out_dir | Yes | |
| reviews | Yes | |
| summary | Yes | |
| next_steps | Yes | |
| unresolved | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare idempotentHint=false and destructiveHint=false, but the description adds real behavioral detail: the refusal to write into a non-empty out_dir, that force overwrites only the harness's own files (not the whole directory), and the runtime profile (seconds offline vs minutes on a local model). This is exactly the extra context annotations cannot carry.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences: the core action and output location lead, then the speed/flag tradeoffs, then the safety constraint last. Every clause carries information and nothing repeats the name or schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, return values need not be described, and the description covers the risky behaviors an agent must know before calling. The only real gap is the undocumented max_revisions parameter, so it is near-complete rather than fully so.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does for four of six parameters: offline, review, force, and out_dir all get meaning beyond their names. spec and max_revisions are left unexplained, which keeps it below a 5 despite the strong overall compensation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb (generate) plus resource (ready-to-run chaos test harness) and names the input (an OpenAPI spec) and output location (out_dir). That is enough to separate it from siblings like inspect_spec or activate_chaos, though it never names or contrasts an alternative explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives concrete operational guidance for the flags: offline=true for a fast deterministic run, review=false to roughly halve an LLM run, and force=true to proceed when out_dir is non-empty. There is no guidance on when to use this tool versus the other five siblings, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Load and validate an OpenAPI spec (file path or http(s) URL) and list its operations.
| Name | Required | Description | Default |
|---|---|---|---|
| spec | Yes |
| Name | Required | Description |
|---|---|---|
| title | Yes | |
| version | Yes | |
| warnings | Yes | |
| operations | Yes | |
| openapi_version | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already establishes this is a safe, non-mutating operation, lowering the disclosure bar. The description adds the source types (file path or http(s) URL) and the resulting behavior (list operations), but says nothing about validation failure behavior, auth requirements for remote URLs, or latency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence with no redundant or filler content; every clause (sources, validation, operation listing) earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described, and the tool is a simple single-parameter read. The description covers the essential call contract; only error/validation-failure semantics are omitted, which is a minor gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and the lone parameter is titled only 'Spec', so the description must compensate. It does add key meaning by specifying the accepted formats (file path or http(s) URL), but does not clarify relative/absolute path expectations or URL constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States specific verbs (load, validate, list) on a clear resource (OpenAPI spec) and names the accepted source formats. However, it offers no differentiation from sibling tools like 'doctor' or 'generate_harness', which an agent might reasonably confuse or pair with this one.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the purpose (inspect a spec before generating a harness or running chaos), but there is no explicit when-to-use, when-not, or alternative selection guidance, and no mention of how it relates to the siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the chaos scenarios of a generated harness.
| Name | Required | Description | Default |
|---|---|---|---|
| harness_dir | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe, non-mutating read, so the description's only added behavioral signal is the implicit 'generated harness' precondition. It omits what happens if the harness_dir doesn't exist or isn't a generated harness, and says nothing about result volume or pagination.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no wasted words. It is efficient, though the brevity contributes to the under-specification noted elsewhere.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With readOnlyHint covering safety and an output schema covering return values, the description needn't explain results. But for a tool taking a harness_dir with no schema documentation, it should at least clarify the parameter's expected form and the precondition that a harness exists, which it does not.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for the single required parameter harness_dir, so the description carries the full burden. It only implies the parameter points at a generated harness, giving no format, path convention, or error behavior to compensate for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('List') and resource ('chaos scenarios of a generated harness'), so the agent knows exactly what it returns. It does not, however, distinguish itself from siblings like activate_chaos/deactivate_chaos or clarify how it relates to generate_harness, which would be needed for a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this versus sibling tools such as inspect_spec or the chaos activate/deactivate tools, nor any stated precondition (e.g., that a harness must already be generated). The agent must infer usage entirely from the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0activate_chaosdeactivate_chaosdoctorgenerate_harnessinspect_speclist_chaos_scenariosScored across 6 tools
Each tool targets a clearly distinct step: provider health check (doctor), spec loading/validation (inspect_spec), harness generation (generate_harness), and scenario listing/toggling (list/activate/deactivate_chaos). There is no meaningful overlap between any pair.
Five of six tools follow a clean snake_case verb_noun pattern (inspect_spec, generate_harness, list_chaos_scenarios, activate_chaos, deactivate_chaos). The lone outlier 'doctor' is a bare noun, a minor but visible deviation from the otherwise consistent convention.
Six tools is well-scoped for a focused chaos-testing pipeline, with each tool mapping to a necessary step from provider check to scenario toggling. Nothing feels padded or missing at the count level.
The surface covers the core lifecycle: verify provider, inspect spec, generate harness, and toggle scenarios. Minor gaps exist around harness execution/teardown (no explicit start/stop/reset of WireMock, and deactivate is all-or-nothing), but these are workable around.
Build, validate, and manage API simulations in WireMock Cloud from MCP-compatible AI agents.
Point Gecko at an OpenAPI spec; get first-call-correct, auth-hidden agent tools.
Free AI test helpers: injection inputs, OWASP mapping, release plans, response and tool-call packs.
AI-callable tools for API mocking, testing, monitoring, security, and automation.
