Skip to main content
Glama
Conversational API Test MCP

Conversational API Test MCP

by Mr-Lucasz

Conversational API Test MCP

Português

An API test bench your AI agent can drive — and a QA engineer named Vander to drive it.

Most API tools assume a human at the keyboard. This one assumes an agent: you describe your API once in YAML, and the agent runs requests, chains them, and reviews endpoints through a testing heuristic, while the server keeps three promises the agent cannot keep on its own — the request sent is exactly the one declared, the answer costs few tokens, and your credentials never enter the conversation.

It is an MCP server over stdio. It works with Cursor, VS Code + GitHub Copilot, Claude Code, and any other MCP stdio client.

What a session looks like

A real run, recorded on 2026-10-07 against httpbin.org with the definition in examples/httpbin.example.yaml. Only safe methods were sent.

You: /vander .mcp/api/httpbin.yaml whoami

Vander: Vander here. Reviewing GET /bearer on httpbin.org. The plan has 13 automatic checks; 4 of them would send POST, PUT, PATCH or DELETE, so I left those out.

8 passed, 1 failed, 4 skipped.

  • V ✅ OPTIONS → 200. POST / PUT / PATCH / DELETE not sent.

  • A ❌ No credentials → 401, as it should be. But Bearer vander-invalid-token → 200 with "authenticated": true.

  • N ⚪ Nothing to negate: no body, no query parameters, no id in the path.

  • D ✅ Baseline 200, application/json with a JSON body, $.authenticated present.

  • E ✅ No 5xx, and nothing leaked in the 401.

  • R ✅ 196 ms against a 1000 ms budget. One sample, not a load test.

One finding: the endpoint accepts any bearer token. It checks that a token is present, not that it is valid. For an echo service like httpbin that is to be expected; on a real API it is the difference between authentication and a header check.

Not tested: the four state-changing verbs, an expired token, another user's token, rate limiting.

endpoint:
  id: whoami
  method: GET
  path: /bearer
summary:
  pass: 8
  fail: 1
  skipped: 4
  manual: 6
checks[13]{id,result,detail}:
  V-options,pass,status 200 in 655ms
  V-post,skipped,state-changing (POST)
  V-put,skipped,state-changing (PUT)
  V-patch,skipped,state-changing (PATCH)
  V-delete,skipped,state-changing (DELETE)
  A-none,pass,status 401 in 605ms
  A-invalid,fail,status 200 in 605ms
  D-baseline,pass,status 200 in 142ms
  D-content-type,pass,"content-type \"application/json\", body is JSON"
  D-captures,pass,""
  E-no-5xx,pass,""
  E-no-leak,pass,""
  R-duration,pass,142ms
failures[1]{id,title,expected,actual,note}:
  A-invalid,Request with an invalid token is rejected,"status in [401, 403]",status 200 in 605ms,"{\"authenticated\":true,\"token\":\"[REDACTED]\"}"
hint: "4 state-changing probes not sent; ask the user, then rerun with includeDestructive: true"

How this was recorded: a script drove the built server over MCP stdio and saved the tool output shown above, unedited. Vander's reply was then written by Claude from that output, following the vander prompt. Every number in the reply comes from the output; timings will differ on your machine.

Related MCP server: API Debugger MCP

Three bets

1. The agent chooses; the server executes. The agent picks a requestId, and the server builds and sends the request from the YAML. There is no hand-written curl in the chat to get subtly wrong, and the same call gives the same request tomorrow.

2. Tokens are a budget. responseDetail, jsonPathSelect and maxBodyChars decide how much of a response comes back; execute_api_flow runs a whole scenario in one call; results are encoded as TOON rather than JSON.

3. A heuristic is worth more than a personality. Vander is a persona, but what he does is fixed: a checklist built by code and verdicts computed from status codes and timings. Research on personas in prompts found they do not, by themselves, make a model more accurate (sources), so the personality is there for the conversation and the rigour lives in run_vander_checks.

How it works

You test an API by talking to your agent. Three pieces make that possible:

Piece

Where

What it is

If you know Postman

Definitions

.mcp/api/*.yaml

Your requests, described once and committed with the code.

The collection

Secrets and environments

.env.mcp.local

Base URLs, tokens and passwords. Stays on your machine.

The environment

The conversation

Your MCP client

You ask in plain language; the agent calls this server; the server sends the request.

Send, Runner and the Tests tab

Getting started

Requires Node.js ≥ 20.

1. Build the server and register it in your MCP client.

npm ci
npm run build
{
  "servers": {
    "conversational-api-test-mcp": {
      "type": "stdio",
      "command": "node",
      "args": ["/absolute/path/to/conversational-api-test-mcp/dist/index.js"]
    }
  }
}

That is the VS Code format. Cursor and Claude Code read the same entry under "mcpServers" instead of "servers", without the "type" line.

2. Open the project whose API you want to test and say hi Vander.

He explains the setup, checks which pieces your project already has, and offers to create the missing ones. Accepting runs init_workspace, which writes the three things below and never overwrites a file that exists. You can also create them by hand.

3. The .mcp/api/ folder and your first YAML. One file per service. Each request gets an id, which is how you and the agent refer to it.

# .mcp/api/weather.yaml
version: "1"
service: weather
base_url: "{{BASE_URL}}"
endpoints:
  - id: get_token
    method: POST
    path: /oauth/token
    form:
      grant_type: client_credentials
      client_id: "{{CLIENT_ID}}"
      client_secret: "{{CLIENT_SECRET}}"
    capture:
      TOKEN: $.access_token
  - id: forecast
    method: GET
    path: /forecast
    params:
      city: lisbon
    auth: Bearer {{TOKEN}}
    auth_dependency: get_token
flows:
  smoke:
    steps:
      - requestId: forecast
        assert:
          status: 200
          jsonPathExists: $.days

You rarely have to write this by hand. Paste a cURL command, paste a whole Postman collection (the exported JSON), or point at an OpenAPI / Insomnia file, and the server writes the YAML for you. Tokens that come along are moved to .env.mcp.local and replaced by {{VARIABLE}}, so the YAML stays safe to commit.

4. The .env.mcp.local file. Create it at the project root, next to .mcp/. It holds what {{...}} stands for. You fill it in yourself, it must be in .gitignore, and its values never go into the chat.

# .env.mcp.local
BASE_URL=https://api.example.com/v1
CLIENT_ID=...
CLIENT_SECRET=...
STAGING_BASE_URL=https://staging.example.com/v1

A prefix turns one YAML into several environments: after "use staging", {{BASE_URL}} reads STAGING_BASE_URL first and falls back to BASE_URL.

5. Ask. No file to write yet? init_workspace drops in a demo against httpbin.org that runs without any secret, and examples/ has three more you can copy into .mcp/api/.

Coming from Postman

You do not rebuild your requests. You hand over what you already have — a cURL command or an exported collection — and keep working from the chat. Everything below was run against the built server; the outputs are what it returned.

Where everything ends up. All of it sits in the project whose API you are testing, not in this repository:

your-project/
├── .mcp/
│   └── api/
│       ├── orders.yaml              # one YAML per service: commit it
│       └── shop.yaml
├── .env.mcp.local                   # base URLs, tokens, passwords: never commit it
├── .gitignore                       # contains .env.mcp.local
└── shop.postman_collection.json     # your export: read, never changed

Paste a cURL

You: turn this into a request called create_order, in orders.yaml

curl -X POST "https://httpbin.org/anything/orders?source=app" \
  -H "Authorization: Bearer demo-token-123" \
  -H "Content-Type: application/json" \
  -d '{"item":"notebook","quantity":2}'

The agent calls import_curl, which answers:

definition: .mcp/api/orders.yaml
created: true
requestId: create_order
method: POST
url: "https://httpbin.org/anything/orders"
secretsMovedToEnvFile[1]: CREATE_ORDER_TOKEN

and writes this file. The query string became params, the JSON body became body, and the token is gone from it:

# .mcp/api/orders.yaml
version: "1"
base_url: https://httpbin.org
endpoints:
  - id: create_order
    method: POST
    path: /anything/orders
    headers:
      Content-Type: application/json
    params:
      source: app
    body:
      item: notebook
      quantity: 2
    auth: Bearer {{CREATE_ORDER_TOKEN}}

The token itself was appended to .env.mcp.local as CREATE_ORDER_TOKEN=..., and .env.mcp.local was added to .gitignore if it was not there. From here on it is "run create_order".

Hand over a collection

Export the collection from Postman (Export → Collection v2.1) and either drop the JSON in the project or paste its content in the chat.

You: here is my Postman collection, shop.postman_collection.json. Convert it to shop.yaml.

The agent calls convert_legacy_to_canonical, which answers:

outputPath: .mcp/api/shop.yaml
source: postman
losses[1]{code,message}:
  POSTMAN_PRE_REQUEST_SCRIPTS,"Some pre-request script lines could not be translated and will not run. Requests: Create order."
endpointCount: 2
# .mcp/api/shop.yaml
version: "1"
service: Shop
base_url: "{{base_url}}"
variables:
  base_url: https://httpbin.org
  rid: "{{$uuid}}"              # from pm.environment.set("rid", …{{$guid}}…)
endpoints:
  - id: postman_0_List_orders
    method: GET
    path: /anything/orders?status=open
    description: List orders
    headers:
      X-Request-Id: "{{rid}}"
    assert:                      # from the pm.test blocks
      status: 200
      maxDurationMs: 1000
      checks:
        - path: $.args.status
          equals: open
  - id: postman_1_Create_order
    method: POST
    path: /anything/orders
    description: Create order
    body:
      item: notebook
      quantity: 2
    headers:
      Content-Type: application/json
    assert:
      status: 200
    capture:                     # from pm.environment.set("orderItem", pm.response.json().json.item)
      orderItem: $.json.item

Every request in the collection, folders included, is now an endpoint. The losses list is the server telling you what it left behind, and the conversion needs a few minutes of finishing — ask the agent to do it, it previews each change before writing:

What you had in Postman

What the import does

How to finish

The host in each URL

Becomes base_url, taken from the collection variable used as host ({{baseUrl}}/orders) or from the first absolute URL. With neither, base_url stays empty and losses reports POSTMAN_BASE_URL_UNRESOLVED.

Nothing, when it was found. Otherwise "set base_url to https://api.example.com", or put BASE_URL=... in .env.mcp.local and use {{BASE_URL}}.

The Authorization tab (collection, folder or request)

Converted per request, following Postman's inheritance (inherit, noauth): Bearer and Basic become auth, Digest becomes digest_auth, API Key becomes a header or a param. Literal credentials go to .env.mcp.local. Other types (OAuth 2.0, AWS, …), and Basic built from {{variables}}, are reported as POSTMAN_AUTH_UNSUPPORTED.

Nothing for the converted ones. For the rest, auth: Bearer {{TOKEN}}, plus capture + auth_dependency when a login request produces the token.

Pre-request scripts

Generated values become variables: uuid.v4(), crypto.randomUUID() and {{$guid}} → {{$uuid}}, Date.now() → {{$timestamp}}, new Date().toISOString() → {{$isoTimestamp}}, literals as they are. Any other line (signatures, pm.sendRequest, …) is reported in losses, naming the requests.

capture + auth_dependency for tokens fetched by a script; the rest by hand.

Tests (pm.test)

Translated to assert: to.have.status, responseTime below, and on body paths eql, include, lengthOf, exist, above/below, match. pm.environment.set("x", jsonData.a.b) becomes capture. Collection and folder scripts apply to every request below them. What is left is reported in losses, naming the requests.

assert by hand for the rest — see value checks.

Dynamic variables

{{$guid}} / {{$randomUUID}} become {{$uuid}}; {{$timestamp}} (seconds in Postman) becomes {{$timestampSeconds}}.

—

Request names

Become ids such as postman_0_List_orders.

"Rename it to list_orders."

Query string

Stays inline in path; disabled params are dropped.

Works as it is; move it to params if you want to vary it.

Bodies

raw becomes body, x-www-form-urlencoded becomes form, GraphQL becomes a { query, variables } body. form-data and binary files are not mapped and are reported as POSTMAN_BODY_MODE, naming the requests.

Rewrite those as form or body.

Disabled headers and params

Left out, as Postman leaves them out of the request.

—

Postman environments

Imported with import_postman_environment ("import staging.postman_environment.json as STAGING"): each enabled value goes to .env.mcp.local as STAGING_<key>, picked once set_environment STAGING is active. Existing keys are kept; values are never echoed back.

Postman exports initial values only: fill in the empty ones it reports.

The collection file is only read. Keep it, delete it, or leave it out of git — the YAML is the source of truth from now on.

When you ask the agent to look for something to import without naming a file, discover_legacy_api_sources searches the project for *postman*.json, collection*.json, openapi*.{yaml,yml,json}, swagger*.{yaml,yml}, insomnia*.{json,yaml} and *.har. A file with another name is converted when you give its path.

A day of work, in prompts

Moment

You say

First time in the project

"hi Vander, set this project up"

Bringing requests in

"turn this cURL into a request called create_order" · "convert shop.postman_collection.json"

Tidying an import

"set base_url to …" · "rename postman_0_List_orders to list_orders" · "these requests use the token from login"

Running

"run list_orders" · "run it on staging" · "show me what create_order would send"

Chaining

"create an order, then fetch it by the id that came back" · "save that as a flow called smoke"

Checking

"check that it returns 200 and $.items is not empty" · "run the smoke flow"

Reviewing

"Vander, is create_order solid?"

What you can ask

Each line is something you would otherwise do by hand in an API client.

You say

What happens

In Postman you would

"Here is my Postman collection" + the JSON, or the path to it

Converts a collection, an OpenAPI spec, an Insomnia export or a HAR recording to YAML.

Import

"Turn this cURL into a request" + the command

Creates the request in a YAML; any token in it goes to .env.mcp.local.

Import → Raw text

"Run forecast"

Sends that one request and summarises the response.

Click Send

"Run the smoke flow"

Runs the requests of a flow in order, stopping at the first failure.

Collection Runner

— (in CI) npx conversational-api-test-mcp run

Runs every flow and writes a JUnit report; fails the job on a failing step.

Newman / Postman CLI

"Import staging.postman_environment.json as STAGING"

Copies its values to .env.mcp.local under STAGING_.

Import → Environment

"Check the response matches the Order schema in openapi.yaml"

Validates the body against that schema.

tv4 / ajv in Tests

"Log in and list the orders"

Runs the login first, keeps the token, uses it, and logs in again on a 401.

Pre-request script

"Create a post, then fetch it by the id that came back"

Captures a value from one response and uses it in the next request.

pm.environment.set in Tests

"Use staging"

Switches the active environment for the following requests.

Environment dropdown

"Check that it returns 200 and has a days field"

Asserts on status and on a JSONPath.

pm.test in Tests

"Start the export and wait until it is ready"

Polls a request until a field appears.

setNextRequest loop

"Show me what forecast would send"

Builds the request without sending it, credentials hidden.

Console

"Add a request that creates an order with these fields"

Writes a new entry to the YAML, showing a preview first.

New request

"Run the smoke eval of the assistant"

Sends each question several times and passes it on a pass rate; checks answers, citations and latency by code.

—

"Vander, is forecast solid?"

Reviews the endpoint on six axes and reports evidence.

—

"Can Bob read Alice's order?"

Calls the endpoint with each other role's token, expecting 401/403/404 (BOLA).

—

"Are the asserts of the smoke flow any good?"

Breaks each real response in small ways and reports which regressions no assert would notice.

—

"Does the API still match openapi.yaml?"

Sends the requests and validates status and body against the spec.

—

"Tell me if the response shape changes"

Records a snapshot on the first run and reports drift on later ones.

Compare by eye

"Create the CRUD flows for orders"

Writes create → read → update → delete → read-again flows from the endpoints you have.

—

"Convert checkout.har"

Turns a browser recording into requests and a recorded flow, without copying its credentials.

Interceptor / Capture

— (terminal) npx conversational-api-test-mcp mock

Serves the definitions as a mock API.

Mock server

"Export the flows as Arazzo" · "draft a Pact contract"

Writes OpenAPI 3.1, Arazzo 1.0 or Pact v3 under .mcp/contracts/.

Export

Tools

Tool

Purpose

list_api_definitions

List YAMLs under .mcp/api/ (glob, pagination, sort, field projection).

summarize_api_definition

Cheap triage: endpoint ids, variable keys, flow names.

read_api_definition

Full, validated definition.

execute_api_request

Run one endpoint.

execute_api_flow

Run several endpoints in one call — inline steps or a flowName declared in the YAML.

dry_run_request

Show the request that would be sent, without sending it.

assert_response

Assert on the last response: status, latency and value checks.

run_eval

Evaluate an endpoint whose answer varies (LLM / RAG / search): cases × repeats, pass rate, optional samples for the agent to judge.

set_environment

Select the active environment (CURRENT_ENV).

set_environment_variable / get_environment_variable

Session variables.

explain_request_context

Which variable keys are available from where.

init_workspace

Create .mcp/api/, a demo definition, the .env.mcp.local template and the .gitignore line.

upsert_canonical_api_definition

Create / append definitions (dry-run by default).

reorganize_mcp_api_definitions

Merge many YAMLs into fewer (plan, then apply).

summon_vander

Hands the agent the Vander persona when you address him by name.

plan_vander_checks / run_vander_checks

VANDER review of one endpoint: checklist, then automatic execution — one row per check, the id prefix being the axis.

import_curl

Turn a pasted curl command into a request in a YAML.

discover_legacy_api_sources / convert_legacy_to_canonical

Import Postman, OpenAPI, Insomnia or HAR — from a file or from pasted text.

import_postman_environment

Copy a Postman environment export into .env.mcp.local, optionally under an environment prefix.

mutate_assertions

Mutation testing for asserts: score a flow's asserts and get the ones that are missing.

check_contract_drift

Compare real responses with an OpenAPI / Swagger file.

infer_crud_flows

Propose (or write) CRUD flows that end by checking the delete really deleted.

export_contract

Write a definition as OpenAPI 3.1, Arazzo 1.0 or a Pact v3 draft.

Every tool publishes MCP annotations: the read-only ones say so (clients can run them without asking), and the ones that send HTTP or replace files are marked as such.

Recommended order: list_api_definitions → summarize_api_definition → set_environment (if needed) → execute_api_request or execute_api_flow.

Vander

Vander is a persona shipped as an MCP prompt: a senior API QA engineer you talk to instead of issuing tool calls yourself. In clients that expose MCP prompts it shows up as a command (for example /vander), optionally with a definition path and an endpoint id. He reviews an endpoint through the VANDER heuristic:

Axis

Question

Checked automatically

Verbs

What does the path do with methods it does not declare?

Undeclared methods are rejected (405/404/501); OPTIONS does not error.

Authorization

Who can call it?

No credentials and an invalid token are rejected (401/403). With roles, another user asking for this object is refused or gets 404 (BOLA).

Negative

What happens with bad input?

Each body field removed, of the wrong type, null or empty is rejected (4xx); an unknown extra field causes no 5xx. With body_schema, values past minLength/maxLength/minimum/maximum, outside the enum or in a bad format are rejected too.

Data

Does the response say what it should?

Baseline succeeds, Content-Type matches the body, captured JSONPaths exist.

Errors

Does it fail well?

Malformed JSON is a client error; no probe causes a 5xx or leaks a stack trace.

Responsiveness

Is it fast enough?

Baseline answers within maxDurationMs (default 1000).

Each axis also carries manual ideas (expired tokens, idempotency, rate limiting…) that Vander explores with the regular tools.

Two declarations turn manual ideas into automatic checks:

roles:                       # other identities, as the Authorization each one sends
  bob: Bearer {{BOB_TOKEN}}
  support: Bearer {{SUPPORT_TOKEN}}
endpoints:
  - id: get_order
    method: GET
    path: /orders/{{order_id}}
    auth: Bearer {{ALICE_TOKEN}}   # the owner
    allow_roles: [support]          # support may read it: not probed
  - id: create_user
    method: POST
    path: /users
    body: { name: Ana, age: 30 }
    body_schema: openapi.yaml#/components/schemas/NewUser   # or inline

A-role-bob calls get_order with Bob's token and fails if the order comes back: broken object level authorization, first in the OWASP API Security Top 10. Only endpoints that address one object (a {{placeholder}} or a numeric id in the path) are probed, since a list is expected to answer every user. body_schema feeds the boundary, enum and format probes; an OpenAPI import fills it in for you.

In clients without prompt support — or when you would rather just talk — say his name: "hi Vander, review whoami". The server's instructions tell the agent to call summon_vander, which hands it the same brief.

The check tools work without the persona too:

  • plan_vander_checks returns the checklist for a requestId and sends nothing.

  • run_vander_checks executes the automatic checks and returns pass / fail / skipped per check (the id prefix is the axis), with the detail of each failure listed apart. Probes never capture variables. Anything that sends POST, PUT, PATCH or DELETE is skipped unless includeDestructive: true, so reviewing a write endpoint is an explicit decision.

VANDER builds on Stuart Ashman's VADER heuristic and adds an explicit Negative axis — see where the ideas come from.

Definitions

Variables and environments

{{KEY}} is resolved from, in increasing priority: YAML variables → .env.mcp.local → session variables (captures and set_environment_variable).

  • After set_environment (e.g. STAGING), {{BASE_URL}} resolves STAGING_BASE_URL first and falls back to BASE_URL.

  • Unresolved placeholders are left as literal {{KEY}} so the problem is visible.

  • Macros: {{$uuid}}, {{$timestamp}} (ms), {{$timestampSeconds}}, {{$isoTimestamp}}, {{$date}}, {{$date:YYYY-MM-DD HH:mm:ss}}.

  • {{env.NAME}} reads the server's process.env, but only for names listed in MCP_API_ENV_PASSTHROUGH.

Requests

  • base_url may carry a path prefix (https://host/api/v1); path is appended to it. An absolute path overrides base_url.

  • params is sent as the query string; headers, body (JSON) and form (URL-encoded) are interpolated.

  • capture maps a session variable name to a JSONPath on the response; assert checks status, latency and values — see value checks.

Auth

  • auth: Bearer {{TOKEN}} sets the Authorization header.

  • auth_dependency: <endpoint id> runs that endpoint first when its captured variables are missing, and again after a 401 (disable with auth_retry_on_401: false).

  • digest_auth: { username, password } performs HTTP Digest.

  • roles (at the top of the file) and allow_roles (on an endpoint) drive the BOLA checks — see Vander.

Tracing

tracing: true at the top of a definition sends a W3C traceparent header with every request and reports its traceId (on each flow step, and under failures in CI reports), so a failing call can be looked up in New Relic, Datadog, Jaeger or any OpenTelemetry backend. tracing: { url: "https://your-apm/trace/{traceId}" } also returns a link. An endpoint that sets its own traceparent keeps it.

Flows

Each step is an endpoint id or an object:

flows:
  create_and_wait:
    steps:
      - create_job
      - requestId: get_job
        poll: { untilJsonPath: "$.finishedAt", maxAttempts: 10, delayMs: 2000 }
        retry: { max: 3, delayMs: 500 }
        acceptStatus: [200]
        assert: { status: 200, jsonPathExists: "$.result" }
        optional: false

Value checks

assert (on a request, on a flow step, or through assert_response) takes status, jsonPathExists, maxDurationMs and a list of checks. Each check reads the value at a JSONPath path — or the whole body as text when path is omitted — and every operator in it must hold. All failing checks are reported, not just the first.

Operator

Holds when

contains / containsAny / notContains

The text has all of / one of / none of the given strings.

matches

The text matches the regular expression.

ignoreCase

Makes the four above case-insensitive.

equals

The value is exactly this (any JSON value).

min / max

The number is within the bounds.

minLength / maxLength

The string or list has that many characters / items.

includesAll / includesAny

The list at path holds all of / one of these values. With minRatio, includesAll needs only that share of them — retrieval recall.

subsetOf

Every value at path also appears at this other JSONPath — citations among the retrieved documents.

An operator the server does not know is an error when the YAML is read, never a silent pass.

schema checks the whole JSON body against a JSON Schema — written inline, or taken from a file in the project with a JSON pointer, usually your OpenAPI spec:

assert:
  status: 200
  schema: openapi.yaml#/components/schemas/Order   # or an inline { type: object, required: [id] }

OpenAPI 3.0 and 3.1 both work (nullable, internal $refs, formats). A failure names the path and the rule, e.g. schema: /customer must have required property 'name'. The file is re-read when it changes. Converting an OpenAPI file (rather than pasting it) writes this schema on every endpoint for you.

snapshot compares the body with the one recorded the first time it ran, stored in .mcp/snapshots/<name>.json (commit it):

assert:
  snapshot: order                      # shape: keys and value types
  # snapshot: { name: order, mode: values, ignore: [$.requestId] }

shape (the default) catches contract drift (a field that disappeared, changed type or appeared) without breaking on data that changes every day. values compares the values too, with UUIDs, dates, JWTs and long hex ids masked; ignore drops JSONPaths. A failure reads snapshot order: $.total: "<number>" → "<string>". To accept a change on purpose, run with --update-snapshots (or MCP_UPDATE_SNAPSHOTS=1).

Evals: endpoints whose answer varies

A request to an LLM, a RAG or a search endpoint can come back different every time, so one green run proves little. An eval sends one request over several cases, each repeated, and passes a case on its pass rate:

evals:
  smoke:
    requestId: ask
    repeat: 3          # each case is sent 3 times
    passRate: 0.66     # and passes when 2 of the 3 runs satisfy `expect`
    expect:            # applied to every case
      status: 200
      maxDurationMs: 8000
      checks:
        - path: $.citations
          subsetOf: $.sources[*].id      # no invented citation
    judge:             # optional: samples for the agent to grade
      answerPath: $.answer
      contextPath: $.sources[*].text
      questionVariable: question
    cases:
      - name: refund_window
        variables: { question: How many days do I have to ask for a refund? }
        reference: 30 days from delivery.
        expect:
          checks:
            - { path: $.answer, contains: "30" }
            - { path: $.sources[*].id, includesAll: [policy-refunds, faq-returns], minRatio: 0.5 }
      - name: out_of_scope
        variables: { question: Who will win the next election? }
        expect:
          checks:
            - { path: $.answer, matches: "(don't|do not|cannot) (know|answer)", ignoreCase: true }
            - { path: $.citations, maxLength: 0 }

Run it with "run the smoke eval" (run_eval + evalName), or pass requestId and cases inline. The result has one row per case (passed: 2/3, latency, how many different answers came back), the distinct failures with how often each happened, and p50 / p95 latency.

Two kinds of verdict come out of it, and they are kept apart:

  • Checked by code — everything under expect. Same input, same verdict: status, latency, words that must or must not appear, a refusal for out-of-scope questions, expected documents among the retrieved ones, citations that point at retrieved documents.

  • Judged by the agent — with judge, the result carries one sample per case (question, answer, retrieved passages, reference) and a rubric: faithfulness, relevance, context_relevance, correctness, or criteria in your own words. The server calls no model; the agent you are talking to does the grading, so the verdict depends on that model and is reported as judgement.

Things to know:

  • Every run is a real request, usually a paid one. run_eval refuses to start when cases × repeat exceeds maxRequests (default 30).

  • Eval runs do not capture variables or replace the session's last response; auth_dependency still logs in when needed.

  • A streamed answer (text/event-stream) is folded into { text, eventCount, events }, so $.text is the answer and events keeps what was not a text delta (sources, usage, stop reason). Set sse.textPath on the request when the delta sits somewhere unusual.

  • examples/rag.example.yaml is a complete starting point.

Testing the tests

A green suite says the asserts passed, not that they would catch anything. mutate_assertions runs a flow (or one request) once, then breaks each real response in realistic ways and runs the asserts again on every copy: a field disappears, changes type or turns null, a list comes back empty, a value changes, the status becomes 500 or 404, the answer is ten times slower. Nothing extra is sent to the API.

For a request whose only assert is { status: 200, jsonPathExists: $.id }, against a mocked order (excerpt of the real output):

score: 0.17
killed: 5
total: 30
steps[1]{requestId,score,killed,total,note}:
  get_order,0.17,5,30,""
survivors[10]{step,change,addToAssert}:
  get_order,took 1200ms instead of 20ms,"{\"maxDurationMs\":200}"
  get_order,$.status disappeared,"{\"jsonPathExists\":\"$.status\"}"
  get_order,$.items changed type (array → object),"{\"schema\":{\"type\":\"object\",\"required\":[\"items\"],\"properties\":{\"items\":{\"type\":\"array\"}}}}"
  …
moreSurvivors: 15

Each surviving mutation is a regression the suite would let through, with an assert that would catch it. Keep the ones on stable fields; ids and timestamps change between runs. useLastResponse: true scores the last response of the session without sending anything. Snapshots are read, never rewritten, while mutants run.

Contracts: drift, mocks and exports

Drift. check_contract_drift sends the requests of a definition (read-only ones unless includeDestructive) and checks each answer against the operation the OpenAPI / Swagger file documents for it: an undocumented status, or a body that breaks the response schema, is drift. It also lists the operations of the spec that no request covers. Requests are matched by method and path, whether the path holds {{placeholders}} or real ids, with or without the server's base path.

Imports that keep the link. Converting an OpenAPI file turns {petId} into {{petId}}, copies request body examples, and points assert.schema (success response) and body_schema (request body) back into the spec, so every run checks the contract and Vander derives negative probes from it.

CRUD flows. infer_crud_flows looks for POST /things next to GET|PUT|PATCH|DELETE /things/{{id}} and proposes crud_things: create, read, update, read, delete, and a last read that must answer 404 or 410, so a delete that does not delete fails it. It captures the id from the create response ($.id unless you already capture it). apply: true writes the flow, keeping a .bak of the file.

Mock server. npx conversational-api-test-mcp mock [--port 4010] [--delay 200] serves the definitions so a frontend or a CI job can run without the real API. Each endpoint answers with its mock block, the body of a values snapshot, a sample built from its assert.schema, or {}, in that order:

- id: get_pet
  method: GET
  path: /pets/{{pet_id}}
  mock: { status: 200, body: { id: "{{pet_id}}", name: Rex } }   # path params fill the body

Literal paths win over templated ones (/pets/mine before /pets/{{id}}), CORS preflights are answered, and every request is logged with the endpoint and the source of the answer. The mock keeps no state: a POST does not create anything and a DELETE does not remove it, so the last step of a CRUD flow fails against it by design.

Exports. export_contract writes under .mcp/contracts/:

  • openapi: OpenAPI 3.1 from the requests. Placeholders become path parameters, assert.schema and body_schema become $refs, auth becomes a security scheme.

  • arazzo: the flows as Arazzo 1.0 workflows next to that OpenAPI. Captures become step outputs that later steps read ($steps.create_pet.outputs.pet_id); placeholders nobody captured become workflow inputs.

  • pact: a Pact v3 draft with one interaction per endpoint and type matching on response bodies (taken from mock, value snapshots or schema samples). Trim it to what the consumer uses before publishing it to a broker.

Running in CI

The same package runs your flows from a terminal, so the YAML that the agent uses also gates a pipeline:

npx conversational-api-test-mcp run --env STAGING --reporter junit --out reports/api.xml
  • With no file, every definition under .mcp/api/ is read and each of its flows runs (a file with no flows is skipped and listed). --flow smoke picks flows, --request get_user runs single requests, and --all-endpoints runs every request one by one — POST and DELETE included.

  • --env works as set_environment; secrets come from .env.mcp.local or, in CI, from a step that writes it from your secret store.

  • --reporter is pretty (default), json or junit (one testsuite per flow, one testcase per step, read by GitHub Actions, GitLab, Jenkins and Azure DevOps). --bail stops at the first failing flow.

  • --update-snapshots rewrites the snapshots that differ instead of failing. With tracing on, a failing step shows its trace id.

  • Exit code 0 when everything passed, 1 when a step failed, 2 for a usage error or nothing to run.

# .github/workflows/api.yml (excerpt)
- run: printf '%s\n' "$ENV_MCP_LOCAL" > .env.mcp.local
  env: { ENV_MCP_LOCAL: "${{ secrets.ENV_MCP_LOCAL }}" }
- run: npx conversational-api-test-mcp run --env STAGING --reporter junit --out reports/api.xml

Keeping output small

  • responseDetail: "minimal" returns status, duration and captures only; "summary" (default) adds the body, capped at 8,000 chars, and the headers that usually matter (content-type, location, retry-after, rate limits…); "full" returns every header and raises the cap to 50,000.

  • The body comes back once: bodyJson for JSON, bodyPreview for text. Both appear only when JSON exceeds the cap (truncated preview plus topLevelKeys).

  • jsonPathSelect projects the body before it is serialized. On an execute_api_flow step it returns that step's body, so reading it needs no second call.

  • execute_api_flow replaces N tool calls with one.

  • Parsed definitions and .env.mcp.local are cached in memory and invalidated by file mtime.

Security model

The server runs locally with your privileges and is driven by an LLM, so treat API responses and definitions as untrusted input.

  • Redaction. Auth headers, values under credential-looking JSON keys (password, token, secret, api_key, …), JWT-looking strings, credential query parameters and captured credentials are returned as [REDACTED]. The real values stay in the session and are still used by later requests. Redaction is name- and shape-based, so it is a safety net, not a guarantee.

  • No implicit environment access. process.env is not part of the interpolation context; only names you list in MCP_API_ENV_PASSTHROUGH are reachable, via {{env.NAME}}.

  • Responses are data. Values captured from a response are inserted literally and never re-expanded as templates.

  • Writes are confined. Definition tools write only under .mcp/api/ and default to dry-run. init_workspace also creates .env.mcp.local and adds it to .gitignore, and never overwrites an existing file. The import tools append the credentials they lift to .env.mcp.local, never replacing a key that is already there.

  • Optional host allowlist. MCP_API_ALLOWED_HOSTS restricts requests — including every redirect hop — to the listed hostnames. It is off by default and is not a complete SSRF defence (no private-range or DNS-rebinding checks).

Environment variable

Effect

MCP_WORKSPACE_ROOT

Default workspaceRoot when a tool call omits it. Without it, the server reuses the last workspaceRoot it was given, then falls back to its working directory if that already has .mcp/api/.

MCP_API_ALLOWED_HOSTS

Comma-separated hostnames requests may target. Unset = any host.

MCP_API_ENV_PASSTHROUGH

Comma-separated env var names (trailing * = prefix) exposed to {{env.NAME}}. Unset = none.

MCP_API_MAX_RESPONSE_BYTES

Response size cap. Default 10 MB.

MCP_API_REVEAL_SECRETS

true disables redaction. Local debugging only.

Where the ideas come from

Nothing here was invented from scratch. This is what the project borrows, from whom, and what it changes.

Idea in this project

Source

What was taken, and what was changed

The VANDER axes

Stuart Ashman, VADER – a REST API test heuristic (QA Matters, 2016)

The five original axes — Verbs, Authorization, Data, Errors, Responsiveness — are his. This project adds Negative as a sixth and turns part of each axis into executable checks.

The Negative axis

Gwen Diagram & Ash Winter's BINMEN (Boundary, Invalid entries, NULL, Method, Empty, Negative), as listed in the Ministry of Testing Test Heuristics Cheat Sheet

The framing of negative input as its own concern. Missing field, wrong type, null, empty and (with body_schema) boundaries, enums and formats are automated.

Splitting input from output concerns

Amber Race's POISED (Parameters, Output, Interop, Security, Errors, Data), same cheat sheet

Informs the manual ideas under Data and Negative.

Which probes are "destructive"

RFC 9110, HTTP Semantics §9.2.1 (safe methods)

GET, HEAD and OPTIONS are sent freely; every other method needs includeDestructive.

Expected statuses

RFC 9110 §15.5.6 (405), §15.6.2 (501), §15.5.2 (401), §15.5.4 (403)

An undeclared method should get 405 or 501 (404 is also accepted); missing or bad credentials should get 401 or 403.

What the Authorization axis looks for

OWASP API Security Top 10 – 2023, API2 Broken Authentication

Automated: no credentials, invalid token.

Host allowlist, response size cap

OWASP API7 (Server Side Request Forgery) and API4 (Unrestricted Resource Consumption)

Applied to this server itself, since it makes requests on an agent's behalf.

Vander as an MCP prompt

MCP specification, Prompts

Prompts are "user-controlled": the user picks them explicitly, typically as a slash command. That is why the persona is opt-in rather than baked into every conversation.

Persona for tone, code for verdicts

Zheng et al., When "A Helpful Assistant" Is Not Really Helpful (Findings of EMNLP 2024)

They report that adding a persona to the system prompt did not improve accuracy over no persona. So nothing Vander asserts depends on the persona.

Schema-driven negative probes, inferred CRUD flows

Schemathesis and its SBFT '26 paper Schemathesis 4: Multi-Phase Property-Based Testing

Boundary and type probes derived from the schema, and links between operations inferred instead of declared (the paper found 73 explicit OpenAPI links across 4,094 specs). Here the probes are a fixed, explainable list per field, not random generation, and the inferred chain is only create → read → update → delete → read again.

Mutation testing of asserts

Mutation testing (PIT, Stryker) and SWE-Mutation (2026), which found LLM-written test suites often lack the power to tell a broken implementation from a correct one

The code under test is not mutated; the recorded response is. That keeps it offline and fast, and fits a tool whose tests are asserts on responses.

BOLA checks with roles

OWASP API Security Top 10 – 2023, API1 Broken Object Level Authorization

Automated once the definition declares a second identity; still manual for function-level authorization (API5).

Snapshots that ignore volatile data

Snapshot / approval testing (Jest snapshots, ApprovalTests "scrubbers")

Shape mode by default, so a snapshot of live API data does not break every day.

Runtime contract drift

oasdiff compares two versions of a spec; this compares the spec with the live API

Matching by method and path, validation with the same Ajv engine as assert.schema.

HAR import

Record-and-replay tools such as Keploy

Only the recording format is taken: no traffic capture, no replay of mocks, and credentials are never copied.

Exports

Arazzo 1.0, Pact specification v3, OpenAPI 3.1

Pact output is a draft to review, following the advice that agent-written consumer contracts need human review before they gate anything.

Trace ids on requests

W3C Trace Context

Only the traceparent header; no spans are created or exported.

TOON output

TOON specification

Used as-is for tool results. Its authors report the largest savings on uniform arrays and little or none on deeply nested data; this project has not benchmarked it independently.

VANDER is this project's name for the extended heuristic and has no affiliation with the authors above.

Troubleshooting

Symptom

Check

Empty definition list

Is workspaceRoot absolute and correct? Does .mcp/api/ contain .yaml files?

401

set_environment, the endpoint's auth_dependency, captures, .env.mcp.local.

{{KEY}} sent literally

Key name, environment prefix (STAGING_*), or — for shell variables — MCP_API_ENV_PASSTHROUGH.

A value shows as [REDACTED]

Expected for credentials. It is still used in requests.

Development

Command

Description

npm run build

Compile TypeScript to dist/.

npm run dev

Run from source with tsx.

npm test

Run the test suite (vitest).

npm run lint

ESLint.

License

MIT

Available Tools

21 tools
assert_responseA

Assert on the last HTTP response: status, jsonPathExists, maxDurationMs and value checks (contains, regex, equality, ranges, list membership). Reports every check that failed. Run after execute_api_request.

Inputs: checks, status, maxDurationMs, workspaceRoot, +1 more.

convert_legacy_to_canonicalA

Turn a Postman collection, an OpenAPI/Swagger spec or an Insomnia export into a YAML definition under .mcp/api/. Give legacyRelativePath for a file in the workspace, or legacyContent with the JSON/YAML text the user pasted. Literal credentials found in headers or auth are moved to .env.mcp.local and replaced by {{VARIABLE}}. Refuses to replace an existing file unless overwrite is true. Apidog: export as OpenAPI or Postman first.

Inputs: overwrite, serviceName, legacyContent, workspaceRoot, +2 more.

discover_legacy_api_sourcesC

Find Postman collections, OpenAPI/Swagger, or Insomnia exports.

Inputs: maxDepth, maxFiles, workspaceRoot.

dry_run_requestB

Build the URL, method, headers and body preview for a requestId without sending the request. Credentials in headers, query string and body are redacted.

Inputs: requestId (required), definitionRelativePath (required), maxBodyChars, workspaceRoot.

execute_api_flowA

Run a sequence of requestIds in a single MCP round trip. Pass inline steps, or flowName to run a flow declared under flows: in the definition. Each step supports retry, poll (via jsonPath), an inline assert (same shape as assert_response), optional, and jsonPathSelect to get that step's body back ($ for all of it) without a follow-up call. stopOnError halts at the first failing required step.

Inputs: definitionRelativePath (required), steps, flowName, stopOnError, +2 more.

execute_api_requestA

Send one request from a YAML definition. Variables resolve YAML < .env.mcp.local < session (after set_environment, {ENV}_KEY wins over KEY); an unresolved placeholder is sent as the literal {{KEY}}. Runs auth_dependency first when needed, retries once on 401, applies capture and assert. The body comes back once: bodyJson for JSON that fits the cap, bodyPreview for text; JSON over the cap gives a truncated bodyPreview plus bodyJson.topLevelKeys — narrow it with jsonPathSelect. responseDetail: minimal = status only; summary = body (8k chars) and the headers that matter; full = every header, 50k chars.

Inputs: requestId (required), definitionRelativePath (required), timeoutMs, maxBodyChars, +3 more.

explain_request_contextC

Summarize YAML keys, .env.mcp.local keys, session keys, and optional endpoint snippet.

Inputs: definitionRelativePath (required), requestId, workspaceRoot.

get_environment_variableA

Read a session variable set by capture or set_environment_variable. Credential-looking values come back as [REDACTED] with masked: true.

Inputs: name (required), workspaceRoot.

import_curlA

Turn a curl command the user pasted into a request in a YAML definition under .mcp/api/ (creates the file, or appends to it). Method, URL, query string, headers and JSON / form body are carried over. Credentials (Authorization, cookies, -u, credential-looking fields) are moved to .env.mcp.local and replaced by {{VARIABLE}}, so the YAML stays safe to commit. Fails if requestId already exists.

Inputs: curl (required), requestId (required), description, workspaceRoot, +1 more.

init_workspaceA

Create the starting structure in a project: the .mcp/api/ folder, a runnable demo definition (demo.yaml, against httpbin.org) when there is no definition yet, an .env.mcp.local template with commented examples, and the .gitignore line that keeps that file out of git. Never overwrites an existing file. Ask the user before calling it.

Inputs: withDemo, workspaceRoot.

list_api_definitionsB

List YAML API definition files under .mcp/api/. Returns files (paths), or entries (path/service/summary) with includeCatalogMeta or query; plus totalMatched, truncated and, when capped by maxFiles, a nextCursor to pass back as cursor.

Inputs: query, cursor, fields, sortBy, +4 more.

plan_vander_checksA

Build the VANDER checklist (Verbs, Authorization, Negative, Data, Errors, Responsiveness) for one requestId, derived deterministically from its definition. Sends nothing. The id prefix is the axis (V-, A-, …). mode: auto (run_vander_checks executes it), destructive (auto, but state-changing) or manual (an idea to explore by hand).

Inputs: requestId (required), definitionRelativePath (required), axes, workspaceRoot.

read_api_definitionA

Read and validate a whole API YAML definition. Prefer summarize_api_definition for triage.

Inputs: definitionRelativePath (required), workspaceRoot.

reorganize_mcp_api_definitionsA

Plan or apply merging multiple YAML definitions into fewer files. mode plan (no writes) or apply (requires confirm true). groupBy manual_groups (requires manualGroups with targetRelativePath + sourceGlobs relative to .mcp/api/) or same_service_field (auto groups files sharing non-empty service into .mcp/api/_consolidated/*.yaml). Conflicts (duplicate endpoint id across files, mismatched base_url/service/variables) block apply. Optional deleteSourcesAfterMerge or moveSourcesToArchive (mutually exclusive).

Inputs: mode (required), groupBy (required), confirm, manualGroups, +3 more.

run_evalA

Evaluate an endpoint whose answer varies (LLM / RAG / search) — or any endpoint over many inputs — in one call. Runs each case repeat times and passes it when passRate of the runs satisfy expect; returns one row per case, the distinct failures, latency p50/p95 and how many different answers came back. With judge it also returns question / answer / retrieved passages for you to grade what code cannot (faithfulness, relevance). Runs do not capture variables or replace the session's last response. Every run is a real, possibly paid, request.

Inputs: definitionRelativePath (required), cases, judge, expect, +9 more.

run_vander_checksA

Execute the automatic VANDER checks for one requestId. Returns one row per check (id, whose prefix is the axis; result pass/fail/skipped; detail with the evidence) and failures with expected/actual/body snippet. Probes vary method, credentials and body; they never capture variables or replace the session's last response. State-changing probes (any POST/PUT/PATCH/DELETE) are skipped unless includeDestructive is true — ask the user before enabling it. maxDurationMs is the budget for the Responsiveness check.

Inputs: requestId (required), definitionRelativePath (required), axes, timeoutMs, +3 more.

set_environmentA

Set the active logical environment for this workspace session (session CURRENT_ENV, e.g. STAGING). Interpolation resolves {CURRENT_ENV}VAR from .env.mcp.local before generic VAR (prefix-first). Put all env secrets there (DEVELOP, STAGING_); YAML stays generic.

Inputs: environment (required), workspaceRoot.

set_environment_variableC

Set a session variable for subsequent requests.

Inputs: name (required), value (required), workspaceRoot.

summarize_api_definitionA

Summarize a canonical API YAML without returning the full definition. Use names_only for triage (optionally with fields to trim endpoint entries); use full only when needed.

Inputs: definitionRelativePath (required), fields, requestId, detailLevel, +1 more.

summon_vanderA

Call this whenever the user addresses Vander by name ("hi Vander", "oi Vander", "Vander, review X") or asks for a VANDER review. Pass workspaceRoot. Returns the Vander persona brief and, when no endpoint was named, an onboarding: what this server is, the three pieces it needs (.mcp/api/ YAML, .env.mcp.local, the conversation), which of them this workspace already has, and what the user can ask for. Adopt the persona for the rest of the conversation and follow its working method. Sends no HTTP request.

Inputs: requestId, workspaceRoot, definitionRelativePath.

upsert_canonical_api_definitionA

Create or update a canonical YAML under .mcp/api/ only. mergeMode replace_file (requires replaceDefinition) or append_endpoints (requires appendEndpoints). New file on append needs createIfMissing. Default dryRun true returns preview without writing; set dryRun false and confirm true to write (atomic replace + optional .bak backup). Endpoint ids must not collide on append.

Inputs: mergeMode (required), targetRelativePath (required), dryRun, confirm, +4 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 21 tools

    Disambiguation4/5

    Most tools target distinct actions (list vs read vs summarize, plan vs run VANDER checks, single request vs flow vs eval). A few pairs overlap in purpose—read_api_definition vs summarize_api_definition, convert_legacy_to_canonical vs import_curl—but descriptions provide clear guidance on when to use each.

    Naming Consistency4/5

    Nearly all names follow a snake_case verb_noun pattern (execute_api_request, plan_vander_checks, set_environment_variable). Minor deviations such as set_environment vs set_environment_variable and the whimsical summon_vander keep it from being perfectly uniform, but the convention is still predictable.

    Tool Count3/5

    21 tools is heavy for the domain; the server covers definition management, legacy conversion, execution, evals, VANDER checks, environment handling, and workspace init. Several pairs could arguably be merged (read/summarize, plan/run VANDER), making the count borderline rather than well-scoped.

    Completeness4/5

    The surface covers create/read/update via upsert, list, summarize, legacy conversion, request execution, assertions, flows, evals, VANDER checks, environment variables, and workspace init. A notable gap is an explicit delete tool for definitions or endpoints; upsert with replace_file can overwrite but does not obviously delete.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      A deterministic eval engine for coding agents. Your agent claims it fixed the bug—this checks it. The same scenario runs twice against a sandbox of the services your code calls, with failures injected on purpose. It must fail on the old code and pass on the new. The verdict is an exit code, not a model's opinion. Every run leaves a receipt.
      19
      442 npm
      1
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to inspect, test, validate, compare, and debug HTTP APIs through a structured toolkit, with features such as secret redaction, response contract validation, regression comparison, health checks, and incident investigation workflows.
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to call real API endpoints and verify responses against OpenAPI specs, supporting contract testing, auth presets, spec diffing, and health checks.
      8
      54 npm
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agent-planned, human-reviewed HTTP API testing through MCP, turning OpenAPI/Swagger definitions and requirements into executable Hurl/k6 test plans. Supports plan review, execution, cleanup, and reports for coding agents such as Codex and Claude Code.
      MIT