# collection-mcp-check

Turn a Postman collection into an MCP server, then check whether an AI agent can actually use it.

```
cmcp lint   collection.json          # agent-readiness findings, CI exit code
cmcp tools  collection.json          # the MCP tools/list payload it would expose
cmcp serve  collection.json          # run it as an MCP server on stdio
cmcp check  collection.json --call-readonly   # spawn the server, speak MCP to it, call the safe tools
```

Python 3.10+, standard library only. No runtime dependencies.

## Why this exists

Postman already turns collections into MCP servers (the MCP Generator on the API Network gives you one tool per request). The gap is the step after that: a collection written for humans clicking "Send" makes a shaky tool list for a model.

I ran it on `echo-v2.postman_collection.json` from postmanlabs/newman (the Postman Echo collection, 22 requests). What it found:

- **`Set Cookies` and `Delete Cookies` are GETs.** A generator that marks every GET `readOnlyHint: true` tells the client these are safe to call without asking. `cmcp` reads the name and path, flags it (CM040), and sets `readOnlyHint: false`.
- **`DigestAuth Success` needs `{{echo_digest_realm}}` and `{{echo_digest_nonce}}`**, which a test script in a *different* request writes with `postman.setGlobalVariable`. In Postman you run the folder in order. An agent calls one tool. CM023 catches that chain.
- **Digest and OAuth 1.0 auth** aren't replayed by the generated server (CM060), so those calls go out unsigned.
- **8 GET requests carry a `formdata` body** (including a `client_secret` field) that the tool silently drops (CM022, CM030).

Then `cmcp check --call-readonly` called the 15 read-only tools against the live postman-echo.com. 12 returned 200. Of the 3 failures, 2 are exactly what lint predicted (missing chained variables, unsupported OAuth 1.0). The third is the first half of the digest flow, which gets a 401 on purpose and stores the nonce from it.

That's the point of shipping lint and live check together: the lint tells you which calls will fail before an agent finds out the hard way.

## Quickstart

```bash
git clone https://github.com/DeveshB-1/collection-mcp-check && cd collection-mcp-check
uv run cmcp lint examples/postman-echo-v2.postman_collection.json
uv run cmcp check examples/postman-echo-v2.postman_collection.json --call-readonly

# your own collection + environment export
uv run cmcp lint my.postman_collection.json --env my.postman_environment.json --strict
```

Use it from Claude Desktop (or any MCP client):

```json
{
  "mcpServers": {
    "orders-api": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/collection-mcp-check",
               "cmcp", "serve", "/path/to/orders.postman_collection.json",
               "--env", "/path/to/prod.postman_environment.json"]
    }
  }
}
```

Variables resolve in this order: collection variables, then the `--env` file (a Postman environment export or a flat JSON object), then `CMCP_VAR_<name>` process env vars. Secrets stay out of the collection file.

## What the lint checks

| Rule | Severity | Catches |
|------|----------|---------|
| CM001 | warn | two requests slug to the same tool name (renamed `_2`, and agents mix them up) |
| CM002 | warn | name over 64 chars, truncated |
| CM003 | error | name fails `^[a-zA-Z0-9_-]{1,64}$` (the rule the Anthropic and OpenAI tool APIs enforce) |
| CM010 | warn | request has no description, so the model only sees `GET {{baseUrl}}/...` |
| CM011 | info | description over 1,500 chars (sent on every `tools/list`) |
| CM012 | warn | path or query parameter with no description |
| CM020 | error | `{{variable}}` defined nowhere (collection or env) |
| CM021 | error | body declared JSON that doesn't parse |
| CM022 | warn | GET/HEAD with a body (dropped from the tool) |
| CM023 | warn | depends on a variable another request's script sets |
| CM030 | warn | literal credential in a header, query, form field or auth block |
| CM040 | warn | GET whose name/path reads as a write; `readOnlyHint` set to false |
| CM050 | error | request with no URL |
| CM060 | warn | auth type the server won't replay (digest, oauth1, hawk, ...) |

`agent-readiness` = share of tools with zero errors or warnings. `lint` exits 1 on errors (`--strict`: on warnings too), so it drops into CI next to a Newman run.

## What the server does

- One tool per request. Name from `folder_request` in snake_case, deduped, kept to 64 chars.
- Input schema: path variables (required, with saved examples), query params (defaults from the collection), JSON body as an object with the saved example as default, form fields as an object.
- Annotations: `readOnlyHint` for GET/HEAD/OPTIONS that don't read as writes, `destructiveHint` for DELETE/PUT/PATCH and cancel/remove/reset-style names, `openWorldHint` always.
- Auth inherited collection → folder → request, as Postman does. Basic, bearer and API-key (header) are replayed.
- Postman dynamic variables `{{$guid}}`, `{{$timestamp}}`, `{{$isoTimestamp}}`, `{{$randomInt}}`, `{{$randomUUID}}` are filled at call time.
- HTTP 4xx/5xx come back as `isError: true` with the status, latency and the first 4,000 chars of the body.
- MCP over stdio, protocol versions 2024-11-05 through 2025-06-18: `initialize`, `ping`, `tools/list`, `tools/call`.

It sets its own `User-Agent`, because postman-echo.com returns 403 to urllib's default `Python-urllib/3.x` (that's how I found out).

## Sample run

Live run against postman-echo.com, 2026-10-25 (full output in `demo/`):

```
$ cmcp lint examples/postman-echo-v2.postman_collection.json
collection: Postman Echo (V2)
tools: 22  clean: 9  agent-readiness: 41%
errors: 0  warnings: 26  info: 0

warn  CM023 auth_digest_digestauth_success: depends on variables another request's script sets ({{echo_digest_nonce}}, {{echo_digest_realm}}); an agent calling this tool on its own won't have them.
warn  CM040 cookies_delete_cookies: GET whose name/path reads as a write ('Delete Cookies'); readOnlyHint set to false so clients ask before calling it.
warn  CM040 cookies_set_cookies: GET whose name/path reads as a write ('Set Cookies'); readOnlyHint set to false so clients ask before calling it.
warn  CM060 auth_digest_digestauth_success: auth type 'digest' is not replayed by the generated server; calls will go out unauthenticated.
warn  CM060 auth_others_oauth1_0_verify_signature: auth type 'oauth1' is not replayed by the generated server; calls will go out unauthenticated.
... (21 more in demo/echo-lint.txt)

$ cmcp check examples/postman-echo-v2.postman_collection.json --call-readonly
protocol: 2025-06-18  tools listed: 22  schema problems: 0
read-only calls:
  FAIL auth_digest_digestauth_request: HTTP 401 (328 ms) GET https://postman-echo.com/digest-auth
  FAIL auth_digest_digestauth_success: Variable '{{echo_digest_realm}}' is not set. Pass --env <postman_environment.json> or export CMCP_VAR_echo_digest_realm=<value>.
  ok   auth_others_basic_auth: HTTP 200 (271 ms) GET https://postman-echo.com/basic-auth
  FAIL auth_others_oauth1_0_verify_signature: HTTP 401 (274 ms) GET https://postman-echo.com/oauth1
  ok   cookies_get_cookies: HTTP 200 (291 ms) GET https://postman-echo.com/cookies
  ok   headers_request_headers: HTTP 200 (261 ms) GET https://postman-echo.com/headers
  ok   headers_response_headers: HTTP 200 (268 ms) GET https://postman-echo.com/response-headers?Content-Type=text%2Fhtml&Server=apibin
  ok   request_methods_get_request: HTTP 200 (244 ms) GET https://postman-echo.com/get?test=123
  ok   utilities_response_status_code: HTTP 200 (255 ms) GET https://postman-echo.com/status/200
  ok   utilities_streamed_response: HTTP 200 (296 ms) GET https://postman-echo.com/stream/10
  ok   utilities_delay_response: HTTP 200 (3275 ms) GET https://postman-echo.com/delay/3
  ok   utilities_get_utf8_encoded_response: HTTP 200 (280 ms) GET https://postman-echo.com/encoding/utf8
  ok   utilities_gzip_compressed_response: HTTP 200 (263 ms) GET https://postman-echo.com/gzip
  ok   utilities_deflate_compressed_response: HTTP 200 (264 ms) GET https://postman-echo.com/deflate
  ok   draft_auth_oauth2_0_oauth2_0_get_resource: HTTP 200 (275 ms) GET https://postman-echo.com/oauth2/user/info
result: FAIL
```

Write tools (POST/PUT/PATCH/DELETE, and the two cookie GETs) are never called by `check`.

## Tests

```
$ uv run --with pytest --with mcp python -m pytest -q
................                                                         [100%]
16 passed in 3.68s
```

The suite runs offline against a local HTTP mock: parsing (v2.0 and v2.1, auth inheritance, string URLs), naming, schemas, every lint rule on `examples/orders-api.postman_collection.json` (a fixture built to trip them), the JSON-RPC loop, `check` spawning the server as a subprocess, and one interop test that drives the server with the official MCP Python SDK client (`mcp` 2.3.0; skipped when it isn't installed).

## Limits

- Pre-request and test scripts are read only to find which variables they set. They aren't executed.
- No digest, OAuth 1.0, Hawk, AWS SigV4 or NTLM signing (lint warns instead).
- `formdata` file fields are sent as plain text fields.
- Mutating-GET detection is a word list (`delete`, `set`, `cancel`, ...). It will miss some.

## Files

```
cmcp/collection.py  parse v2.0/v2.1 collections and environments
cmcp/tools.py       requests -> MCP tool definitions
cmcp/lint.py        agent-readiness rules
cmcp/runtime.py     variable resolution + HTTP call
cmcp/server.py      MCP stdio server (JSON-RPC 2.0)
cmcp/check.py       stdio client that validates a running server
examples/           Postman Echo collection (from postmanlabs/newman, Apache-2.0) + a fixture
```

`examples/postman-echo-v2.postman_collection.json` is copied unchanged from [postmanlabs/newman](https://github.com/postmanlabs/newman/blob/develop/test/integration/echo-v2.postman_collection.json) (Apache-2.0). Everything else is MIT.