# crawlcheck-mcp-server

MCP server for [crawlcheck](https://github.com/jabeer4148-ops/crawlcheck). Crawls a website in a real
Chromium browser and reports JavaScript console errors, uncaught exceptions, failed requests, broken
links, and accessibility violations — classifying console errors and failed requests as **first-party**
(your code) or **third-party** (embeds and scripts you don't control).

Runs locally over stdio, so crawls use your machine's browser and nobody pays for hosted compute.

## Prerequisite

```bash
npx playwright install chromium
```

Without it the tools return: `Chromium is not installed for Playwright. Run: npx playwright install chromium`.

## Configuration

**Claude Desktop / Claude Code** — `.mcp.json`:

```json
{
  "mcpServers": {
    "crawlcheck": {
      "command": "npx",
      "args": ["-y", "crawlcheck-mcp-server"]
    }
  }
}
```

**Cursor** — same shape in `mcp.json`.

## Tools

| Tool | Purpose |
| --- | --- |
| `crawlcheck_scan_site` | Crawl a site. Returns a summary, the top 20 issues, and a `scan_id`. Never returns the full issue list. |
| `crawlcheck_check_page` | Check a single page without following links — the fast path for "is this page broken?". |
| `crawlcheck_get_issues` | Filtered, paginated issues from a completed scan. Reads from cache; never re-crawls, never blocks. |

Typical flow: `scan_site` for the summary, then `get_issues` to drill in without re-crawling or
flooding the context window.

### Two behaviours worth knowing

**Classification is not universal.** Only `console-error` and `failed-request` carry a
`classification`, because those are the only issue types for which the engine computes a resource
origin. `page-error`, `broken-link` and `a11y` have no classification field at all — they are
reported as `unclassified`, which does **not** mean third-party. `crawlcheck_get_issues` therefore
defaults to `classification: "all"`; filtering to `"first-party"` would silently hide every broken
link and accessibility violation.

**One crawl at a time.** A second concurrent scan is rejected immediately rather than queued:

> A crawl is already running. crawlcheck runs one crawl at a time — wait for the current scan to
> finish, then retry. If you have a scan_id from an earlier scan, use crawlcheck_get_issues instead;
> it reads from cache and never blocks.

This is a correctness requirement, not throttling. `runCrawl` in crawlcheck 0.2.x redirects the
global `console.log` while crawling (so that progress output can't corrupt this server's JSON-RPC
frames on stdout), and overlapping calls would interleave that redirect. Rejecting beats queueing
because MCP clients time out tool calls anyway, so a queued crawl usually dies waiting.

## Exit semantics

`summary.exitCode` mirrors crawlcheck's process exit code and is only ever `0` or `1`. Accessibility
violations are advisory and do not affect it unless `strict: true`. Third-party console errors and
failed requests are excluded unless `include_third_party: true`.

## Development

```bash
npm install        # crawlcheck resolves to ../crawlcheck until 0.2.0 is on npm
npm run build      # generates src/types/report.v1.d.ts, then compiles
npm test           # builds, then runs the unit tests
npm run check:types  # regenerate types and fail if they drift from the schema
npm run inspect    # MCP Inspector against the built server
```

`src/types/report.v1.d.ts` is **generated** from crawlcheck's published JSON Schema and committed.
Don't hand-edit it — `npm run check:types` regenerates and fails on any diff, so a crawlcheck schema
bump can't slip through unnoticed. The only hand-written declaration is `runCrawl`'s signature in
`src/types/crawlcheck.d.ts`, which deliberately omits crawlcheck's internal `raw` field so that
reaching for it is a compile error.

### Verifying

Unit tests cover ranking, digest limits, cache TTL/LRU, single-flight (including that a *failed*
crawl releases the lock), and error mapping. What they can't cover is a real crawl — for that:

```bash
npm run build
npx @modelcontextprotocol/inspector node dist/index.js
```

In Inspector v2.x the server appears in a **Servers** list as `Disconnected` — flip the toggle beside
it to start the process before the tools show up.

Concurrency is the one behaviour the Inspector can't exercise, because it sends a single tool call at
a time. For that:

```bash
node test/live-concurrency.mjs [url]
```

It drives the built server over stdio and fires a second `crawlcheck_scan_site` while the first is
still crawling. Expected: the second is rejected with "A crawl is already running", the first
completes, and a third succeeds afterwards — proving the lock releases rather than wedging the
server. Exits non-zero on failure. Excluded from `npm test` because it launches a browser and hits
the network.

## License

MIT