# MCP Documentation Crawler

A local, configurable [Model Context Protocol](https://modelcontextprotocol.io/) server that crawls a documentation site, stores a local index, and gives Claude (or another MCP client) search and page-retrieval tools.

The default configuration targets GE Vernova Proficy Plant Applications 2025 documentation. It only follows links within the configured domain and path prefix.

## Install and crawl

```bash
git clone https://github.com/codecracker2020/mcp-server-crawler.git
cd mcp-server-crawler
npm install
cp config.example.json config.json
npm run crawl -- --config=config.json
npm run search -- --config=config.json "production unit"
```

The generated local index is `data/index.json` (ignored by Git). Re-run `npm run crawl` to refresh it.

## HTTP API for other applications

Start the API after crawling:

```bash
npm run serve -- --config=config.json
```

It listens on `http://127.0.0.1:3100` by default and exposes:

```bash
curl "http://127.0.0.1:3100/health"
curl "http://127.0.0.1:3100/search?q=releasing%20a%20route&limit=5"
curl "http://127.0.0.1:3100/search?q=releasing%20a%20route&summarize=true"
```

`GET /search` returns JSON with the query, index timestamp, and matching `url`, `title`, `excerpt`, and `score` fields. Add `summarize=true` to include a Claude summary grounded only in those matching excerpts. This uses AWS Bedrock's normal credential chain (for example `AWS_PROFILE`, access keys, or an IAM role) and the configured `bedrockRegion` and `bedrockModelId`; it makes a billed Bedrock request. Use `apiHost`, `apiPort`, and `apiCorsOrigin` to configure the server. Keep the default loopback host unless another machine truly needs access; if you expose it on a network, set `MCP_CRAWLER_API_KEY` and send it as the `X-API-Key` request header.

## Claude Desktop / Claude Code / VS Code configuration

Add this server configuration to the MCP settings used by your client, adjusting the absolute paths:

```json
{
  "mcpServers": {
    "plant-applications-docs": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-server-crawler/src/server.js", "--config=/absolute/path/to/mcp-server-crawler/config.json"]
    }
  }
}
```

For Claude Code, the equivalent command is:

```bash
claude mcp add --transport stdio plant-applications-docs -- node /absolute/path/to/mcp-server-crawler/src/server.js --config=/absolute/path/to/mcp-server-crawler/config.json
```

After restarting or reloading the client, it exposes four tools: `crawl_documentation`, `search_documentation`, `get_documentation_page`, and `crawler_status`.

## Configuration

Copy `config.example.json` to `config.json` and change any field. `startUrl`, `allowedDomains`, and `urlPathPrefix` form the crawl boundary. The server also accepts every major setting as an environment variable, which is useful for VS Code settings or CI:

| JSON field | Environment variable |
| --- | --- |
| `startUrl` | `MCP_CRAWLER_START_URL` |
| `allowedDomains` | `MCP_CRAWLER_ALLOWED_DOMAINS` (comma-separated) |
| `urlPathPrefix` | `MCP_CRAWLER_URL_PATH_PREFIX` |
| `maxPages` | `MCP_CRAWLER_MAX_PAGES` |
| `concurrency` | `MCP_CRAWLER_CONCURRENCY` |
| `requestDelayMs` | `MCP_CRAWLER_REQUEST_DELAY_MS` |
| `maxContentChars` | `MCP_CRAWLER_MAX_CONTENT_CHARS` |
| `dataDirectory` | `MCP_CRAWLER_DATA_DIR` |
| `apiHost` | `MCP_CRAWLER_API_HOST` |
| `apiPort` | `MCP_CRAWLER_API_PORT` |
| `apiCorsOrigin` | `MCP_CRAWLER_API_CORS_ORIGIN` |
| `apiKey` | `MCP_CRAWLER_API_KEY` |
| `bedrockRegion` | `MCP_CRAWLER_BEDROCK_REGION` (or `AWS_REGION`) |
| `bedrockModelId` | `MCP_CRAWLER_BEDROCK_MODEL_ID` |
| `bedrockMaxTokens` | `MCP_CRAWLER_BEDROCK_MAX_TOKENS` |

Use the configuration conservatively and comply with the documentation site's terms and robots policy. The crawler deliberately ignores non-HTML pages and removes navigation, scripts, styles, and other page chrome before indexing.