MCP Documentation Crawler

A local, configurable Model Context Protocol server that crawls a documentation site, stores a local index, and gives Claude (or another MCP client) search and page-retrieval tools.

The default configuration targets GE Vernova Proficy Plant Applications 2025 documentation. It only follows links within the configured domain and path prefix.

Install and crawl

git clone https://github.com/codecracker2020/mcp-server-crawler.git
cd mcp-server-crawler
npm install
cp config.example.json config.json
npm run crawl -- --config=config.json
npm run search -- --config=config.json "production unit"

The generated local index is data/index.json (ignored by Git). Re-run npm run crawl to refresh it.

Related MCP server: rock-mcp

HTTP API for other applications

Start the API after crawling:

npm run serve -- --config=config.json

It listens on http://127.0.0.1:3100 by default and exposes:

curl "http://127.0.0.1:3100/health"
curl "http://127.0.0.1:3100/search?q=releasing%20a%20route&limit=5"
curl "http://127.0.0.1:3100/search?q=releasing%20a%20route&summarize=true"

GET /search returns JSON with the query, index timestamp, and matching url, title, excerpt, and score fields. Add summarize=true to include a Claude summary grounded only in those matching excerpts. This uses AWS Bedrock's normal credential chain (for example AWS_PROFILE, access keys, or an IAM role) and the configured bedrockRegion and bedrockModelId; it makes a billed Bedrock request. Use apiHost, apiPort, and apiCorsOrigin to configure the server. Keep the default loopback host unless another machine truly needs access; if you expose it on a network, set MCP_CRAWLER_API_KEY and send it as the X-API-Key request header.

Claude Desktop / Claude Code / VS Code configuration

Add this server configuration to the MCP settings used by your client, adjusting the absolute paths:

{
  "mcpServers": {
    "plant-applications-docs": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-server-crawler/src/server.js", "--config=/absolute/path/to/mcp-server-crawler/config.json"]
    }
  }
}

For Claude Code, the equivalent command is:

claude mcp add --transport stdio plant-applications-docs -- node /absolute/path/to/mcp-server-crawler/src/server.js --config=/absolute/path/to/mcp-server-crawler/config.json

After restarting or reloading the client, it exposes four tools: crawl_documentation, search_documentation, get_documentation_page, and crawler_status.

Configuration

Copy config.example.json to config.json and change any field. startUrl, allowedDomains, and urlPathPrefix form the crawl boundary. The server also accepts every major setting as an environment variable, which is useful for VS Code settings or CI:

Use the configuration conservatively and comply with the documentation site's terms and robots policy. The crawler deliberately ignores non-HTML pages and removes navigation, scripts, styles, and other page chrome before indexing.