MCP Documentation Crawler

A local, configurable Model Context Protocol server that crawls a documentation site, stores a local index, and gives Claude (or another MCP client) search and page-retrieval tools.

The default configuration targets GE Vernova Proficy Plant Applications 2025 documentation. It only follows links within the configured domain and path prefix.

Install and crawl

git clone https://github.com/codecracker2020/mcp-server-crawler.git
cd mcp-server-crawler
npm install
cp config.example.json config.json
npm run crawl -- --config=config.json
npm run search -- --config=config.json "production unit"

The generated local index is data/index.json (ignored by Git). Re-run npm run crawl to refresh it.

Related MCP server: rock-mcp

Claude Desktop / Claude Code / VS Code configuration

Add this server configuration to the MCP settings used by your client, adjusting the absolute paths:

{
  "mcpServers": {
    "plant-applications-docs": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-server-crawler/src/server.js", "--config=/absolute/path/to/mcp-server-crawler/config.json"]
    }
  }
}

For Claude Code, the equivalent command is:

claude mcp add --transport stdio plant-applications-docs -- node /absolute/path/to/mcp-server-crawler/src/server.js --config=/absolute/path/to/mcp-server-crawler/config.json

After restarting or reloading the client, it exposes four tools: crawl_documentation, search_documentation, get_documentation_page, and crawler_status.

Configuration

Copy config.example.json to config.json and change any field. startUrl, allowedDomains, and urlPathPrefix form the crawl boundary. The server also accepts every major setting as an environment variable, which is useful for VS Code settings or CI:

Use the configuration conservatively and comply with the documentation site's terms and robots policy. The crawler deliberately ignores non-HTML pages and removes navigation, scripts, styles, and other page chrome before indexing.