Integrations
mcp-server-webcrawl
Bridge the gap between your web crawl and AI language models using Model Context Protocol (MCP). With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously. The server includes a full-text search interface with boolean support, resource filtering by type, HTTP status, and more.
mcp-server-webcrawl provides the LLM a complete menu with which to search your web content, and works with a variety of web crawlers:
mcp-server-webcrawl is free and open source, and requires Claude Desktop, Python (>=3.10). It is installed on the command line, via pip install:
Features
- Claude Desktop ready
- Fulltext search support
- Filter by type, status, and more
- Multi-crawler compatible
- Quick MCP configuration
- ChatGPT support coming soon
MCP Configuration
From the Claude Desktop menu, navigate to File > Settings > Developer. Click Edit Config to locate the configuration file, open in the editor of your choice and modify the example to reflect your datasrc path.
You can set up more mcp-server-webcrawl connections under mcpServers as needed.
Important Note for macOS Users
On macOS, you must use the absolute path to the mcp-server-webcrawl
executable in the command
field, rather than just the command name. This is different from Windows configuration.
For example:
To find the absolute path of the mcp-server-webcrawl
executable on your system:
- Open Terminal
- Run
which mcp-server-webcrawl
- Copy the full path returned and use it in your config file
wget (using --mirror)
The datasrc argument should be set to the parent directory of the mirrors.
WARC
The datasrc argument should be set to the parent directory of the WARC files.
InterroBot
The datasrc argument should be set to the direct path to the database.
Katana
The datasrc argument should be set to the parent directory of the text cache files.
SiteOne (using archiving)
The datasrc argument should be set to the parent directory of the archives, archiving must be enabled.
This server cannot be installed
local-only server
The server can only run on the client's local machine because it depends on local resources.
Bridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content.
Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.
Related MCP Servers
- -securityAlicense-qualityCrawl4AI MCP Server is an intelligent information retrieval server offering robust search capabilities and LLM-optimized web content understanding, utilizing multi-engine search and intelligent content extraction to efficiently gather and comprehend internet information.Last updated -36PythonMIT License
- AsecurityAlicenseAqualityA headless browser MCP server that allows AI agents to fetch web content and perform Google searches without API keys, supporting various output formats like Markdown, JSON, HTML, and text.Last updated -24TypeScriptMIT License
- -securityAlicense-qualityAn MCP server that helps AI assistants access text content from websites that implement bot detection, bridging the gap between what you can see in your browser and what the AI can access.Last updated -1PythonApache 2.0
- -securityFlicense-qualityAn MCP server that crawls API documentation websites and exposes their content to AI models, enabling them to search, browse, and reference API specifications.Last updated -Python