web-crawler-mcp
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
web-crawler-mcp
A minimal crawl4ai crawler exposed as an MCP server, deployable to Cloud Run.
What it does
Exposes a single MCP tool, crawl(url, max_length), which fetches a page with crawl4ai's headless-browser crawler and returns its content as markdown.
Related MCP server: crawl-mcp-server
Local development
pip install -r requirements.txt
playwright install --with-deps chromium
crawl4ai-setup
python src/server.pyThe server listens on PORT (default 8080) using the MCP Streamable HTTP transport, reachable at http://localhost:8080/mcp.
Deployment
.github/workflows/deploy.yml builds the Docker image with Cloud Build and deploys it to Cloud Run on every push to main that touches src/, Dockerfile, or requirements.txt.
Required GitHub Actions secrets:
GCP_SA_KEY— service account key JSON used to authenticate to Google Cloud.GCP_PROJECT_ID— the target GCP project ID.
The Cloud Run service (web-crawler-mcp, region us-central1) is deployed with --allow-unauthenticated, matching the reference deployment pattern. Restrict access with IAM invoker bindings or a load balancer in front if the endpoint needs to be private.
Connecting an MCP client
Point an MCP client (Streamable HTTP transport) at:
https://<cloud-run-service-url>/mcpTool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Unified 15-tool SEO, SERP, clean Markdown & GEO (LLM citation Share-of-Voice) MCP server.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- FlicenseNot gradedqualityNot gradedmaintenanceAn MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.48 npm-
- AlicenseAqualityCmaintenanceMCP server for web scraping — extract clean markdown, links, and metadata from any URL. Free Firecrawl alternative.5102 npm6MIT
- AlicenseNot gradedqualityDmaintenanceA lightweight MCP server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents, enabling single-page scraping and multi-page crawling with adaptive stopping.118MIT