mcp-crawler
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
MCP Service
This is a Model Control Protocol (MCP) service that provides tools for Cursor and Claude applications.
Connecting to the Service
For Cursor Users
Open Cursor
Go to Settings > AI > Advanced
Under "MCP Endpoint", enter:
https://your-render-url.onrender.comSave settings and restart Cursor
For Claude App Users
Open Claude App
Go to Settings > Advanced
Under "MCP Connection", enter:
https://your-render-url.onrender.comSave settings
Related MCP server: Open Crawler MCP Server
Available Tools
Web Crawler: Crawls web pages and returns content as markdown
Deployment
This service is deployed on Render.com as a Docker container. The service is accessible at:
https://your-render-url.onrender.comReplace your-render-url.onrender.com with the actual URL provided by Render after deployment.
Environment Variables
MCP_HOST: Host to bind the server (default: 0.0.0.0)MCP_PORT: Port to bind the server (default: 8000)
Monitoring
A health endpoint is available at /health for monitoring the service status.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Crawl a site for SEO issues per page and broken links. Respects robots.txt.
Crawls any documentation or SaaS site and compiles a standards-compliant /llms.txt manifest and J.
Crawl a site for broken links, 404s, dead images, redirect chains and slow pages, with sources
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.613MIT
- AlicenseNot gradedqualityCmaintenanceEnables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.14 npm1MIT
- AlicenseNot gradedqualityBmaintenanceClone, scrape and crawl any website. Free Firecrawl alternative with no API keys or rate limits.16 npmMIT

MeshArcofficial
AlicenseNot gradedqualityBmaintenanceCrawl and scrape websites to clean markdown, map their sitemaps, and track what changed between crawls. Use the hosted server or run it locally.905 PyPIMIT