mcp-crawler
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
MCP Service
This is a Model Control Protocol (MCP) service that provides tools for Cursor and Claude applications.
Connecting to the Service
For Cursor Users
Open Cursor
Go to Settings > AI > Advanced
Under "MCP Endpoint", enter:
https://your-render-url.onrender.comSave settings and restart Cursor
For Claude App Users
Open Claude App
Go to Settings > Advanced
Under "MCP Connection", enter:
https://your-render-url.onrender.comSave settings
Related MCP server: Sitemap MCP Server
Available Tools
Web Crawler: Crawls web pages and returns content as markdown
Deployment
This service is deployed on Render.com as a Docker container. The service is accessible at:
https://your-render-url.onrender.comReplace your-render-url.onrender.com with the actual URL provided by Render after deployment.
Environment Variables
MCP_HOST: Host to bind the server (default: 0.0.0.0)MCP_PORT: Port to bind the server (default: 8000)
Monitoring
A health endpoint is available at /health for monitoring the service status.
This server cannot be deployed
Maintenance
Related MCP Connectors
Crawls any documentation or SaaS site and compiles a standards-compliant /llms.txt manifest and J.
Crawl a site for broken links, 404s, dead images, redirect chains and slow pages, with sources
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.613MIT
- AlicenseBqualityDmaintenanceMCP server for fetching, parsing and crawling sitemaps of a given website440 PyPI7MIT
- AlicenseNot gradedqualityCmaintenanceEnables web crawling and content extraction from web pages, supporting multiple output formats like text, markdown, XML, and JSON, with robots.txt compliance and rate limiting.14 npm1MIT
- AlicenseNot gradedqualityAmaintenanceClone, scrape and crawl any website. Free Firecrawl alternative with no API keys or rate limits.17 npmMIT