mcp-crawler
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@mcp-crawlercrawl the latest blog posts from techcrunch.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP Service
This is a Model Control Protocol (MCP) service that provides tools for Cursor and Claude applications.
Connecting to the Service
For Cursor Users
Open Cursor
Go to Settings > AI > Advanced
Under "MCP Endpoint", enter:
https://your-render-url.onrender.comSave settings and restart Cursor
For Claude App Users
Open Claude App
Go to Settings > Advanced
Under "MCP Connection", enter:
https://your-render-url.onrender.comSave settings
Related MCP server: crawl4ai-mcp
Available Tools
Web Crawler: Crawls web pages and returns content as markdown
Deployment
This service is deployed on Render.com as a Docker container. The service is accessible at:
https://your-render-url.onrender.comReplace your-render-url.onrender.com with the actual URL provided by Render after deployment.
Environment Variables
MCP_HOST: Host to bind the server (default: 0.0.0.0)MCP_PORT: Port to bind the server (default: 8000)
Monitoring
A health endpoint is available at /health for monitoring the service status.
This server cannot be deployed
Maintenance
Related MCP Connectors
Crawl a site for broken links, 404s, dead images, redirect chains and slow pages, with sources
- fastCRWOAuthio.github.us
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Extract data from any website with this web scraper tool.
Scrape and crawl websites into queryable tables: products, leads, Shopify catalogs, real estate.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.613MIT
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- AlicenseBqualityDmaintenanceMCP server for fetching, parsing and crawling sitemaps of a given website47MIT
- FlicenseNot gradedqualityNot gradedmaintenanceEnables querying and retrieving content from webpages by parsing sitemap.xml files and fetching HTML content from specified URLs. Includes built-in rate limiting for protection against abuse.-