mcp-crawler

Integrations

  • Converts crawled web content into markdown format for easier consumption by AI applications

  • Hosts the MCP service as a Docker container, making it accessible via a Render.com URL

MCP Service

This is a Model Control Protocol (MCP) service that provides tools for Cursor and Claude applications.

Connecting to the Service

For Cursor Users

  1. Open Cursor
  2. Go to Settings > AI > Advanced
  3. Under "MCP Endpoint", enter: https://your-render-url.onrender.com
  4. Save settings and restart Cursor

For Claude App Users

  1. Open Claude App
  2. Go to Settings > Advanced
  3. Under "MCP Connection", enter: https://your-render-url.onrender.com
  4. Save settings

Available Tools

  • Web Crawler: Crawls web pages and returns content as markdown

Deployment

This service is deployed on Render.com as a Docker container. The service is accessible at:

https://your-render-url.onrender.com

Replace your-render-url.onrender.com with the actual URL provided by Render after deployment.

Environment Variables

  • MCP_HOST: Host to bind the server (default: 0.0.0.0)
  • MCP_PORT: Port to bind the server (default: 8000)

Monitoring

A health endpoint is available at /health for monitoring the service status.

-
security - not tested
F
license - not found
-
quality - not tested

It crawls website

  1. Connecting to the Service
    1. For Cursor Users
    2. For Claude App Users
  2. Available Tools
    1. Deployment
      1. Environment Variables
        1. Monitoring

          Related MCP Servers

          • A
            security
            A
            license
            A
            quality
            Enables web content scanning and analysis by fetching, analyzing, and extracting information from web pages using tools like page fetching, link extraction, site crawling, and more.
            Last updated -
            6
            7
            TypeScript
            MIT License
          • A
            security
            A
            license
            A
            quality
            This server facilitates interaction with cosense/Scrapbox projects, enabling users to retrieve, list, search, and create pages while supporting various query operations and secure access to private projects.
            Last updated -
            4
            12
            TypeScript
            MIT License
            • Apple
          • A
            security
            A
            license
            A
            quality
            A server that provides web scraping and intelligent content searching capabilities using the Firecrawl API, enabling AI agents to extract structured data from websites and perform content searches.
            Last updated -
            5
            2
            TypeScript
            MIT License
            • Apple
            • Linux
          • -
            security
            F
            license
            -
            quality
            Bridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.
            Last updated -
            Python
            • Apple

          View all related MCP servers

          ID: wtt5eqpn0n