deepcrawl-mcp

Clone, scrape and crawl any website. Free Firecrawl alternative. No API keys, no rate limits, no subscription.

Quick Start

Cursor (one-click): Click the "Install in Cursor" button above.

Manual MCP config:

{
  "mcpServers": {
    "deepcrawl-mcp": {
      "command": "npx",
      "args": ["-y", "deepcrawl-mcp@latest"]
    }
  }
}

CLI:

npx deepcrawl-mcp@latest

Related MCP server: mcp-server-scraper

Tools

deepcrawl_scrape

Scrape a single page and return clean markdown. Extracts title, description, links, images, and metadata. Strips navigation, footer, ads, and tracking.

"Scrape https://example.com and give me the main content"

deepcrawl_clone

Clone a full page with all assets: HTML, CSS, JS, images, fonts, favicons. Downloads everything into a local folder, rewrites URLs to relative paths. Open index.html in a browser and it works.

"Clone https://competitor.com into a local folder"

deepcrawl_crawl

Crawl an entire site following internal links. Returns every page as clean markdown. Great for content analysis, SEO audit, or feeding a RAG pipeline.

"Crawl https://docs.example.com and return all pages as markdown"

deepcrawl_map

Discover all URLs from a site via sitemap.xml parsing and homepage link crawling. Run this before a crawl to see the site's scope.

"Map all pages on https://example.com"

vs Firecrawl

deepcrawl handles static sites out of the box. For JS-heavy SPAs (React, Next.js, SvelteKit), install Playwright for full rendering:

npm install -g playwright
npx playwright install chromium

Once installed, deepcrawl auto-detects Playwright and enables JS rendering. Use jsRender: true on any tool to activate it. All tools also include UA rotation, realistic browser headers, and random delays to avoid basic bot detection.

Also by Soflution

• brandcheck - Check brand name availability across 27 platforms

• depsonar - Dependency audit, security scan, license check

License

MIT