Skip to main content
Glama
deepcrawl-mcp

deepcrawl-mcp

by Soflutionltd
README.md
<p align="center">
  <img src="static/banner.svg" alt="deepcrawl" width="700" />
</p>

<p align="center">
  <a href="https://www.npmjs.com/package/deepcrawl-mcp"><img src="https://img.shields.io/npm/v/deepcrawl-mcp.svg?style=flat-square&color=06b6d4" alt="npm version" /></a>
  <img src="https://img.shields.io/badge/tools-4-8b5cf6?style=flat-square" alt="4 tools" />
  <img src="https://img.shields.io/badge/price-free-06b6d4?style=flat-square" alt="free" />
  <img src="https://img.shields.io/badge/API_keys-none-06b6d4?style=flat-square" alt="no API keys" />
</p>

<p align="center">
  <a href="cursor://anysphere.cursor-deeplink/mcp/install?name=deepcrawl-mcp&config=eyJ0eXBlIjoic3RkaW8iLCJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsImRlZXBjcmF3bC1tY3BAbGF0ZXN0Il19"><img src="https://img.shields.io/badge/Install_in-Cursor-blue?style=for-the-badge&logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHBhdGggZD0iTTEyIDJMMiAxMmwxMCAxMCAxMC0xMEwxMiAyeiIgZmlsbD0id2hpdGUiLz48L3N2Zz4=" alt="Install in Cursor" /></a>
  <a href="https://soflutionltd.github.io/BrandCheck/install.html?name=deepcrawl-mcp&pkg=deepcrawl-mcp"><img src="https://img.shields.io/badge/Install_via-Web-green?style=for-the-badge" alt="Install via Web" /></a>
</p>

---

**Clone, scrape and crawl any website.** Free Firecrawl alternative. No API keys, no rate limits, no subscription.

## Quick Start

**Cursor (one-click):** Click the "Install in Cursor" button above.

**Manual MCP config:**
```json
{
  "mcpServers": {
    "deepcrawl-mcp": {
      "command": "npx",
      "args": ["-y", "deepcrawl-mcp@latest"]
    }
  }
}
```

### Automatic updates

Every install below keeps itself up to date, without restarting Claude or Cursor. The MCP client talks to a small supervisor (`bin/supervise.cjs`) that stays connected while the server runs behind it. At start-up and every 6 hours it asks npm for the latest `deepcrawl-mcp`, installs it in the background in `~/.soflution/mcp/deepcrawl-mcp/`, and switches over as soon as no request is in flight (the client is told the tool list may have changed). A version that fails to start is rolled back at once and never picked again on that machine. npm checks the integrity of every download, and releases are published from GitHub Actions with provenance.

The only network call is that version check to `registry.npmjs.org`; nothing about your machine or your use is sent. Turn it off with `NO_UPDATE_NOTIFIER=1` (or `SOFLUTION_AUTO_UPDATE=0`). Started directly with `node dist/...`, deepcrawl-mcp does not update itself and only mentions a newer version in its first tool result.

**CLI:**
```bash
npx deepcrawl-mcp@latest
```

## Tools

### `deepcrawl_scrape`
Scrape a single page and return clean markdown. Extracts title, description, links, images, and metadata. Strips navigation, footer, ads, and tracking.

```
"Scrape https://example.com and give me the main content"
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `url` | required | Page URL to scrape |
| `mainContentOnly` | `true` | Extract only main content (skip nav/footer) |
| `includeLinks` | `true` | Include discovered links |
| `includeImages` | `true` | Include image URLs |

### `deepcrawl_clone`
Clone a full page with all assets: HTML, CSS, JS, images, fonts, favicons. Downloads everything into a local folder, rewrites URLs to relative paths. Open `index.html` in a browser and it works.

```
"Clone https://competitor.com into a local folder"
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `url` | required | Page URL to clone |
| `outputDir` | `~/deepcrawl-clones/<domain>` | Output folder |
| `depth` | `0` | Link depth: 0 = single page, 1+ = follow links |

### `deepcrawl_crawl`
Crawl an entire site following internal links. Returns every page as clean markdown. Great for content analysis, SEO audit, or feeding a RAG pipeline.

```
"Crawl https://docs.example.com and return all pages as markdown"
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `url` | required | Starting URL |
| `maxPages` | `20` | Max pages to crawl (max: 100) |
| `includeImages` | `false` | Include image URLs per page |

### `deepcrawl_map`
Discover all URLs from a site via sitemap.xml parsing and homepage link crawling. Run this before a crawl to see the site's scope.

```
"Map all pages on https://example.com"
```

| Parameter | Default | Description |
|-----------|---------|-------------|
| `url` | required | Site URL |
| `maxUrls` | `200` | Max URLs to discover |

## vs Firecrawl

| | deepcrawl | Firecrawl |
|---|---|---|
| Price | Free | $19+/mo |
| API key | None | Required |
| Rate limits | None | Yes |
| Scrape to markdown | Yes | Yes |
| Full site crawl | Yes | Yes |
| Site map | Yes | Yes |
| Clone with assets | **Yes** | No |
| JS rendering | **Yes** (via Playwright) | Yes |
| Anti-bot bypass | Partial (UA rotation, headers, delays) | Yes |

deepcrawl handles static sites out of the box. For JS-heavy SPAs (React, Next.js, SvelteKit), install Playwright for full rendering:

```bash
npm install -g playwright
npx playwright install chromium
```

Once installed, deepcrawl auto-detects Playwright and enables JS rendering. Use `jsRender: true` on any tool to activate it. All tools also include UA rotation, realistic browser headers, and random delays to avoid basic bot detection.

## Also by Soflution

- **[brandcheck](https://github.com/Soflutionltd/BrandCheck)** - Check brand name availability across 27 platforms
- **[depsonar](https://github.com/Soflutionltd/depsonar)** - Dependency audit, security scan, license check

## License

MIT