FireCrawl MCP Server

A Model Context Protocol (MCP) server implementation that integrates with FireCrawl for advanced web scraping capabilities.

Features

Web scraping with JavaScript rendering
Batch scraping with parallel processing and queuing
URL discovery and crawling
Web search with content extraction
Automatic retries with exponential backoff
Credit usage monitoring for cloud API
Comprehensive logging system
Support for cloud and self-hosted FireCrawl instances
Mobile/Desktop viewport support
Smart content filtering with tag inclusion/exclusion

Installation

Installing via Smithery

To install FireCrawl for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install mcp-server-firecrawl --client claude

Manual Installation

npm install -g mcp-server-firecrawl

Configuration

Environment Variables

FIRE_CRAWL_API_KEY: Your FireCrawl API key
- Required when using cloud API (default)
- Optional when using self-hosted instance with FIRE_CRAWL_API_URL
FIRE_CRAWL_API_URL (Optional): Custom API endpoint for self-hosted instances
- Example: https://firecrawl.your-domain.com
- If not provided, the cloud API will be used (requires API key)

Configuration Examples

For cloud API usage (default):

export FIRE_CRAWL_API_KEY=your-api-key

For self-hosted instance without authentication:

export FIRE_CRAWL_API_URL=https://firecrawl.your-domain.com

For self-hosted instance with authentication:

export FIRE_CRAWL_API_URL=https://firecrawl.your-domain.com
export FIRE_CRAWL_API_KEY=your-api-key  # Optional for authenticated self-hosted instances

Usage with Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "mcp-server-firecrawl"],
      "env": {
        "FIRE_CRAWL_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }
}

System Configuration

The server includes several configurable parameters:

const CONFIG = {
  retry: {
    maxAttempts: 3,
    initialDelay: 1000, // 1 second
    maxDelay: 10000, // 10 seconds
    backoffFactor: 2,
  },
  batch: {
    delayBetweenRequests: 2000, // 2 seconds
    maxParallelOperations: 3,
  },
  credit: {
    warningThreshold: 1000,
    criticalThreshold: 100,
  },
};

Rate Limits

The server implements rate limiting to prevent API abuse:

3 requests per minute on free tier
Automatic retries with exponential backoff
Parallel processing for batch operations
Higher limits available on paid plans

Available Tools

1. Scrape Tool (`fire_crawl_scrape`)

Scrape content from a single URL with advanced options.

{
  "name": "fire_crawl_scrape",
  "arguments": {
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 1000,
    "timeout": 30000,
    "mobile": false,
    "includeTags": ["article", "main"],
    "excludeTags": ["nav", "footer"],
    "skipTlsVerification": false
  }
}

2. Batch Scrape Tool (`fire_crawl_batch_scrape`)

Scrape multiple URLs with parallel processing and queuing.

{
  "name": "fire_crawl_batch_scrape",
  "arguments": {
    "urls": ["https://example1.com", "https://example2.com"],
    "options": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

Response includes operation ID for status checking:

{
  "content": [
    {
      "type": "text",
      "text": "Batch operation queued with ID: batch_1. Use fire_crawl_check_batch_status to check progress."
    }
  ],
  "isError": false
}

3. Check Batch Status (`fire_crawl_check_batch_status`)

Check the status of a batch operation.

{
  "name": "fire_crawl_check_batch_status",
  "arguments": {
    "id": "batch_1"
  }
}

4. Search Tool (`fire_crawl_search`)

Search the web and optionally extract content from search results.

{
  "name": "fire_crawl_search",
  "arguments": {
    "query": "your search query",
    "limit": 5,
    "lang": "en",
    "country": "us",
    "scrapeOptions": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

5. Crawl Tool (`fire_crawl_crawl`)

Start an asynchronous crawl with advanced options.

{
  "name": "fire_crawl_crawl",
  "arguments": {
    "url": "https://example.com",
    "maxDepth": 2,
    "limit": 100,
    "allowExternalLinks": false,
    "deduplicateSimilarURLs": true
  }
}

Logging System

The server includes comprehensive logging:

Operation status and progress
Performance metrics
Credit usage monitoring
Rate limit tracking
Error conditions

Example log messages:

[INFO] FireCrawl MCP Server initialized successfully
[INFO] Starting scrape for URL: https://example.com
[INFO] Batch operation queued with ID: batch_1
[WARNING] Credit usage has reached warning threshold
[ERROR] Rate limit exceeded, retrying in 2s...

Error Handling

The server provides robust error handling:

Automatic retries for transient errors
Rate limit handling with backoff
Detailed error messages
Credit usage warnings
Network resilience

Example error response:

{
  "content": [
    {
      "type": "text",
      "text": "Error: Rate limit exceeded. Retrying in 2 seconds..."
    }
  ],
  "isError": true
}

Development

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

Contributing

Fork the repository
Create your feature branch
Run tests: npm test
Submit a pull request

License

MIT License - see LICENSE file for details