Firecrawl Agent MCP Server

# Firecrawl Agent MCP Server - Complete Documentation **Version:** 1.0.0 **Last Updated:** December 20, 2025 **Repository:** [Replicant-Partners/Firecrawler-MCP](https://github.com/Replicant-Partners/Firecrawler-MCP) --- ## Table of Contents 1. [Overview](#overview) 2. [Features](#features) 3. [Architecture](#architecture) 4. [Prerequisites](#prerequisites) 5. [Installation](#installation) 6. [Configuration](#configuration) 7. [MCP Tools Reference](#mcp-tools-reference) 8. [Usage Examples](#usage-examples) 9. [API Integration](#api-integration) 10. [Development Guide](#development-guide) 11. [Testing](#testing) 12. [Troubleshooting](#troubleshooting) 13. [Cost Management](#cost-management) 14. [Advanced Topics](#advanced-topics) 15. [Contributing](#contributing) 16. [License](#license) --- ## Overview The Firecrawl Agent MCP Server is a Model Context Protocol (MCP) implementation that provides AI-powered web data extraction and research capabilities through [Firecrawl's API](https://www.firecrawl.dev/). It enables Claude Code and other MCP-compatible clients to autonomously search, navigate, and gather data from complex websites. ### What is Firecrawl Agent? Firecrawl Agent is an AI-powered web scraping and research tool that: - **Requires No URLs**: Simply describe what data you need - **Autonomous Navigation**: Searches and navigates websites automatically - **Parallel Processing**: Handles multiple sources simultaneously - **Structured Output**: Returns data in your specified JSON schema format - **Dynamic Billing**: Costs scale with task complexity ### What is MCP? The Model Context Protocol (MCP) is a standard protocol for connecting AI assistants to external tools and data sources. This server implements MCP to expose Firecrawl's capabilities to Claude Code and other MCP clients. --- ## Features ### Core Capabilities - ✅ **Web Scraping**: Extract content from single URLs in multiple formats - ✅ **Web Search**: Search and scrape multiple results simultaneously - 🚧 **AI Agent Mode**: Autonomous research and data gathering (requires API access) - 📊 **Structured Extraction**: Define JSON schemas for type-safe data - 🔄 **Async Jobs**: Start long-running tasks and poll for results - 💰 **Cost Control**: Set credit limits per request ### Output Formats - **Markdown**: Clean, formatted text - **HTML**: Full HTML content - **Raw HTML**: Unprocessed HTML - **Links**: Extracted URLs - **Screenshot**: Page captures (PNG) ### Transport Modes - **stdio**: Direct process communication (default) - **SSE**: HTTP Server-Sent Events for web-based clients --- ## Architecture ### Project Structure ``` firecrawl-agent-mcp/ ├── src/ │ ├── server.ts # Main MCP server implementation │ ├── services/ │ │ └── firecrawl-client.ts # Firecrawl API client │ ├── types/ │ │ └── firecrawl.ts # TypeScript type definitions │ └── test.ts # Test suite ├── dist/ # Compiled JavaScript ├── node_modules/ # Dependencies ├── .env # Environment configuration (not in git) ├── .env.example # Example configuration ├── .gitignore # Git ignore rules ├── package.json # NPM configuration ├── tsconfig.json # TypeScript configuration ├── README.md # Quick start guide └── DOCUMENTATION.md # This file ``` ### Technology Stack - **Language**: TypeScript 5.7 - **Runtime**: Node.js 18+ - **MCP SDK**: @modelcontextprotocol/sdk 1.0.4 - **HTTP Client**: Native Fetch API - **Validation**: Zod 3.24 - **Build Tool**: TypeScript Compiler - **Transport**: stdio / SSE ### Component Architecture ``` ┌─────────────────────────────────────────┐ │ Claude Code / MCP Client │ └────────────────┬────────────────────────┘ │ MCP Protocol │ ┌────────────────▼────────────────────────┐ │ Firecrawl Agent MCP Server │ │ ┌───────────────────────────────────┐ │ │ │ server.ts │ │ │ │ - Tool registration │ │ │ │ - Request routing │ │ │ │ - Response formatting │ │ │ └──────────────┬────────────────────┘ │ │ │ │ │ ┌──────────────▼────────────────────┐ │ │ │ firecrawl-client.ts │ │ │ │ - API authentication │ │ │ │ - Request construction │ │ │ │ - Response parsing │ │ │ │ - Error handling │ │ │ └──────────────┬────────────────────┘ │ └─────────────────┼────────────────────────┘ │ HTTPS │ ┌─────────────────▼────────────────────────┐ │ Firecrawl API (api.firecrawl.dev)│ │ - /v1/scrape │ │ - /v1/search │ │ - /v1/agent │ │ - /v1/agent/start │ │ - /v1/agent/{jobId} │ └──────────────────────────────────────────┘ ``` --- ## Prerequisites ### Required - **Node.js**: Version 18.0.0 or higher - **npm**: Comes with Node.js - **Firecrawl API Key**: Get from [firecrawl.dev](https://www.firecrawl.dev/) - **Claude Code**: Or any MCP-compatible client ### Optional - **Git**: For version control - **GitHub CLI** (`gh`): For repository operations - **TypeScript**: Global installation for development ### System Requirements - **OS**: Linux, macOS, or Windows (with WSL2) - **RAM**: 512MB minimum - **Disk**: 100MB for installation --- ## Installation ### Quick Start ```bash # 1. Clone the repository git clone https://github.com/Replicant-Partners/Firecrawler-MCP.git cd Firecrawler-MCP # 2. Install dependencies npm install # 3. Configure API key cp .env.example .env # Edit .env and add your FIRECRAWL_API_KEY # 4. Build the project npm run build # 5. Test the installation npm test ``` ### From Scratch Setup ```bash # Create project directory mkdir firecrawl-agent-mcp cd firecrawl-agent-mcp # Clone repository contents git clone https://github.com/Replicant-Partners/Firecrawler-MCP.git . # Install dependencies npm install # Configure cp .env.example .env nano .env # Add your API key # Build npm run build ``` ### Verify Installation ```bash # Check Node.js version node --version # Should be 18+ # Check build output ls -la dist/ # Should contain compiled .js files # Run tests npm test # Should show successful API connections ``` --- ## Configuration ### Environment Variables Create a `.env` file in the project root: ```env # Required: Your Firecrawl API key FIRECRAWL_API_KEY=fc-your-key-here # Optional: Custom API base URL FIRECRAWL_API_BASE=https://api.firecrawl.dev # Optional: Transport mode (stdio or sse) TRANSPORT=stdio # Optional: SSE server port (only for SSE transport) PORT=3000 ``` ### Getting a Firecrawl API Key 1. Visit [https://www.firecrawl.dev/](https://www.firecrawl.dev/) 2. Sign up for an account 3. Navigate to your dashboard 4. Copy your API key (format: `fc-...`) 5. Add to your `.env` file ### MCP Configuration #### For Claude Code Add to your `.claude/settings.json` or project `.mcp.json`: ```json { "mcpServers": { "firecrawl-agent": { "command": "node", "args": ["dist/server.js"], "cwd": "/absolute/path/to/firecrawl-agent-mcp", "type": "stdio", "env": { "FIRECRAWL_API_KEY": "fc-your-key-here", "TRANSPORT": "stdio" } } } } ``` #### For Other MCP Clients **stdio Transport** (recommended): ```json { "command": "node", "args": ["/path/to/dist/server.js"], "env": { "FIRECRAWL_API_KEY": "fc-your-key-here" } } ``` **SSE Transport** (for web clients): ```json { "url": "http://localhost:3000/sse", "env": { "PORT": "3000", "TRANSPORT": "sse", "FIRECRAWL_API_KEY": "fc-your-key-here" } } ``` --- ## MCP Tools Reference ### 1. agent_execute **Description**: Execute Firecrawl Agent synchronously. Waits for completion and returns results immediately. **Use When**: You need immediate results for research tasks. **Parameters**: ```typescript { prompt: string; // Required: Describe what data to extract urls?: string[]; // Optional: Specific URLs to search schema?: object; // Optional: JSON schema for structured output maxCredits?: number; // Optional: Credit limit for cost control } ``` **Returns**: ```typescript { success: boolean; data: any; // Extracted data (format depends on schema) creditsUsed?: number; // Credits consumed by this request error?: string; // Error message if failed } ``` **Example**: ```json { "prompt": "Find the founders of Anthropic and when it was founded", "schema": { "type": "object", "properties": { "founders": { "type": "array", "items": { "type": "string" } }, "founded": { "type": "number" } } }, "maxCredits": 50 } ``` --- ### 2. agent_start **Description**: Start an agent job asynchronously. Returns job ID immediately without waiting. **Use When**: Long-running research tasks where you want to poll for results. **Parameters**: Same as `agent_execute` **Returns**: ```typescript { success: boolean; jobId?: string; // Job ID for status checking message?: string; // Instructions for next steps error?: string; // Error message if failed } ``` **Example**: ```json { "prompt": "Research all major AI companies founded in 2023 and their funding", "maxCredits": 100 } ``` **Response**: ```json { "success": true, "jobId": "job_abc123xyz", "message": "Agent job started. Use agent_status with this jobId to check progress." } ``` --- ### 3. agent_status **Description**: Check the status of an asynchronous agent job. **Use When**: Polling for results from `agent_start`. **Parameters**: ```typescript { jobId: string; // Required: Job ID from agent_start } ``` **Returns**: ```typescript { id: string; // Job ID status: 'pending' | 'running' | 'completed' | 'failed'; data?: any; // Results if completed error?: string; // Error message if failed creditsUsed?: number; // Credits consumed progress?: number; // Progress percentage (0-100) } ``` **Example**: ```json { "jobId": "job_abc123xyz" } ``` **Status Flow**: ``` pending → running → completed ↓ failed ``` **Job Retention**: Results available for 24 hours after completion. --- ### 4. scrape **Description**: Scrape a single URL and extract content in various formats. **Use When**: Simple single-page scraping without AI agent capabilities. **Parameters**: ```typescript { url: string; // Required: URL to scrape formats?: string[]; // Optional: ['markdown', 'html', 'rawHtml', 'links', 'screenshot'] onlyMainContent?: boolean; // Optional: Extract only main content (default: true) includeTags?: string[]; // Optional: HTML tags to include excludeTags?: string[]; // Optional: HTML tags to exclude waitFor?: number; // Optional: Wait time for JS rendering (ms) timeout?: number; // Optional: Request timeout (ms) } ``` **Returns**: ```typescript { success: boolean; data?: { markdown?: string; html?: string; rawHtml?: string; links?: string[]; screenshot?: string; // Base64 encoded PNG metadata?: object; }; error?: string; } ``` **Example**: ```json { "url": "https://www.anthropic.com/", "formats": ["markdown", "links"], "onlyMainContent": true, "excludeTags": ["nav", "footer"] } ``` --- ### 5. search **Description**: Search the web and scrape the results. **Use When**: Finding and extracting data from multiple sources at once. **Parameters**: ```typescript { query: string; // Required: Search query limit?: number; // Optional: Max results (default: 5) formats?: string[]; // Optional: Output formats per result } ``` **Returns**: ```typescript { success: boolean; data?: Array<{ url: string; markdown?: string; html?: string; rawHtml?: string; links?: string[]; metadata?: object; }>; error?: string; } ``` **Example**: ```json { "query": "best AI coding assistants 2025", "limit": 5, "formats": ["markdown"] } ``` --- ## Usage Examples ### Example 1: Basic Web Scraping **Task**: Extract the main content from a webpage. ```typescript // Tool: scrape { "url": "https://www.anthropic.com/news", "formats": ["markdown"], "onlyMainContent": true } // Result: { "success": true, "data": { "markdown": "# Latest News\n\n## Anthropic Launches...\n...", "metadata": { "title": "News - Anthropic", "description": "Latest updates..." } } } ``` --- ### Example 2: Structured Data Extraction **Task**: Extract pricing information in a structured format. ```typescript // Tool: agent_execute { "prompt": "Extract all pricing tiers from Anthropic's Claude API pricing page", "urls": ["https://www.anthropic.com/pricing"], "schema": { "type": "object", "properties": { "tiers": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "string" }, "features": { "type": "array", "items": { "type": "string" } } } } } } } } // Result: { "success": true, "data": { "tiers": [ { "name": "Free", "price": "$0", "features": ["5M tokens/month", "Basic support"] }, { "name": "Pro", "price": "$20/month", "features": ["100M tokens/month", "Priority support"] } ] }, "creditsUsed": 15 } ``` --- ### Example 3: Competitive Research **Task**: Research multiple companies and compare features. ```typescript // Tool: agent_execute { "prompt": "Compare features of GitHub Copilot, Cursor, and Claude Code", "schema": { "type": "object", "properties": { "tools": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "pricing": { "type": "string" }, "key_features": { "type": "array", "items": { "type": "string" } }, "ide_support": { "type": "array", "items": { "type": "string" } } } } } } }, "maxCredits": 100 } ``` --- ### Example 4: Multi-Source Search **Task**: Search for information across multiple websites. ```typescript // Tool: search { "query": "Firecrawl API documentation examples", "limit": 3, "formats": ["markdown", "links"] } // Result: { "success": true, "data": [ { "url": "https://docs.firecrawl.dev/", "markdown": "# Firecrawl Documentation\n...", "links": ["https://docs.firecrawl.dev/api-reference", ...] }, { "url": "https://www.firecrawl.dev/blog/", "markdown": "# Blog Posts\n...", "links": [...] } ] } ``` --- ### Example 5: Async Long-Running Task **Task**: Start a comprehensive research job and poll for results. ```typescript // Step 1: Start the job // Tool: agent_start { "prompt": "Research the top 10 AI research labs and their recent breakthrough papers in 2024", "maxCredits": 200 } // Response: { "success": true, "jobId": "job_xyz789" } // Step 2: Check status (poll every 5-10 seconds) // Tool: agent_status { "jobId": "job_xyz789" } // Response (while running): { "id": "job_xyz789", "status": "running", "progress": 45 } // Response (when complete): { "id": "job_xyz789", "status": "completed", "data": { "labs": [...] }, "creditsUsed": 150, "progress": 100 } ``` --- ### Example 6: Contact Information Extraction **Task**: Extract contact information from company websites. ```typescript // Tool: agent_execute { "prompt": "Find contact emails and phone numbers from the contact page", "urls": ["https://www.anthropic.com/contact"], "schema": { "type": "object", "properties": { "emails": { "type": "array", "items": { "type": "string" } }, "phones": { "type": "array", "items": { "type": "string" } }, "address": { "type": "string" } } } } ``` --- ## API Integration ### Firecrawl API Details **Base URL**: `https://api.firecrawl.dev` **Authentication**: Bearer token in Authorization header ``` Authorization: Bearer fc-your-api-key ``` **API Endpoints**: | Endpoint | Method | Purpose | |----------|--------|---------| | `/v1/scrape` | POST | Single page scraping | | `/v1/search` | POST | Web search and scrape | | `/v1/agent` | POST | Synchronous agent execution | | `/v1/agent/start` | POST | Start async agent job | | `/v1/agent/{jobId}` | GET | Get agent job status | ### Request Format All POST requests send JSON with `Content-Type: application/json`: ```typescript // Example scrape request POST /v1/scrape Content-Type: application/json Authorization: Bearer fc-your-key { "url": "https://example.com", "formats": ["markdown"] } ``` ### Response Format All responses return JSON: ```typescript // Success response { "success": true, "data": { /* extracted data */ }, "creditsUsed": 5 } // Error response { "success": false, "error": "Error message here" } ``` ### Error Handling The client automatically handles: - **Network errors**: Connection timeouts, DNS failures - **HTTP errors**: 4xx and 5xx status codes - **JSON parsing errors**: Invalid response format - **Rate limiting**: 429 status codes Error responses include: ```typescript { "success": false, "error": "HTTP 401: Unauthorized" } ``` ### Rate Limiting Firecrawl API implements rate limiting: - **Free tier**: ~100 requests/hour - **Pro tier**: ~1000 requests/hour - **Enterprise**: Custom limits Rate limit errors return HTTP 429 with retry-after header. --- ## Development Guide ### Setup Development Environment ```bash # Clone and install git clone https://github.com/Replicant-Partners/Firecrawler-MCP.git cd Firecrawler-MCP npm install # Configure cp .env.example .env # Add your API key to .env # Start development mode (watch for changes) npm run dev ``` ### Project Scripts ```json { "build": "tsc", // Compile TypeScript to JavaScript "dev": "tsc --watch", // Watch mode for development "start": "node dist/server.js", // Run the server "start:sse": "TRANSPORT=sse node dist/server.js", // SSE mode "test": "node dist/test.js" // Run tests } ``` ### Adding a New Tool 1. **Define the tool in `server.ts`**: ```typescript const TOOLS: Tool[] = [ // ... existing tools { name: 'my_new_tool', description: 'Description of what this tool does', inputSchema: { type: 'object', properties: { myParam: { type: 'string', description: 'Parameter description', }, }, required: ['myParam'], }, }, ]; ``` 2. **Implement the handler**: ```typescript server.setRequestHandler(CallToolRequestSchema, async (request) => { const { name, arguments: args } = request.params; switch (name) { // ... existing cases case 'my_new_tool': { const { myParam } = args as { myParam: string }; // Your implementation here const result = await myService.doSomething(myParam); return { content: [ { type: 'text', text: JSON.stringify(result, null, 2), }, ], }; } } }); ``` 3. **Add types if needed** in `src/types/firecrawl.ts`: ```typescript export interface MyNewToolRequest { myParam: string; } export interface MyNewToolResponse { success: boolean; data?: any; error?: string; } ``` 4. **Rebuild and test**: ```bash npm run build npm test ``` ### Code Style - **TypeScript**: Strict mode enabled - **Formatting**: 2-space indentation - **Naming**: camelCase for variables, PascalCase for types - **Async/Await**: Prefer over promises - **Error Handling**: Always use try-catch blocks ### Testing Your Changes ```bash # Compile npm run build # Run test suite npm test # Test specific tool manually node -e " const { FirecrawlClient } = require('./dist/services/firecrawl-client.js'); const client = new FirecrawlClient({ apiKey: 'your-key' }); client.scrape({ url: 'https://example.com' }) .then(r => console.log(r)) .catch(e => console.error(e)); " ``` --- ## Testing ### Running Tests ```bash # Run all tests npm test # Expected output: # 🔥 Testing Firecrawl Agent MCP Server # ✓ API Key loaded # Test 1: Simple Scrape # ✓ Scrape successful! # Test 2: Web Search # ✓ Search successful! # Test 3: Agent Execute # ✓ Agent execution successful! # ✅ Tests completed! ``` ### Test Coverage The test suite (`src/test.ts`) covers: 1. **API Authentication**: Validates API key is loaded 2. **Scrape Function**: Tests single page scraping 3. **Search Function**: Tests web search capabilities 4. **Agent Execution**: Tests agent mode (if available) ### Manual Testing with Claude Code After installation and configuration: ```typescript // Test 1: Ask Claude to scrape "Use Firecrawl to scrape https://www.anthropic.com/" // Test 2: Ask Claude to search "Search for 'Claude AI features' using Firecrawl" // Test 3: Ask Claude for structured data "Use Firecrawl to find and extract the founding team of Anthropic" ``` ### Debugging Enable debug logging: ```bash # Set debug environment variable DEBUG=firecrawl:* npm start # Or in .env DEBUG=firecrawl:* ``` View server logs: ```bash # The server logs to stderr npm start 2> server.log # View logs tail -f server.log ``` --- ## Troubleshooting ### Common Issues #### 1. "FIRECRAWL_API_KEY environment variable is required" **Problem**: API key not found. **Solution**: ```bash # Check .env file exists ls -la .env # Verify API key is set cat .env | grep FIRECRAWL_API_KEY # Recreate from example cp .env.example .env nano .env # Add your key ``` #### 2. "HTTP 401: Unauthorized" **Problem**: Invalid API key. **Solution**: - Verify API key is correct (starts with `fc-`) - Check key hasn't expired - Get a new key from [firecrawl.dev](https://www.firecrawl.dev/) #### 3. "HTTP 429: Too Many Requests" **Problem**: Rate limit exceeded. **Solution**: - Wait a few minutes before retrying - Reduce request frequency - Upgrade Firecrawl plan for higher limits #### 4. "Repository not found" when pushing to GitHub **Problem**: Repository doesn't exist or no access. **Solution**: ```bash # Create repository first gh repo create Replicant-Partners/Firecrawler-MCP --public # Or check access rights gh auth status ``` #### 5. Tools not showing up in Claude Code **Problem**: MCP configuration not loaded. **Solution**: - Verify `.mcp.json` or `.claude/settings.json` is correct - Use absolute paths for `cwd` - Restart Claude Code - Check build completed: `ls -la dist/` #### 6. "Cannot find module" errors **Problem**: Dependencies not installed or build missing. **Solution**: ```bash # Reinstall dependencies rm -rf node_modules package-lock.json npm install # Rebuild npm run build ``` #### 7. TypeScript compilation errors **Problem**: Type errors in code. **Solution**: ```bash # Check TypeScript version npx tsc --version # Should be 5.7+ # Clean build rm -rf dist/ npm run build # View detailed errors npx tsc --noEmit ``` #### 8. Agent endpoint returning HTML instead of JSON **Problem**: Agent API may not be available for your API tier. **Solution**: - Verify agent access with Firecrawl support - Use `scrape` and `search` tools as alternatives - Check if agent endpoint requires early access ### Getting Help 1. **Check Documentation**: Review this file and README.md 2. **GitHub Issues**: [Report bugs](https://github.com/Replicant-Partners/Firecrawler-MCP/issues) 3. **Firecrawl Support**: Contact for API-related issues 4. **Claude Code Help**: Visit [claude.com/code](https://claude.com/code) --- ## Cost Management ### Understanding Firecrawl Credits Firecrawl uses a credit-based billing system: - **Scrape**: 1-5 credits per page - **Search**: 2-10 credits per query - **Agent**: Variable (5-100+ credits depending on complexity) ### Controlling Costs #### 1. Set Credit Limits ```typescript { "prompt": "Your research task", "maxCredits": 50 // Stop after 50 credits } ``` #### 2. Use Specific URLs Instead of letting agent search: ```typescript // More expensive (agent searches) { "prompt": "Find pricing from Anthropic" } // Less expensive (direct URL) { "prompt": "Extract pricing", "urls": ["https://www.anthropic.com/pricing"] } ``` #### 3. Optimize Schemas More specific schemas = faster extraction = lower cost: ```typescript // Good: Specific schema { "schema": { "type": "object", "properties": { "price": { "type": "number" } } } } // Expensive: Vague or missing schema { "prompt": "Get all information about everything" } ``` #### 4. Use Appropriate Tools | Task | Recommended Tool | Cost | |------|-----------------|------| | Single page content | `scrape` | Low (1-5 credits) | | Find multiple pages | `search` | Medium (5-20 credits) | | Complex research | `agent_execute` | High (20-100+ credits) | ### Monitoring Usage Check credits used in responses: ```typescript { "success": true, "data": { ... }, "creditsUsed": 15 // This request used 15 credits } ``` Track total usage: ```bash # View Firecrawl dashboard open https://www.firecrawl.dev/dashboard # Check API usage curl -H "Authorization: Bearer fc-your-key" \ https://api.firecrawl.dev/v1/account/usage ``` --- ## Advanced Topics ### Custom Transport Implementation Create a custom transport: ```typescript import { Transport } from '@modelcontextprotocol/sdk/shared/transport.js'; class MyCustomTransport implements Transport { async start(): Promise<void> { // Initialize connection } async send(message: any): Promise<void> { // Send message } async close(): Promise<void> { // Close connection } } // Use in server const transport = new MyCustomTransport(); await server.connect(transport); ``` ### Environment-Specific Configuration ```typescript // src/config/environments.ts export const config = { development: { apiBase: 'https://api-dev.firecrawl.dev', timeout: 60000, }, production: { apiBase: 'https://api.firecrawl.dev', timeout: 30000, }, }; // Use const env = process.env.NODE_ENV || 'production'; const apiBase = config[env].apiBase; ``` ### Caching Responses Implement response caching: ```typescript import { createHash } from 'crypto'; const cache = new Map<string, any>(); function getCacheKey(request: any): string { return createHash('md5') .update(JSON.stringify(request)) .digest('hex'); } async function cachedRequest(request: any) { const key = getCacheKey(request); if (cache.has(key)) { return cache.get(key); } const result = await firecrawl.executeAgent(request); cache.set(key, result); return result; } ``` ### Webhook Integration Receive webhook notifications for async jobs: ```typescript import express from 'express'; const app = express(); app.post('/webhook/firecrawl', express.json(), (req, res) => { const { jobId, status, data } = req.body; console.log(`Job ${jobId} status: ${status}`); if (status === 'completed') { // Process results handleJobCompletion(data); } res.sendStatus(200); }); app.listen(8080); ``` ### Rate Limiting Implementation Implement client-side rate limiting: ```typescript import { RateLimiter } from 'limiter'; const limiter = new RateLimiter({ tokensPerInterval: 100, interval: 'hour', }); async function rateLimitedRequest(fn: () => Promise<any>) { await limiter.removeTokens(1); return fn(); } // Use const result = await rateLimitedRequest(() => firecrawl.scrape({ url: 'https://example.com' }) ); ``` ### Custom Error Handling Implement custom error handlers: ```typescript class FirecrawlError extends Error { constructor( message: string, public statusCode?: number, public retryable?: boolean ) { super(message); this.name = 'FirecrawlError'; } } function handleError(error: any): FirecrawlError { if (error.statusCode === 429) { return new FirecrawlError( 'Rate limit exceeded', 429, true // Can retry ); } return new FirecrawlError(error.message); } ``` --- ## Contributing ### How to Contribute 1. **Fork the repository** 2. **Create a feature branch**: `git checkout -b feature/amazing-feature` 3. **Make your changes** 4. **Test thoroughly**: `npm test` 5. **Commit with descriptive message** 6. **Push to your fork**: `git push origin feature/amazing-feature` 7. **Open a Pull Request** ### Code Quality Standards - All TypeScript code must compile without errors - Add JSDoc comments for public functions - Include tests for new features - Follow existing code style - Update documentation ### Pull Request Template ```markdown ## Description Brief description of changes ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update ## Testing - [ ] Tests pass locally - [ ] Added new tests - [ ] Manual testing completed ## Checklist - [ ] Code compiles - [ ] Documentation updated - [ ] No breaking changes (or documented) ``` ### Development Workflow ```bash # 1. Fork and clone git clone https://github.com/YOUR_USERNAME/Firecrawler-MCP.git cd Firecrawler-MCP # 2. Create branch git checkout -b feature/my-feature # 3. Make changes # ... edit files ... # 4. Test npm run build npm test # 5. Commit git add . git commit -m "feat: add amazing feature" # 6. Push git push origin feature/my-feature # 7. Create PR on GitHub ``` --- ## License MIT License Copyright (c) 2025 Replicant Partners Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- ## Appendix ### Useful Links - **Repository**: https://github.com/Replicant-Partners/Firecrawler-MCP - **Firecrawl Website**: https://www.firecrawl.dev/ - **Firecrawl Docs**: https://docs.firecrawl.dev/ - **Firecrawl Agent**: https://www.firecrawl.dev/agent - **MCP Protocol**: https://modelcontextprotocol.io/ - **Claude Code**: https://claude.com/claude-code ### Version History - **1.0.0** (2025-12-20): Initial release - Complete MCP server implementation - 5 MCP tools (agent_execute, agent_start, agent_status, scrape, search) - TypeScript with full type safety - Comprehensive documentation - Test suite ### Acknowledgments - Built with [Model Context Protocol](https://modelcontextprotocol.io/) - Powered by [Firecrawl API](https://www.firecrawl.dev/) - Developed for [Claude Code](https://claude.com/claude-code) --- **Last Updated**: December 20, 2025 **Document Version**: 1.0.0 **Maintained By**: Replicant Partners