shantycrawl-mcp

Why ShantyCrawl MCP

Most full-featured MCP servers register every tool on every session — including the ones you'll rarely touch — with full JSON schemas re-sent on each tool-list fetch. ShantyCrawl loads 6 core tools up front and lazy-loads the rest on demand, which measured out to ~18,000 fewer tokens on a fresh session in our testing:

Is This For You?

• ✅ Good fit — you use Firecrawl mostly for scrape/crawl/search, run long or multi-tool agent sessions, or are tight on context budget.

• ⚪ Less critical — you're already using tool_enable-style dynamic loading elsewhere, or you live almost entirely in the monitoring/research toolset and rarely call the core 6 — in that case the savings are smaller, since you'll be loading most of the schema anyway.

Related MCP server: webcrawl-mcp

🚀 Quick Start

Run instantly via npx — no installation required:

FIRECRAWL_API_URL=http://localhost:3002 npx shantycrawl-mcp

Environment variables:

Requirements: Node.js 18+ and a reachable Firecrawl instance (self-hosted or cloud).

⚙️ Configuration

Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "shantycrawl": {
      "command": "npx",
      "args": ["shantycrawl-mcp"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

OpenCode

Add this to your workspace settings:

{
  "mcp": {
    "shantycrawl": {
      "type": "local",
      "command": ["npx", "shantycrawl-mcp"],
      "environment": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

🛠️ Tool Architecture

Always Available (6 Core Tools)

Lazy-Loaded Tools (22 Advanced)

Activate any tool instantly during a session — e.g. tool_enable map unblocks the mapping capability.

🧠 How Dynamic Loading Works

1. The client sees a minimal 6-tool schema on connect.

2. When the AI needs an advanced capability, it calls tool_enable <tool_name>.

3. The server fires a notifications/tools/list_changed event, and the client's available tools update instantly — no reconnect required.

4. Call tool_disable <tool_name> once you're done to free the context back up.

⚠️ Known Issues

Some agents don't call tool_enable on their own. A compelling tool description isn't always enough — if another skill or system prompt has already taught the agent a different path to the same outcome (e.g. calling the Firecrawl API directly via curl/raw HTTP), it can follow that instead of discovering tool_enable. This shows up most with coding agents that have built-in or third-party Firecrawl skills already loaded.

Fix: add an explicit instruction to your agent's system prompt or AGENTS.md/CLAUDE.md:

This project uses the shantycrawl-mcp server. Advanced tools (map, extract, parse,
agent, interact, research_*, monitor_*) are not available until activated with
tool_enable("<tool_name>"). Always use the MCP server's tools directly — never call
the Firecrawl API via curl, raw HTTP, or other bash-based workarounds.

We also ship a ready-made SKILL.md that encodes this protocol — drop it into your skills directory if your agent supports skill loading, and pair it with the snippet above for the strongest guarantee.

🏗️ Local Development

git clone https://github.com/schlemperdev/shantycrawl-mcp.git
cd shantycrawl-mcp
npm install
npm run build
npm start

🤝 Contributing

Issues and PRs are welcome. If you're adding a new lazy-loaded tool, please keep its schema as lean as the core 6 — that's the whole point of this project.

📄 License

MIT © schlemperdev