Skip to main content
Glama

Why ShantyCrawl MCP

Most full-featured MCP servers register every tool on every session — including the ones you'll rarely touch — with full JSON schemas re-sent on each tool-list fetch. ShantyCrawl loads 6 core tools up front and lazy-loads the rest on demand, which measured out to ~18,000 fewer tokens on a fresh session in our testing:

firecrawl-mcp (official)

shantycrawl-mcp

Tools on session start

28

6

Initial schema footprint

Full schemas for all 28 tools

Minimal schemas for 6 tools

Advanced tools (research, monitoring, agent, etc.)

Always loaded

Loaded on demand via tool_enable

Measured token cost, fresh session

Baseline

~18k tokens lighter

Runtime dependencies

—

Zero third-party wrappers, native fetch

Is This For You?

  • ✅ Good fit — you use Firecrawl mostly for scrape/crawl/search, run long or multi-tool agent sessions, or are tight on context budget.

  • ⚪ Less critical — you're already using tool_enable-style dynamic loading elsewhere, or you live almost entirely in the monitoring/research toolset and rarely call the core 6 — in that case the savings are smaller, since you'll be loading most of the schema anyway.

Related MCP server: webcrawl-mcp

🚀 Quick Start

Run instantly via npx — no installation required:

FIRECRAWL_API_URL=http://localhost:3002 npx shantycrawl-mcp

Environment variables:

Variable

Required

Default

Description

FIRECRAWL_API_URL

No

http://localhost:3002

URL of your Firecrawl instance (self-hosted or cloud)

FIRECRAWL_API_KEY

Only for Firecrawl Cloud

—

API key, required if FIRECRAWL_API_URL points to a hosted/cloud instance

Requirements: Node.js 18+ and a reachable Firecrawl instance (self-hosted or cloud).

⚙️ Configuration

Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "shantycrawl": {
      "command": "npx",
      "args": ["shantycrawl-mcp"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

OpenCode

Add this to your workspace settings:

{
  "mcp": {
    "shantycrawl": {
      "type": "local",
      "command": ["npx", "shantycrawl-mcp"],
      "environment": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

🛠️ Tool Architecture

Always Available (6 Core Tools)

Tool

Description

scrape

Extract clean markdown from any URL

crawl

Crawl a target website recursively

search

Execute web searches via Firecrawl

tool_enable

Dynamically inject an advanced tool into the active session

tool_disable

Unload an advanced tool to free up context

check_crawl_status

Check the status of an async crawl job

Lazy-Loaded Tools (22 Advanced)

Activate any tool instantly during a session — e.g. tool_enable map unblocks the mapping capability.

Category

Tools

Discovery

map, extract, parse

AI Agent

agent, agent_status

Browser Interaction

interact, interact_stop

Research & Academic

research_search_papers, research_inspect_paper, research_read_paper, research_related_papers, research_search_github

Monitoring

monitor_create, monitor_check, monitor_checks, monitor_delete, monitor_get, monitor_list, monitor_run, monitor_update

Feedback

search_feedback, feedback

🧠 How Dynamic Loading Works

  1. The client sees a minimal 6-tool schema on connect.

  2. When the AI needs an advanced capability, it calls tool_enable <tool_name>.

  3. The server fires a notifications/tools/list_changed event, and the client's available tools update instantly — no reconnect required.

  4. Call tool_disable <tool_name> once you're done to free the context back up.

⚠️ Known Issues

Some agents don't call tool_enable on their own. A compelling tool description isn't always enough — if another skill or system prompt has already taught the agent a different path to the same outcome (e.g. calling the Firecrawl API directly via curl/raw HTTP), it can follow that instead of discovering tool_enable. This shows up most with coding agents that have built-in or third-party Firecrawl skills already loaded.

Fix: add an explicit instruction to your agent's system prompt or AGENTS.md/CLAUDE.md:

This project uses the shantycrawl-mcp server. Advanced tools (map, extract, parse,
agent, interact, research_*, monitor_*) are not available until activated with
tool_enable("<tool_name>"). Always use the MCP server's tools directly — never call
the Firecrawl API via curl, raw HTTP, or other bash-based workarounds.

We also ship a ready-made SKILL.md that encodes this protocol — drop it into your skills directory if your agent supports skill loading, and pair it with the snippet above for the strongest guarantee.

🏗️ Local Development

git clone https://github.com/schlemperdev/shantycrawl-mcp.git
cd shantycrawl-mcp
npm install
npm run build
npm start

🤝 Contributing

Issues and PRs are welcome. If you're adding a new lazy-loaded tool, please keep its schema as lean as the core 6 — that's the whole point of this project.

📄 License

MIT © schlemperdev

Available Tools

6 tools
check_crawl_statusC

Check crawl job progress

Inputs: id (required).

crawlB

Crawl a website for page content

Inputs: url (required), limit, maxDepth.

scrapeC

Extract markdown from a URL

Inputs: url (required), formats, waitFor, onlyMainContent.

tool_disableA

Deactivate an advanced tool for this session. Use without arguments to list currently active tools. See tool_enable for the full list of available tools.

Inputs: tool_name.

tool_enableB

Activate an advanced tool for this session. Use without arguments to list available tools.

Inputs: tool_name.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.2/5.0

    Scored across 6 tools

    Disambiguation5/5

    Each tool has a clear distinct purpose: crawl for site crawling, scrape for single URL extraction, search for web search, check_crawl_status for monitoring, and tool_enable/disable for session management. No significant overlap.

    Naming Consistency3/5

    Naming is mixed: some tools are single verbs (crawl, scrape, search), one uses verb_noun with underscore (check_crawl_status), and two use a 'tool_' prefix (tool_enable, tool_disable). This inconsistency could confuse an agent.

    Tool Count5/5

    With 6 tools, the server is well-scoped for a focused crawler/scraper. Every tool serves a necessary function, and the count is neither excessive nor too thin.

    Completeness3/5

    Basic crawling, scraping, and search are covered, but missing operations like cancel crawl or list jobs limit the surface. The meta-tools don't address domain gaps.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      Not graded
      maintenance
      A lightweight and fast MCP server that enables AI agents to efficiently discover and execute tools through progressive disclosure, minimizing context consumption while supporting safe code execution in external environments.
      14 npm
      -
    • A
      license
      B
      quality
      C
      maintenance
      Lightweight MCP server for web scraping, search, and crawling. Uses local trafilatura/DuckDuckGo by default with optional Firecrawl fallback for transport-blocked pages.
      4
      16
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      A lightweight MCP server that auto-discovers tool kits, enabling tools like SQLite queries and web searches via Tavily API.
      3
      -