shantycrawl-mcp
The shantycrawl-mcp server is a token-efficient Firecrawl interface that loads 6 core tools upfront and lets you dynamically enable 22 advanced tools on demand.
Core Tools (Always Available)
scrape— Extract clean markdown content from any URL (with options for output format, JS wait time, and main-content filtering)crawl— Recursively crawl a website with controls for max pages and depthsearch— Execute web searches and retrieve resultscheck_crawl_status— Poll the progress of an async crawl job by IDtool_enable— Dynamically activate any advanced tool mid-session; call without arguments to list all available toolstool_disable— Unload an active advanced tool to free up context; call without arguments to list currently active tools
Advanced Tools (Lazy-Loaded via tool_enable)
Discovery:
map,extract,parse— site mapping and structured data extractionAI Agent:
agent,agent_status— autonomous browsing/research agentsBrowser Interaction:
interact,interact_stop— programmatic browser controlResearch & Academic:
research_search_papers,research_inspect_paper,research_read_paper,research_related_papers,research_search_githubMonitoring:
monitor_create,monitor_check,monitor_checks,monitor_delete,monitor_get,monitor_list,monitor_run,monitor_updateFeedback:
search_feedback,feedback
This design saves ~18,000 tokens compared to loading all tools upfront.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@shantycrawl-mcp scrape https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Why ShantyCrawl MCP
Most full-featured MCP servers register every tool on every session — including the ones you'll rarely touch — with full JSON schemas re-sent on each tool-list fetch. ShantyCrawl loads 6 core tools up front and lazy-loads the rest on demand, which measured out to ~18,000 fewer tokens on a fresh session in our testing:
|
| |
Tools on session start | 28 | 6 |
Initial schema footprint | Full schemas for all 28 tools | Minimal schemas for 6 tools |
Advanced tools (research, monitoring, agent, etc.) | Always loaded | Loaded on demand via |
Measured token cost, fresh session | Baseline | ~18k tokens lighter |
Runtime dependencies | — | Zero third-party wrappers, native |
Is This For You?
✅ Good fit — you use Firecrawl mostly for
scrape/crawl/search, run long or multi-tool agent sessions, or are tight on context budget.⚪ Less critical — you're already using
tool_enable-style dynamic loading elsewhere, or you live almost entirely in the monitoring/research toolset and rarely call the core 6 — in that case the savings are smaller, since you'll be loading most of the schema anyway.
Related MCP server: webcrawl-mcp
🚀 Quick Start
Run instantly via npx — no installation required:
FIRECRAWL_API_URL=http://localhost:3002 npx shantycrawl-mcpEnvironment variables:
Variable | Required | Default | Description |
| No |
| URL of your Firecrawl instance (self-hosted or cloud) |
| Only for Firecrawl Cloud | — | API key, required if |
Requirements: Node.js 18+ and a reachable Firecrawl instance (self-hosted or cloud).
⚙️ Configuration
Claude Desktop
Add this to your claude_desktop_config.json:
{
"mcpServers": {
"shantycrawl": {
"command": "npx",
"args": ["shantycrawl-mcp"],
"env": {
"FIRECRAWL_API_URL": "http://localhost:3002"
}
}
}
}OpenCode
Add this to your workspace settings:
{
"mcp": {
"shantycrawl": {
"type": "local",
"command": ["npx", "shantycrawl-mcp"],
"environment": {
"FIRECRAWL_API_URL": "http://localhost:3002"
}
}
}
}🛠️ Tool Architecture
Always Available (6 Core Tools)
Tool | Description |
| Extract clean markdown from any URL |
| Crawl a target website recursively |
| Execute web searches via Firecrawl |
| Dynamically inject an advanced tool into the active session |
| Unload an advanced tool to free up context |
| Check the status of an async |
Lazy-Loaded Tools (22 Advanced)
Activate any tool instantly during a session — e.g. tool_enable map unblocks the mapping capability.
Category | Tools |
Discovery |
|
AI Agent |
|
Browser Interaction |
|
Research & Academic |
|
Monitoring |
|
Feedback |
|
🧠 How Dynamic Loading Works
The client sees a minimal 6-tool schema on connect.
When the AI needs an advanced capability, it calls
tool_enable <tool_name>.The server fires a
notifications/tools/list_changedevent, and the client's available tools update instantly — no reconnect required.Call
tool_disable <tool_name>once you're done to free the context back up.
⚠️ Known Issues
Some agents don't call tool_enable on their own. A compelling tool description isn't always enough — if another skill or system prompt has already taught the agent a different path to the same outcome (e.g. calling the Firecrawl API directly via curl/raw HTTP), it can follow that instead of discovering tool_enable. This shows up most with coding agents that have built-in or third-party Firecrawl skills already loaded.
Fix: add an explicit instruction to your agent's system prompt or AGENTS.md/CLAUDE.md:
This project uses the shantycrawl-mcp server. Advanced tools (map, extract, parse,
agent, interact, research_*, monitor_*) are not available until activated with
tool_enable("<tool_name>"). Always use the MCP server's tools directly — never call
the Firecrawl API via curl, raw HTTP, or other bash-based workarounds.We also ship a ready-made SKILL.md that encodes this protocol — drop it into your skills directory if your agent supports skill loading, and pair it with the snippet above for the strongest guarantee.
🏗️ Local Development
git clone https://github.com/schlemperdev/shantycrawl-mcp.git
cd shantycrawl-mcp
npm install
npm run build
npm start🤝 Contributing
Issues and PRs are welcome. If you're adding a new lazy-loaded tool, please keep its schema as lean as the core 6 — that's the whole point of this project.
📄 License
MIT © schlemperdev
Available Tools
6 toolscheck_crawl_statusC
Check crawl job progress
Inputs: id (required).
crawlB
Crawl a website for page content
Inputs: url (required), limit, maxDepth.
scrapeC
Extract markdown from a URL
Inputs: url (required), formats, waitFor, onlyMainContent.
searchC
Search the web
Inputs: query (required), limit.
tool_disableA
Deactivate an advanced tool for this session. Use without arguments to list currently active tools. See tool_enable for the full list of available tools.
Inputs: tool_name.
tool_enableB
Activate an advanced tool for this session. Use without arguments to list available tools.
Inputs: tool_name.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Each tool has a clear distinct purpose: crawl for site crawling, scrape for single URL extraction, search for web search, check_crawl_status for monitoring, and tool_enable/disable for session management. No significant overlap.
Naming is mixed: some tools are single verbs (crawl, scrape, search), one uses verb_noun with underscore (check_crawl_status), and two use a 'tool_' prefix (tool_enable, tool_disable). This inconsistency could confuse an agent.
With 6 tools, the server is well-scoped for a focused crawler/scraper. Every tool serves a necessary function, and the count is neither excessive nor too thin.
Basic crawling, scraping, and search are covered, but missing operations like cancel crawl or list jobs limit the surface. The meta-tools don't address domain gaps.
Maintenance
Related MCP Connectors
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Unified 15-tool SEO, SERP, clean Markdown & GEO (LLM citation Share-of-Voice) MCP server.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
Related MCP Servers
- AlicenseNot gradedqualityNot gradedmaintenanceA lightweight and fast MCP server that enables AI agents to efficiently discover and execute tools through progressive disclosure, minimizing context consumption while supporting safe code execution in external environments.14 npm-
- AlicenseBqualityCmaintenanceLightweight MCP server for web scraping, search, and crawling. Uses local trafilatura/DuckDuckGo by default with optional Firecrawl fallback for transport-blocked pages.416MIT
- FlicenseNot gradedqualityDmaintenanceA lightweight MCP server that auto-discovers tool kits, enabling tools like SQLite queries and web searches via Tavily API.3-
- AlicenseNot gradedqualityDmaintenanceA unified MCP server aggregating 15 web search and extraction tools across 5 providers (Jina, Tavily, Exa, Firecrawl, Bocha) with automatic API key validation and plugin architecture.MIT