The shantycrawl-mcp server is a token-efficient Firecrawl interface that loads 6 core tools upfront and lets you dynamically enable 22 advanced tools on demand.
Core Tools (Always Available)
scrape — Extract clean markdown content from any URL (with options for output format, JS wait time, and main-content filtering)
crawl — Recursively crawl a website with controls for max pages and depth
search — Execute web searches and retrieve results
check_crawl_status — Poll the progress of an async crawl job by ID
tool_enable — Dynamically activate any advanced tool mid-session; call without arguments to list all available tools
tool_disable — Unload an active advanced tool to free up context; call without arguments to list currently active tools
Advanced Tools (Lazy-Loaded via tool_enable)
Discovery: map, extract, parse — site mapping and structured data extraction
AI Agent: agent, agent_status — autonomous browsing/research agents
Browser Interaction: interact, interact_stop — programmatic browser control
Research & Academic: research_search_papers, research_inspect_paper, research_read_paper, research_related_papers, research_search_github
Monitoring: monitor_create, monitor_check, monitor_checks, monitor_delete, monitor_get, monitor_list, monitor_run, monitor_update
Feedback: search_feedback, feedback
This design saves ~18,000 tokens compared to loading all tools upfront.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@shantycrawl-mcp scrape https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Most full-featured MCP servers register every tool on every session — including the ones you'll rarely touch — with full JSON schemas re-sent on each tool-list fetch. ShantyCrawl loads 6 core tools up front and lazy-loads the rest on demand, which measured out to ~18,000 fewer tokens on a fresh session in our testing:
|
| |
Tools on session start | 28 | 6 |
Initial schema footprint | Full schemas for all 28 tools | Minimal schemas for 6 tools |
Advanced tools (research, monitoring, agent, etc.) | Always loaded | Loaded on demand via |
Measured token cost, fresh session | Baseline | ~18k tokens lighter |
Runtime dependencies | — | Zero third-party wrappers, native |
✅ Good fit — you use Firecrawl mostly for scrape/crawl/search, run long or multi-tool agent sessions, or are tight on context budget.
⚪ Less critical — you're already using tool_enable-style dynamic loading elsewhere, or you live almost entirely in the monitoring/research toolset and rarely call the core 6 — in that case the savings are smaller, since you'll be loading most of the schema anyway.
Related MCP server: webcrawl-mcp
Run instantly via npx — no installation required:
FIRECRAWL_API_URL=http://localhost:3002 npx shantycrawl-mcpEnvironment variables:
Variable | Required | Default | Description |
| No |
| URL of your Firecrawl instance (self-hosted or cloud) |
| Only for Firecrawl Cloud | — | API key, required if |
Requirements: Node.js 18+ and a reachable Firecrawl instance (self-hosted or cloud).
Add this to your claude_desktop_config.json:
{
"mcpServers": {
"shantycrawl": {
"command": "npx",
"args": ["shantycrawl-mcp"],
"env": {
"FIRECRAWL_API_URL": "http://localhost:3002"
}
}
}
}Add this to your workspace settings:
{
"mcp": {
"shantycrawl": {
"type": "local",
"command": ["npx", "shantycrawl-mcp"],
"environment": {
"FIRECRAWL_API_URL": "http://localhost:3002"
}
}
}
}Tool | Description |
| Extract clean markdown from any URL |
| Crawl a target website recursively |
| Execute web searches via Firecrawl |
| Dynamically inject an advanced tool into the active session |
| Unload an advanced tool to free up context |
| Check the status of an async |
Activate any tool instantly during a session — e.g. tool_enable map unblocks the mapping capability.
Category | Tools |
Discovery |
|
AI Agent |
|
Browser Interaction |
|
Research & Academic |
|
Monitoring |
|
Feedback |
|
The client sees a minimal 6-tool schema on connect.
When the AI needs an advanced capability, it calls tool_enable <tool_name>.
The server fires a notifications/tools/list_changed event, and the client's available tools update instantly — no reconnect required.
Call tool_disable <tool_name> once you're done to free the context back up.
Some agents don't call tool_enable on their own. A compelling tool description isn't always enough — if another skill or system prompt has already taught the agent a different path to the same outcome (e.g. calling the Firecrawl API directly via curl/raw HTTP), it can follow that instead of discovering tool_enable. This shows up most with coding agents that have built-in or third-party Firecrawl skills already loaded.
Fix: add an explicit instruction to your agent's system prompt or AGENTS.md/CLAUDE.md:
This project uses the shantycrawl-mcp server. Advanced tools (map, extract, parse,
agent, interact, research_*, monitor_*) are not available until activated with
tool_enable("<tool_name>"). Always use the MCP server's tools directly — never call
the Firecrawl API via curl, raw HTTP, or other bash-based workarounds.We also ship a ready-made SKILL.md that encodes this protocol — drop it into your skills directory if your agent supports skill loading, and pair it with the snippet above for the strongest guarantee.
git clone https://github.com/schlemperdev/shantycrawl-mcp.git
cd shantycrawl-mcp
npm install
npm run build
npm startIssues and PRs are welcome. If you're adding a new lazy-loaded tool, please keep its schema as lean as the core 6 — that's the whole point of this project.
MIT © schlemperdev
Check crawl job progress
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Crawl job ID |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It only says 'check progress' without disclosing whether the operation is idempotent, safe (read-only), or what the output format is (e.g., status, percentage, errors). This is insufficient for an agent to understand behavioral implications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, concise and front-loaded. However, it could be slightly more informative without becoming verbose. It earns its place but leaves room for improvement.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of output schema and annotations, the description should explain the return value or result of checking progress. It does not mention what fields or statuses are returned, making it incomplete for an agent to interpret the response. The tool's simplicity (1 param) does not excuse this omission.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with one parameter 'id' described as 'Crawl job ID'. The description adds no additional meaning beyond the schema. Baseline score of 3 is appropriate as the schema already documents the parameter adequately.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Check crawl job progress' clearly states the verb and resource, indicating the tool monitors an ongoing crawl job. It distinguishes from sibling tools like 'crawl' (initiates), 'scrape' (one-time), and 'search' (queries). However, it could be more specific about what 'progress' entails.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. For example, it does not say to use after starting a crawl with the 'crawl' tool or to avoid for completed jobs. The description lacks context for appropriate invocation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Crawl a website for page content
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL | |
| limit | No | Max pages | |
| maxDepth | No | Max depth |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and description does not disclose behavioral traits such as rate limits, robots.txt adherence, page count limits, or depth behavior. Essentially opaque aside from basic function.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Very concise single sentence, but lacks any structure like bullet points or front-loading of key information. Could be improved without increasing length significantly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Description does not explain return format or content structure, nor does it clarify boundary behaviors (e.g., what happens when maxDepth or limit is omitted). Incomplete for a tool with no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers all parameters with brief descriptions (url: 'Start URL', maxDepth: 'Max depth', limit: 'Max pages'). Description adds no additional meaning beyond schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'Crawl', resource 'a website', and outcome 'page content'. It effectively distinguishes from siblings like 'scrape' and 'search' which imply different actions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use crawl vs alternatives like scrape or search. Missing context about appropriate use cases or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract markdown from a URL
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| formats | No | Output format(s) | |
| waitFor | No | JS wait (ms) | |
| onlyMainContent | No | Main content only |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only states the basic function without disclosing behavioral traits such as JavaScript execution (implied by waitFor parameter), rate limits, or error handling. The description adds minimal transparency beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence: 'Extract markdown from a URL'. It is front-loaded and contains no redundant information, making it easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 4 parameters, no output schema, and no annotations, the description is insufficient. It does not explain the effects of optional parameters (formats, onlyMainContent, waitFor) or typical use cases, leaving the agent without enough context to use the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description does not add meaning beyond the parameter names and schema descriptions; it repeats the same level of detail without clarifying valid formats or the purpose of waitFor.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Extract markdown from a URL' clearly states a specific action and resource. It is not a tautology and conveys the core function. However, it does not differentiate from sibling tools like 'crawl', which may extract content from multiple URLs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives (e.g., 'crawl' for multi-page extraction, 'search' for querying). It lacks any when-to-use or when-not-to-use context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search the web
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max results | |
| query | Yes | Search query |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and description does not disclose any behavioral traits such as rate limits, result format, or safety. The description adds minimal value beyond the name.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Very concise at one sentence, but could be more structured. No additional information is provided that would help an agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and simple tool, description lacks details on return values, pagination, or any constraints. Incomplete for effective selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. Description adds no additional meaning beyond the schema's 'Search query' and 'Max results'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description 'Search the web' states a verb and resource, but is too vague. It doesn't differentiate from sibling tools like 'crawl' or 'scrape'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. Missing context for when search is appropriate compared to crawl or scrape.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deactivate an advanced tool for this session. Use without arguments to list currently active tools. See tool_enable for the full list of available tools.
| Name | Required | Description | Default |
|---|---|---|---|
| tool_name | No | Tool name (omit to list active tools) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description discloses deactivation for session and listing behavior, but does not cover permissions, reversibility, or side effects of deactivation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, each adding value, front-loaded with main action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequately covers two modes and references sibling for more depth; missing details on confirmation or immediate effect, but minor given tool simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with one parameter; description adds minimal extra meaning ('omit to list active tools') which is already in schema description. Baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states verb (deactivate), resource (advanced tool), and scope (session). Distinguishes from sibling tool_enable by referencing it for full list.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly describes two usage modes (list active tools or deactivate) and references tool_enable for alternatives, but lacks explicit 'when not to use' for other siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Activate an advanced tool for this session. Use without arguments to list available tools.
| Name | Required | Description | Default |
|---|---|---|---|
| tool_name | No | Tool name (omit to list available tools) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility. It discloses activation and listing behavior but omits important details like session scope, side effects, permissions required, or what 'activate' entails (e.g., success response, error states). Minimal behavioral disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the primary action, no wasted words. Every sentence provides value: action and usage variant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and a single parameter, the description should clarify what happens upon activation (e.g., confirmation, tool enabled for the session). It only implies listing behavior but does not describe activation outcome or return value, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with the description 'Tool name (omit to list available tools)'. The tool description adds 'Use without arguments to list available tools', which reinforces but does not add significant new meaning beyond the schema. Baseline 3 appropriate due to high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool activates an advanced tool for the session, using 'Activate' as the verb and specifying 'advanced tool'. It differentiates from the sibling 'tool_disable' by implication, though not explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides clear usage guidance: 'Use without arguments to list available tools'. However, lacks guidance on when to use this tool versus the sibling 'tool_disable' or other alternatives, and no explicit 'when not to use'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Each tool has a clear distinct purpose: crawl for site crawling, scrape for single URL extraction, search for web search, check_crawl_status for monitoring, and tool_enable/disable for session management. No significant overlap.
Naming is mixed: some tools are single verbs (crawl, scrape, search), one uses verb_noun with underscore (check_crawl_status), and two use a 'tool_' prefix (tool_enable, tool_disable). This inconsistency could confuse an agent.
With 6 tools, the server is well-scoped for a focused crawler/scraper. Every tool serves a necessary function, and the count is neither excessive nor too thin.
Basic crawling, scraping, and search are covered, but missing operations like cancel crawl or list jobs limit the surface. The meta-tools don't address domain gaps.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.
The Remote MCP server acts as a standardized bridge between LLM applications (like Claude, ChatGPT, and Cursor) and external services, enabling AI agents to access external tools and resources. Its primary capability is providing a centralized search tool to discover other MCP servers and their respective tools. Unlike local implementations, it runs remotely with OAuth authentication and permission controls for security.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/schlemperdev/shantycrawl-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
