webcrawl-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| USER_AGENT | No | HTTP User-Agent | Mozilla/5.0 (compatible; WebcrawlMCP/1.0; …) |
| POLITE_MODE | No | On a 429 with Retry-After, retry once after the indicated wait. | true |
| REQUEST_TIMEOUT | No | Seconds before request timeout | 30 |
| FIRECRAWL_API_KEY | No | If set, enables Firecrawl fallback for low-quality extractions. Leave unset for a fully free setup. | |
| FIRECRAWL_API_URL | No | Firecrawl endpoint | https://api.firecrawl.dev/v1 |
| FALLBACK_ON_TRANSPORT_ERROR | No | If true and FIRECRAWL_API_KEY is set, route bot-blocked statuses to Firecrawl instead of raising. | false |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
| logging | {} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
| extensions | {
"io.modelcontextprotocol/ui": {}
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| webcrawl_scrapeA | Fetch a URL and extract main content as markdown. |
| webcrawl_searchC | Search the web using DuckDuckGo. |
| webcrawl_mapA | Discover URLs on a website. Fetches the given URL and extracts all same-domain links. |
| webcrawl_crawlA | Crawl multiple pages starting from a URL. Uses BFS to discover and fetch pages up to max_depth links away. Respects rate limiting between requests. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
Tools are mostly distinct: crawl is multi-page BFS, map is single-page link discovery, scrape extracts content, search is external web search. Some overlap between crawl and map (both find links), but descriptions clarify the difference.
All tools use a consistent 'webcrawl_' prefix followed by a clear verb (crawl, map, scrape, search). The pattern is uniform and each verb accurately describes the action.
Four tools is slightly lean for a web crawling server, but each serves a clear purpose (crawl, map, scrape, search). The count is appropriate for a minimal but functional set.
Covers core web operations: link discovery, crawling, content extraction, and search. Missing features like robots.txt handling, sitemap parsing, or raw HTML retrieval, which are notable gaps for advanced use cases.