webcrawl-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| USER_AGENT | No | HTTP User-Agent | Mozilla/5.0 (compatible; WebcrawlMCP/1.0; …) |
| POLITE_MODE | No | On a 429 with Retry-After, retry once after the indicated wait. | true |
| REQUEST_TIMEOUT | No | Seconds before request timeout | 30 |
| FIRECRAWL_API_KEY | No | If set, enables Firecrawl fallback for low-quality extractions. Leave unset for a fully free setup. | |
| FIRECRAWL_API_URL | No | Firecrawl endpoint | https://api.firecrawl.dev/v1 |
| FALLBACK_ON_TRANSPORT_ERROR | No | If true and FIRECRAWL_API_KEY is set, route bot-blocked statuses to Firecrawl instead of raising. | false |
Capabilities
Features and capabilities supported by this server
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
| logging | {} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
| extensions | {
"io.modelcontextprotocol/ui": {}
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| webcrawl_scrapeA | Fetch a URL and extract main content as markdown. |
| webcrawl_searchC | Search the web using DuckDuckGo. |
| webcrawl_mapA | Discover URLs on a website. Fetches the given URL and extracts all same-domain links. |
| webcrawl_crawlA | Crawl multiple pages starting from a URL. Uses BFS to discover and fetch pages up to max_depth links away. Respects rate limiting between requests. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
Latest Blog Posts
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/andyliszewski/webcrawl-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server