crawler-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| MCP_HTTP_PORT | No | Port for HTTP mode | 3001 |
| MCP_TRANSPORT | No | Transport mode: stdio or http | stdio |
| CRAWLER_MAX_CHARS | No | Default cap on returned page content in characters | 20000 |
| CRAWLER_TIMEOUT_MS | No | Per-request timeout in milliseconds | 15000 |
| CRAWLER_USER_AGENT | No | User-Agent for all requests | crawler-mcp/1.0 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| fetch_pageA | Fetch a single web page and return its readable content as Markdown, plain text, or raw HTML. Automatically renders JavaScript-heavy pages with a headless browser when needed. |
| extract_linksB | Extract all hyperlinks from a web page, resolved to absolute URLs. Optionally restrict to links on the same domain. |
| crawl_siteA | Recursively crawl a website starting from a URL, following links up to a maximum depth and page count. Returns a short content summary for each page visited. Stays on the same domain by default. |
| extract_by_selectorB | Extract specific data from a page using a CSS selector. Returns each matching element's text, or an attribute value when |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
Each tool has a clearly distinct purpose: crawl_site handles multi-page crawling with summaries, fetch_page retrieves single-page content in various formats, extract_links focuses on hyperlinks, and extract_by_selector targets specific elements via CSS selectors. There is no overlap or ambiguity between them.
All tool names follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, extract_links). The naming is predictable and intuitive, making it easy for an agent to infer functionality from the name.
With 4 tools, the server covers the core operations of web crawling and extraction without being overly large. The count is slightly on the lower side but still well-scoped for its purpose, and each tool earns its place.
The tool set covers the essential workflow: fetching pages, crawling sites, extracting links, and extracting specific elements. Minor gaps exist, such as the absence of tools for setting request headers or handling authentication, but the core functionality is solid.