Crawl-MCP

Overview Schema Source Related Servers Discussions Admin

deep_crawl_site

Read-only

Crawls multiple pages from a site with configurable depth and page limits, saving per-URL Markdown files plus index.json to a directory for structured, token-efficient site content extraction.

Instructions

Crawl multiple pages from a site with configurable depth. Use output_path (directory) to persist per-URL markdown files + index.json; the response is then slimmed to metadata only.

Input Schema

Name	Required	Description	Default
`url`	Yes	Starting URL
`max_depth`	No	Link depth, 1-2 (default: 2). Values outside the range return an error.
`max_pages`	No	Max pages, 1-10 (default: 5). Values outside the range return an error.
`overwrite`	No	Overwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error).
`output_path`	No	Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). When set, the response is slimmed to metadata+file paths. Failed items (success=False) are NOT written as .md but still recorded in index.json with file=null.
`url_pattern`	No	URL filter pattern
`base_timeout`	No	Timeout per page
`extract_media`	No	Extract media
`crawl_strategy`	No	'bfs'\|'dfs'\|'best_first'	bfs
`score_threshold`	No	Min relevance 0-1
`include_external`	No	Follow external links
`include_content_in_response`	No	When True (with output_path set), also include per-page content/markdown in the response items. Defaults to False so the response stays token-efficient.

Output Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

Changed2 schema fields changedv0.4.2Oct 31, 2026

changedInput schema / properties / max_depth / description

Previous value: -"Link depth (1-2)"New value: +"Link depth, 1-2 (default: 2). Values outside the range return an error."

changedInput schema / properties / max_pages / description

Previous value: -"Max pages (max: 10)"New value: +"Max pages, 1-10 (default: 5). Values outside the range return an error."

Addedv0.3.2May 20, 2026
Removedv0.1.6May 1, 2026
First observedv0.1.7Feb 7, 2026

TDQS

A3.8/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds real behavior the annotations don't convey: setting output_path persists per-URL markdown plus index.json, and the response is then slimmed to metadata only. That is exactly the side effect an agent needs to know before opting into file output. The only gap is that it doesn't reconcile this file-writing behavior with the readOnlyHint annotation, which reads as local-artifact generation rather than a remote-state mutation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, zero padding, with the core action front-loaded and the output_path persistence caveat immediately after. Every clause carries information an agent would otherwise have to infer.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Return values need no explanation because an output schema exists, and the schema fully covers the 12 parameters. However, for a 12-parameter tool sitting in a crowded crawl family, the description leaves the sibling-routing decision (deep_crawl_site vs batch_crawl vs multi_url_crawl vs crawl_url) entirely to the agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all 12 parameters including ranges and defaults. The description nonetheless reinforces the highest-impact one — output_path as a directory with a response-shape consequence — adding value beyond the schema's own wording. Baseline 3 is exceeded slightly for that targeted emphasis.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Crawl multiple pages from a site with configurable depth'), and 'multiple pages' implicitly contrasts with the single-page crawl_url sibling. It does not explicitly differentiate itself from batch_crawl, multi_url_crawl, or search_and_crawl, so an agent still has to disambiguate among the crawl family by name alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: crawl a site (not a single URL) to a given depth. There is no explicit 'use this when you need X instead of batch_crawl/multi_url_crawl' guidance, and no statement of prerequisites or when the tool is a poor fit (e.g., across many domains).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.