Crawl-MCP

Overview Schema Source Related Servers Discussions Admin

multi_url_crawl

Read-only

Crawl up to five URLs per call with wildcard or regex configs, and persist per-URL Markdown plus an index.json when an output directory is set.

Instructions

Multi-URL crawl with pattern-based config. Max 5 URL patterns per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.

Input Schema

Name	Required	Description	Default
`overwrite`	No	Overwrite existing per-URL files inside output_path. Defaults to False.
`output_path`	No	Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null.
`base_timeout`	No	Timeout per URL (default: 30)
`default_config`	No	Default config
`max_concurrent`	No	Has no effect; kept for compatibility. URLs are processed sequentially.
`pattern_matching`	No	Pattern: 'wildcard' or 'regex' (default: wildcard)	wildcard
`url_configurations`	Yes	URL-config mapping (max 5 URLs). Example: {'https://site1.com': {'wait_for_js': true}}
`include_content_in_response`	No	When True (with output_path), keep full markdown/content in each list item. Defaults to False.

Output Schema

Name	Required	Description	Default
`result`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed2 schema fields changedv0.4.2Oct 31, 2026

changedInput schema / properties / max_concurrent / description

Previous value: -"Max concurrent (default: 3)"New value: +"Has no effect; kept for compatibility. URLs are processed sequentially."

changedInput schema / properties / output_path / description

Previous value: -"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null."New value: +"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null."

Addedv0.3.2May 20, 2026
Removedv0.1.6May 1, 2026
First observedv0.1.7Feb 7, 2026

TDQS

B3.4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds genuine behavioral detail beyond the lone readOnlyHint annotation: the return stays a list and successful items gain an output_file key. However, it omits that URLs are processed sequentially and that output_path causes local filesystem writes (which sits in uneasy tension with readOnlyHint), so the side-effect profile is only partly disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the capability and limit, then the persistence behavior. Minor ambiguity from "URL patterns" colliding with the pattern_matching parameter, but nothing is padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema plus fully described parameters carry most of the load, so the description is adequate. Still missing sibling differentiation and the sequential-processing caveat, which are the two things an agent most needs to make a correct call.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so every parameter already carries its own description, including the 5-URL cap and the output_file behavior. The description restates max 5 URLs and the output_file key without adding format or syntax detail, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ("Multi-URL crawl") plus scope ("Max 5 URL patterns per call") and the pattern-based config angle. It doesn't name or distinguish itself from the obvious siblings crawl_url, batch_crawl, or deep_crawl_site, so an agent has to infer the difference from the batch/limit constraint alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage (batch of up to 5 URLs; use output_path if you want files on disk) but never says when to choose this over batch_crawl or crawl_url, nor any prerequisite/exclusion. With four overlapping crawl siblings, the routing decision is not resolved by the description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.