Skip to main content
Glama
Sign In
Crawl-MCP

Crawl-MCP

by walksoda
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Hybrid

batch_crawl

Crawl up to three URLs in one call with fallback support, optionally saving per-URL Markdown files and an index.json for organized batch content extraction.

Instructions

Crawl multiple URLs with fallback. Max 3 URLs per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.

Input Schema

NameRequiredDescriptionDefault
urlsYesURLs to crawl (max 3)
overwriteNoOverwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error, returned as a single-element list).
output_pathNoAbsolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null.
wait_for_jsNoWait for JS (default: False)
base_timeoutNoTimeout per URL (default: 30)
extract_mediaNoExtract media (default: False)
generate_markdownNoGenerate markdown (default: True)
include_content_in_responseNoWhen True (with output_path), keep full markdown/content in each list item. Defaults to False so the response stays token-efficient.

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.4.2
    • changedInput schema / properties / output_path / description
      Previous value: -"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null."New value: +"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null."
  2. Addedv0.3.2
  3. Removedv0.1.6
  4. First observedv0.1.7

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does meaningful work: it discloses the fallback behavior, the 3-URL cap, that return shape stays a list, and that successes gain an output_file key. It omits failure/partial-failure semantics and any auth or rate-limit context, which keeps it below 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with the core action and limit, then the persistence/return-shape contract. No filler and no restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists and the 8 parameters are fully documented, so return values need not be re-explained; the description covers the limits and the persistence contract well. What's missing is routing guidance against the many sibling crawl/search tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the schema descriptions are unusually thorough (overwrite error semantics, output_path rejection rules, include_content_in_response token tradeoff). The description restates output_path persistence and adds the output_file key detail, but adds little beyond what the schema already documents.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Crawl multiple URLs') plus a distinguishing trait ('with fallback') and a hard limit ('Max 3 URLs per call'). It does not name the nearest siblings (multi_url_crawl, batch_search_web, deep_crawl_site), so the agent must still infer which batch tool to pick.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an actionable usage cue for output_path ('Use output_path (directory) to persist...') and a size constraint, which implies the batch-of-few use case. But there is no explicit when-to-use/when-not versus multi_url_crawl or deep_crawl_site, and no stated prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools