UltraSearch MCP

crawl_site

Crawl a website to get a manifest of page titles and snippets. Full page content is cached for later retrieval via fetch_url.

Instructions

Crawl a site and return a manifest of pages with titles and snippets. Full page content is cached - call fetch_url on any page URL for the full text. Strategy: Firecrawl (JS rendering) → sitemap-first → BFS (if enabled).

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl_site"
  }
}

Input Schema

Name	Required	Description
`url`	Yes	Base URL to crawl
`max_pages`	No	Maximum pages to crawl (default 20, max 100)
`exclude_path`	No	Exclude URLs matching this path prefix (e.g. '/blog')
`include_path`	No	Only include URLs matching this path prefix (e.g. '/docs')
`same_domain_only`	No	Restrict crawl to the same domain (default true)

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Sep 9, 2026

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

There are no annotations, so the description carries the full behavioral burden. It discloses caching behavior, the manifest return, the strategy order (Firecrawl JS rendering, sitemap-first, BFS), and the relationship to fetch_url. It does not mention side effects, rate limits, or authorization, but the core behavioral traits are well covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences with no filler. It front-loads the main purpose and output, then gives the cache/retrieval relationship, then the crawl strategy. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given there is no output schema, the description adequately explains the return value as a manifest of titles and snippets. It also covers what to do for full text and the crawl strategy. It could mention whether the crawl runs synchronously or asynchronously, but the current description is sufficient for most agent invocation scenarios.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% parameter description coverage, so the schema already documents all five parameters. The description adds no new parameter-specific meaning, so the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Crawl a site') and a clear deliverable ('manifest of pages with titles and snippets'). It also differentiates itself from fetch_url by noting that full page content is cached and must be retrieved via fetch_url, so an agent can tell these tools apart.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly implies crawl_site is for site-wide discovery and manifest building, and explicitly directs agents to fetch_url when full page text is needed. It lacks explicit exclusions for sibling search tools, but the context is strong enough for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server