Skip to main content
Glama
spidra-io

spidra-mcp-server

Official
by spidra-io

spidra_crawl_report

Read-only

Explain a finished crawl to see how instructions were interpreted, how many pages were read, and which links were skipped by reason. Use it to debug missing pages and adjust crawl settings.

Instructions

Explain a finished crawl: how it read the instruction, how many pages it read only to find the wanted ones (free), and every link it didn't follow, counted by reason. Pass "reason" to list the URLs for one reason, each with the page it was found on.

Use it when a page you expected is missing, then adjust crawlInstruction, includePaths or maxPages and crawl again. Reasons: not_relevant, outside_domain, include_pattern, exclude_pattern, file_type, other_version_or_language, duplicate_page, redirected_off_site, robots_txt, action_link. The report is empty while the crawl runs.

Input Schema

NameRequiredDescriptionDefault
jobIdYesThe crawl's jobId
limitNoSkipped URLs to return (default 200)
offsetNoSkipped URLs to skip, for paging
reasonNoList skipped URLs for this reason only, e.g. "not_relevant"

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.7.0

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnlyHint=true, openWorldHint=false), yet the description adds real context: the report is empty mid-crawl, and pages read only to find wanted ones are free. It doesn't mention cost/limits of listing many skipped URLs or auth needs, so not a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then usage, then the reason vocabulary — a sensible ordering. The inline reason list and parenthetical '(free)' consume space but earn it since no enum exists in the schema; still slightly denser than strictly necessary.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description carries the return-value burden and does so: counts by reason by default, and per-URL detail with source page when 'reason' is passed. Combined with the mid-crawl emptiness caveat, an agent has everything needed to call and interpret it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds meaning the schema lacks: the enumerated valid values for 'reason' (not_relevant, outside_domain, robots_txt, etc.) and the fact that each listed URL comes with the page it was found on. That is genuine value beyond the schema's one-line 'List skipped URLs for this reason only'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — explain a finished crawl — and enumerates the three things the report contains (instruction interpretation, pages read to find wanted ones, skipped links by reason). This is clearly distinguishable from spidra_get_job (status) or spidra_crawl_pages (content), so an agent can route without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit trigger ('Use it when a page you expected is missing') plus the remediation loop (adjust crawlInstruction, includePaths or maxPages and crawl again), and a when-not condition ('The report is empty while the crawl runs'). No sibling tool is named as an alternative, which keeps it short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.