screaming-frog-mcp

Overview Schema Source Related Servers Discussions Admin

crawl_site

Start a background Screaming Frog crawl to analyze website structure and SEO data, saving results to an internal database for later analysis.

Instructions

Start a background Screaming Frog crawl that saves to SF's internal database.

Args: url: The URL to crawl (e.g. https://example.com) config_file: Optional path to a .seospiderconfig file for crawl settings (including crawl limits) label: Optional label for identifying this crawl (e.g. 'freshgovjobs')

Returns: A crawl_id to use with crawl_status to check progress. The crawl runs in the background - use crawl_status to poll.

Note: To limit the number of URLs crawled, export a .seospiderconfig from the SF GUI with the desired crawl limit, then pass it via config_file.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl_site"
  }
}

Input Schema

Name	Required	Description	Default
`url`	Yes
`config_file`	No
`label`	No

Output Schema

Name	Required	Description	Default
`result`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.2.0Apr 15, 2026

TDQS

A4.3/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It discloses key behavioral traits: the crawl runs in the background, returns a crawl_id for tracking, and requires polling via crawl_status. However, it doesn't mention potential side effects (resource consumption), authentication needs, rate limits, or error conditions that would be important for a background job tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Perfectly structured with clear sections (description, Args, Returns, Note), zero wasted sentences, and front-loaded critical information. Every sentence earns its place by providing essential operational details or practical guidance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (background job with 3 parameters), no annotations, but with output schema (implied by 'Returns' section), the description is mostly complete. It covers purpose, parameters, return value, and operational flow, but lacks information about error handling, performance characteristics, or integration with other tools beyond crawl_status.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description fully compensates by explaining all three parameters: url (URL to crawl with example), config_file (optional .seospiderconfig for settings/limits with practical guidance), and label (optional identifier with example). It adds meaningful context beyond basic schema types, especially for config_file usage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Start a background Screaming Frog crawl') and resource ('saves to SF's internal database'), distinguishing it from siblings like crawl_status (check progress) or export_crawl (export data). It uses precise technical terminology that identifies the exact operation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context about when to use this tool (to initiate a crawl) and references crawl_status for checking progress, but doesn't explicitly contrast when to use this versus alternatives like list_crawls or delete_crawl. The note about config_file usage offers practical guidance but not sibling differentiation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server