This MCP server enables headless interaction with Screaming Frog SEO Spider — letting AI assistants run crawls, export data, and analyze SEO results without the GUI open. It provides nine tools:
sf_check — Verify Screaming Frog is installed, accessible, and licensed (useful as a pre-flight check).
crawl_site — Start a background headless crawl of any URL, optionally with a .seospiderconfig file and a label. Returns a crawl_id for polling.
crawl_status — Monitor the progress or completion status of a crawl by crawl_id.
list_crawls — List all crawls stored in Screaming Frog's database, including their IDs, names, and sizes.
export_crawl — Load a saved crawl by Database ID and export it as CSV files, supporting specific tabs (e.g. Internal Pages, Response Codes, Page Titles, Images, Canonicals), bulk exports (e.g. All Inlinks/Outlinks), and saved reports (e.g. Crawl Overview).
read_crawl_data — Read and filter exported CSV files with pagination, column selection, and flexible filtering (substring, exact match, or regex) to return only relevant rows.
aggregate_crawl_data — Get counts and group-by breakdowns (e.g. "how many 404s?", "status code distribution") computed server-side without loading all rows.
delete_crawl — Permanently delete a saved crawl from the database to free up disk space.
storage_summary — View total and per-crawl disk usage of Screaming Frog's crawl storage.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Screaming Frog SEO Spider MCP Server List my saved crawls"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A headless MCP (Model Context Protocol) server for Screaming Frog SEO Spider. It drives the SF command line and saved crawl database directly, so Claude (or any MCP-compatible client) can run crawls, export crawl data, and analyze the results with the Screaming Frog GUI closed: on your laptop, on a server, or inside scheduled audits and CI pipelines.
This is a community project, not affiliated with Screaming Frog. Since SEO Spider v24 there is also an official MCP built into the app. The two work differently and solve different problems.
Screaming Frog shipped an official MCP server in SEO Spider v24. It's substantial: around 29 tools covering crawl control (start, pause, resume, progress), reports and bulk exports with field selection, URL-level inspection, screenshots, embeddings exports, and optionally a Node.js script runner with npm and filesystem read/write tools. It runs in two modes, either a Streamable HTTP server inside the open app, or a STDIO mode where the MCP client launches the Spider itself, headless. Setup is documented for Claude Desktop and LM Studio.
If you want maximum capability in an interactive session (visualizations, crawl comparison, screenshots, scripted post-processing of exports), use the official MCP. It does far more, and it's maintained by the vendor.
This server makes a different trade: it's a small, deliberately limited wrapper around SF's CLI and the saved crawl database, built for runs where nobody is watching.
Locked-down by design. Nine read-and-export tools, nothing else. No script runner, no npm install, no filesystem write access. The official MCP offers all three, and its own docs note that enabling the Node runtime "allows the execution of arbitrary code on your system" and should only be granted to a fully trusted client. There's also an SF_ALLOWED_DOMAINS allowlist to restrict what an agent is able to crawl. When an agent runs unattended on a schedule, a tool surface this small is a feature.
Installs anywhere, plainly. A pip/uv-installable Python package with a one-line stdio config on any MCP client (Claude Code, Cursor, whatever) on macOS, Linux, or Windows. The official STDIO mode ships as a Claude Desktop extension (.mcpb); the HTTP mode means opening the app and starting the server from its settings.
Light process model. Screaming Frog only runs while a tool actually needs it. The official server is the Spider application running for the whole session, whichever mode you pick.
A few tools the official set doesn't have: aggregate_crawl_data for counts and distributions computed server-side (the official path to "how many 404s" is a full export, or a Node script), delete_crawl and storage_summary for cleaning up SF's crawl database (their sf_clear_crawl clears a paused crawl, it doesn't manage stored ones), regex filtering across any column of any export via read_crawl_data, and sf_check pre-flight diagnostics that catch license problems and GUI database locks before you waste a crawl.
What it feels like from chat. The official server is stateful: you ask it to load a crawl by ID, the Spider holds it in memory for the session, and follow-up questions answer in under a second. The cost is that the session owns SF's database the whole conversation, and exports come back as full inline dumps unless the model saves files and writes Node scripts to slice them (the approach their own docs recommend for staying inside the context window). This server is stateless: each export spawns the SF CLI fresh, so the first answer on a crawl takes longer, but you can just ask ("list my crawls, export the latest one") without managing IDs or sessions, reads return only the filtered rows you asked for, and the database is released between calls. For "show me the 404s on a 100k-URL crawl", the difference is the whole export in context versus a page of matching rows.
Typical split: crawl interactively in the GUI with your full config, close it, and let this server handle the unattended side. That covers scheduled audits, CI checks, and agents querying the saved data. Both need a licensed Screaming Frog install on the same machine; neither is a cloud crawler. Note that this server requires the GUI to be closed (SF's database allows one process at a time).
Related MCP server: Screaming Frog MCP Server
The Pre-Launch Website Audit skill for Claude Code uses this MCP server for its technical SEO and on-page audits, site-wide crawl data, custom extractions, bulk analysis across all URLs. The skill runs 5 coordinated sub-audits and works without SF (bash fallbacks), but Screaming Frog is the biggest upgrade for crawl-dependent checks.
Screaming Frog SEO Spider installed on your machine (tested with v23.x and v24.x, should work with v16+). Download from: https://www.screamingfrog.co.uk/seo-spider/
A valid Screaming Frog license. The free version has a 500-URL crawl limit. Most MCP features (headless CLI, saving/loading crawls, exports) require a paid license.
Python 3.10+
Screaming Frog uses an internal database that can only be accessed by one process at a time. This means:
You must close the Screaming Frog GUI before the MCP server can access crawl data.
The typical workflow is:
Run your crawl — either through the SF GUI (with all your custom settings, filters, etc.) or via the MCP crawl_site tool.
Close the Screaming Frog GUI — the GUI locks the crawl database. The MCP server's headless CLI cannot read or export data while the GUI is running.
Use the MCP tools — once the GUI is closed, you can list crawls, export data, read CSVs, and more through your AI assistant.
If you forget to close the GUI, the server will detect it and show a clear error message telling you to quit SF first.
Install as a persistent uv tool so the server starts instantly:
uv tool install screaming-frog-mcpThis puts a screaming-frog-mcp executable on your PATH (typically ~/.local/bin/screaming-frog-mcp). Update later with uv tool upgrade screaming-frog-mcp.
Alternatively, install with pip:
pip install screaming-frog-mcpAvoid
uvx screaming-frog-mcpin MCP client configs.uvxresolves and downloads the package environment at launch. On a cold cache this can exceed the client's 60-second initialize timeout, causing intermittent "Could not attach to MCP server" errors. A persistent install never touches the network at startup.
git clone https://github.com/bzsasson/screaming-frog-mcp.git
cd screaming-frog-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtThe default Screaming Frog CLI path works for macOS. If you're on Linux or Windows, set the SF_CLI_PATH environment variable:
OS | Default Path |
macOS |
|
Linux |
|
Windows |
|
If you cloned the repo, copy .env.example to .env and edit it.
If installed via uv tool install or pip:
{
"mcpServers": {
"screaming-frog": {
"command": "/path/to/screaming-frog-mcp",
"args": [],
"env": {
"SF_CLI_PATH": "/path/to/ScreamingFrogSEOSpiderLauncher"
}
}
}
}Find the executable path with which screaming-frog-mcp (e.g. ~/.local/bin/screaming-frog-mcp for uv tool installs). Use the full absolute path, since GUI apps don't inherit your shell's PATH.
If cloned from source:
{
"mcpServers": {
"screaming-frog": {
"command": "/path/to/screaming-frog-mcp/.venv/bin/python",
"args": ["/path/to/screaming-frog-mcp/sf_mcp.py"]
}
}
}Add to your Claude Desktop config (claude_desktop_config.json), using the same absolute executable path:
{
"mcpServers": {
"screaming-frog": {
"command": "/path/to/screaming-frog-mcp",
"args": [],
"env": {
"SF_CLI_PATH": "/path/to/ScreamingFrogSEOSpiderLauncher"
}
}
}
}Restart Claude Desktop after editing the config.
Tool | Description |
| Verify Screaming Frog is installed, check version and license status |
| Start a headless background crawl (see note below) |
| Check progress of a running crawl |
| List all saved crawls with their Database IDs |
| Export crawl data as CSV files (many export options available) |
| Read exported CSV data with pagination, filtering, and column selection |
| Counts and group-by breakdowns over exported data ("how many 404s", "status code distribution") without reading rows into context |
| Permanently delete a crawl from the database |
| Show disk usage of SF's crawl storage |
"Is Screaming Frog installed and licensed?"
The assistant will call sf_check and report version/license info.
For most use cases, crawl in the Screaming Frog GUI where you have full control over configuration, JavaScript rendering, crawl scope, custom extraction, etc. Then close the GUI and use the MCP to analyze the results:
After you've crawled a site in the Screaming Frog GUI and closed it:
"List my saved crawls" "Export the crawl for example.com" "Show me all pages with missing meta descriptions" "What are the 404 pages?"
"Crawl https://example.com"
The crawl_site tool can kick off headless crawls via CLI. This is useful for quick re-crawls or automated workflows, but note the limitations compared to the GUI:
Uses default crawl settings (no custom extraction, JavaScript rendering config, etc.)
You can pass a .seospiderconfig file to customize settings (including crawl URL limits), but the GUI is easier for complex setups
The crawl must finish and save before you can export data
The server supports all of Screaming Frog's export tabs, bulk exports, and reports. Ask the assistant to read the screaming-frog://export-reference resource for the full list, or specify them directly:
export_tabs: "Internal:All,Response Codes:All,Page Titles:All"
bulk_export: "All Inlinks,All Outlinks"
save_report: "Crawl Overview"Variable | Description | Default |
| Path to the Screaming Frog CLI executable | macOS default path |
| Comma-separated list of allowed crawl target domains. When set, | Empty (all domains allowed) |
| Directory containing |
|
| How long exported CSV files are kept before auto-cleanup. Increase for multi-hour audit sessions. |
|
| Max time to wait for an |
|
read_crawl_data supports three filter modes via the filter_mode parameter:
Mode | Behavior | Example |
| Case-insensitive substring match |
|
| Case-insensitive exact match |
|
| Python regex (case-insensitive) |
|
Exported CSVs are stored in ~/.cache/sf-mcp/exports/ and are automatically cleaned up after 1 hour (configurable via SF_EXPORT_TTL_SECONDS).
Server won't connect at all? ("Could not attach to MCP server", "failed to connect") See TROUBLESHOOTING.md for a step-by-step diagnostic guide: testing the server manually, verifying the MCP handshake, and finding your client's logs.
Problem | Solution |
"GUI is already running" error | Quit the Screaming Frog application, then retry |
Empty CSV exports (headers only, 0 data rows) | The GUI likely has the database locked — close it and re-export |
CLI not found | Check that |
Crawl not appearing in | Make sure you saved the crawl in the GUI (File > Save) before closing |
Export times out | Large crawls may need more time — set |
| Fixed in v0.2.2 — update with |
"Could not attach to MCP server" / initialize timeout | Your config launches the server via |
MIT
Aggregate CSV data from an export: row counts and group-by breakdowns. Use instead of read_crawl_data when the question needs counts or a distribution ("how many 404s", "status code breakdown") rather than the rows themselves.
Args: export_id: The export_id from export_crawl file: CSV filename to aggregate (from the file list in export_crawl output) group_by: Optional column name; counts rows per distinct value of it top: Max distinct values to show, most common first (default 20, max 100) filter_column: Optional column name to filter by before aggregating filter_value: Optional value to match in the filter column filter_mode: How to match filter_value: "contains" (default), "exact", or "regex"
Returns: Total matching row count, plus per-value counts with percentages when group_by is set.
| Name | Required | Description | Default |
|---|---|---|---|
| top | No | ||
| file | Yes | ||
| group_by | No | ||
| export_id | Yes | ||
| filter_mode | No | ||
| filter_value | No | ||
| filter_column | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses that the tool returns total row count and per-value counts with percentages when group_by is set. Annotations indicate readOnlyHint=true, which matches the non-destructive aggregation behavior. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with a single summary sentence, a usage sentence, and a clear bulleted list of parameters. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 7 parameters, an output schema, and a clear sibling context, the description covers all necessary aspects: purpose, usage guidance, parameter details, and return format. No gaps remain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 0% schema description coverage, the description provides a detailed Args section explaining every parameter: export_id, file, group_by, top, filter_column, filter_value, filter_mode, including defaults and the three modes for filter_mode. This fully compensates for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool aggregates CSV data from an export, providing row counts and group-by breakdowns. It uses a specific verb (aggregate) and resource (CSV data from export), and distinguishes it from read_crawl_data which returns raw rows.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says 'Use instead of read_crawl_data when the question needs counts or a distribution', giving clear guidance on when to choose this tool over its sibling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start a background Screaming Frog crawl that saves to SF's internal database.
Args: url: The URL to crawl (e.g. https://example.com) config_file: Optional path to a .seospiderconfig file for crawl settings. To limit the number of URLs crawled, set the limit in a config file (Configuration > Spider > Limits in the SF GUI) and pass it here. label: Optional label for identifying this crawl (e.g. 'freshgovjobs')
Returns: A crawl_id to use with crawl_status to check progress. The crawl runs in the background - use crawl_status to poll.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| label | No | ||
| config_file | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false and destructiveHint=false, so the description adds value by revealing the crawl runs in background and saves to an internal database. This gives behavioral insight beyond the annotations, but lacks specifics on resource consumption or error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a summary line, Args block, and Returns block. It is front-loaded with the core purpose. Slightly verbose but all sentences add value; no redundancy. Could be trimmed slightly but remains efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool complexity (3 params, 1 required) and presence of an output schema (not shown), the description explains the return value and background nature. It covers essential usage for starting a crawl and linking to crawl_status. Missing error or prerequisite details, but sufficient for a straightforward start-crawl operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description adds full meaning: url is explained with an example, config_file includes guidance on limiting URLs, and label is given a usage example. This compensates completely for the bare schema, making parameters self-explanatory.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Start a background...crawl', the resource 'Screaming Frog crawl', and the storage destination 'SF's internal database'. It distinguishes from sibling tools like crawl_status (polling) and delete_crawl (deletion) by specifying the background execution and return of a crawl_id.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides guidance on using crawl_status to poll progress and explains the optional config_file and label parameters. However, it does not explicitly state when not to use this tool or provide comparative guidance against siblings like read_crawl_data or aggregate_crawl_data, though the background execution hint is useful.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check the status of a running or completed crawl.
Args: crawl_id: The crawl_id returned by crawl_site
| Name | Required | Description | Default |
|---|---|---|---|
| crawl_id | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, so the tool is known to be non-destructive. The description adds no additional behavioral context beyond that—no mention of potential errors, latency, or what 'status' means. With annotations covering the safety profile, the description is minimally adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: one clear sentence for purpose, followed by a compact parameter list. Every word is necessary and front-loaded. No redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple status-check tool with an output schema, the description covers the essential purpose and parameter origin. However, it lacks guidance on when to use it (e.g., after crawl_site), error states (invalid crawl_id), or how long the check might take. This is sufficient for a synchronous read, but slightly incomplete given the sibling tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, 'crawl_id', is described as 'The crawl_id returned by crawl_site', which adds significant meaning beyond the input schema's type and title (which have no description). This tells the agent exactly where the value comes from, compensating for the 0% schema description coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Check the status of a running or completed crawl.' It uses a specific verb ('check') and resource ('status of a crawl'), and the distinction from sibling tools like crawl_site (start) and delete_crawl (remove) is implicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention prerequisites (e.g., that crawl_site must have been called first) or situations where this tool is inappropriate. The sibling tools list is provided, but the description does not leverage it to guide selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Delete a crawl from Screaming Frog's internal database to free disk space.
Args: db_id: The Database ID from list_crawls
WARNING: This permanently deletes the crawl data. It cannot be undone.
| Name | Required | Description | Default |
|---|---|---|---|
| db_id | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description explicitly warns that data is permanently deleted and cannot be undone, which goes beyond the annotations (destructiveHint=true). There is no contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, using three sentences to state purpose, argument, and warning. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter, the description covers purpose, argument source, and critical behavior (permanent deletion). It is fully adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description explains the parameter 'db_id' and tells the agent to obtain it from list_crawls, adding essential meaning for correct invocation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'delete' and the resource 'crawl', and it distinguishes from sibling tools like list_crawls, export_crawl, etc., by focusing on removal to free disk space.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear prerequisite (db_id from list_crawls) and a strong warning about permanence. It doesn't explicitly state alternatives, but the action is so specific that misuse is unlikely.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Load a saved crawl from SF's database and export data as CSV files.
Args: db_id: The Database ID from list_crawls (e.g. '1234' or a crawl identifier) export_tabs: Comma-separated export tabs (default: Internal:All,Response Codes:All,Page Titles:All,Meta Description:All,H1:All,H2:All,Images:All,Canonicals:All,Directives:All). See the export-reference resource for all options. bulk_export: Optional bulk export types (e.g. 'Links:All Inlinks,Content:Soft 404 Inlinks') save_report: Optional reports to save (e.g. 'Crawl Overview')
Returns: An export_id and list of generated CSV files. Use read_crawl_data to read them.
| Name | Required | Description | Default |
|---|---|---|---|
| db_id | Yes | ||
| bulk_export | No | ||
| export_tabs | No | ||
| save_report | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the main behavioral effects: loading a saved crawl, generating CSV exports, returning an export_id, and optionally saving reports. The annotations only indicate readOnlyHint=false and destructiveHint=false, so most behavioral burden falls on the description, which it handles well. It could mention persistence or overwrite behavior more explicitly, but the provided details are not misleading.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well organized into a one-sentence purpose, Args list, and Returns note. The long default export_tabs string is verbose but functionally necessary since the schema omits it. No filler or redundant restatements of the tool name or schema fields are present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity, the description covers the prerequisite (existing crawl from list_crawls), all four parameters, the default behavior, and post-export reading via read_crawl_data. The return payload is summarized clearly. There is enough for an agent to invoke this tool correctly without external assumptions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate entirely for parameter meaning. It does so thoroughly: db_id is tied to list_crawls, export_tabs has a full default and points to a reference resource, and bulk_export and save_report both receive concrete examples. This exceeds what the bare schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('export') and resource ('saved crawl' data), with a concrete output format ('CSV files'). It also distinguishes itself from read_crawl_data by indicating that the exported files are read separately. This makes the tool's purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description establishes clear context: it loads an existing crawl identified by a db_id from list_crawls and produces export artifacts. It also tells the agent to use read_crawl_data to consume the results. It does not explicitly exclude alternatives like aggregate_crawl_data, but the workflow context is strong enough to guide correct selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List all crawls saved in Screaming Frog's internal database. Returns crawl names, Database IDs, and sizes. Use the Database ID with export_crawl or delete_crawl.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true. Description adds that the tool returns specific fields, but no additional behavioral context (e.g., performance, limits) beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three succinct sentences that frontline the purpose, describe outputs, and give usage guidance. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter list tool with annotations and an output schema, the description adequately covers purpose, results, and next steps. Minor omission of pagination or limits, but acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Zero parameters with 100% schema coverage; baseline is 4. Description does not need to add parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool lists all crawls from the internal database, specifies return fields (names, DB IDs, sizes), and distinguishes itself from sibling tools by referencing export and delete.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs to use the Database ID with export_crawl or delete_crawl, providing clear guidance on when to use this tool. Implicitly differentiates from sibling tools, though not exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read CSV data from an export. Use after export_crawl.
Args: export_id: The export_id from export_crawl file: CSV filename to read (from the file list in export_crawl output) limit: Max rows to return (default 100, max 1000) offset: Number of rows to skip (for pagination) filter_column: Optional column name to filter by filter_value: Optional value to match in the filter column filter_mode: How to match filter_value: "contains" (default, case-insensitive substring), "exact" (case-insensitive exact match), or "regex" (Python regex) columns: Optional comma-separated column names to return. Wide exports (Internal:All has dozens of columns) flood the context; request just the ones you need, e.g. "Address,Status Code".
Returns: CSV data as formatted text with column headers.
| Name | Required | Description | Default |
|---|---|---|---|
| file | Yes | ||
| limit | No | ||
| offset | No | ||
| columns | No | ||
| export_id | Yes | ||
| filter_mode | No | ||
| filter_value | No | ||
| filter_column | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description aligns with the readOnlyHint annotation and adds significant behavioral details: pagination (limit, offset), filtering options, and a warning about wide exports flooding context. This goes well beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: an introductory sentence, a clear list of parameters, and a returns note. Every sentence is informative, and the information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers all parameters, output format, and usage prerequisite. It provides enough context for an AI to correctly invoke the tool, including handling of wide exports and pagination, making it complete for a data reading tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description fully compensates by explaining all 8 parameters, including defaults, constraints (e.g., limit max 1000), and filter modes. It adds meaning that the schema alone lacks.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool reads CSV data from an export, using the verb 'Read' and specifying the resource. It distinguishes from siblings like export_crawl (which creates the export) and aggregate_crawl_data (which processes data), making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use after export_crawl,' providing a clear prerequisite. It also describes parameters like filter modes and pagination, guiding usage. However, it does not explicitly mention when not to use or alternatives, missing a small opportunity for deeper guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Verify that Screaming Frog SEO Spider is installed and the CLI is accessible. Returns version info and license status.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations mark it as readOnlyHint=true, and the description confirms it returns version info and license status. It adds context about the specific output, which is beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no redundant information. The purpose is front-loaded, and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with zero parameters and a clear purpose, the description fully explains what the tool does and returns. An output schema exists to detail return structure, so no further information is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, so schema coverage is effectively 100%. The description does not need to add parameter information; baseline score of 4 applies due to high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Verify' and the resource 'Screaming Frog SEO Spider installation and CLI accessibility'. It distinguishes itself from siblings by focusing on installation status rather than crawl operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage before other crawling operations but does not explicitly state when to use or not use this tool versus alternatives. No guidance on prerequisites or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Show disk usage of Screaming Frog's internal crawl storage. Returns total size and per-crawl breakdown of ProjectInstanceData.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnlyHint=true, indicating a safe read operation. The description adds specific details about returned values (total size and per-crawl breakdown), enhancing transparency without contradicting annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences, front-loading the purpose and specifying return content. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters, an output schema exists, and annotations are present, the description is complete. It sufficiently explains what the tool does and what it returns.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the baseline is 4. The description does not need to add parameter information.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool shows disk usage of Screaming Frog's internal crawl storage and returns total size and per-crawl breakdown. The verb 'show' and resource 'disk usage' are specific, and the tool is distinct from siblings which focus on crawling and data export.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives. While the purpose is clear, there is no guidance on context or exclusions, leaving usage implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.4.0aggregate_crawl_dataread_crawl_data1 field changedInput schema / properties / columnsAdded value: +{
+ "anyOf": [
+ {
+ "type": "string"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "default": null,
+ "title": "Columns"
+}v0.3.3crawl_sitecrawl_statusdelete_crawlexport_crawllist_crawlsread_crawl_datasf_checkstorage_summaryScored across 9 tools
Each tool maps to a distinct step in the crawl lifecycle: install check, start, status, list, storage, export, read, aggregate, and delete. Even the most similar pair, read_crawl_data and aggregate_crawl_data, is clearly separated by row-level versus count/group-by output.
Most tools follow a clear verb_noun snake_case pattern (crawl_site, export_crawl, delete_crawl), but sf_check is a noun_verb exception and storage_summary is noun-only. The naming is still predictable and readable overall.
Nine tools cover crawl initiation, monitoring, storage management, export, data access, and cleanup without redundancy. The count is appropriate for the server's focused SEO crawling purpose.
The full crawl workflow is covered: start, check status, list crawls, export results, read/aggregate data, and delete to free space. The main gap is no way to stop or cancel an in-progress background crawl, which could leave agents with no abort path for long-running crawls.
Scrape, crawl and search the web for AI agents via MCP.
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
Firecrawl MCP — wraps the Firecrawl API (firecrawl.dev) for web