# screaming-frog-mcp

MCP server to analyze Screaming Frog SEO Spider crawl files directly from an
AI agent. No credentials — it only needs the path to a saved crawl file.

## Purpose

Reads Screaming Frog crawl exports (`.dbseospider`, `.seospider`, `.duckdb`,
or a folder of exported CSVs) without manually exporting anything. Built on
top of Screaming Frog's ~628 pre-computed report tabs, so counts are instant
even on 100K+ URL crawls — nothing gets loaded fully into memory unless you
explicitly sample or export it.

## Requirements

- Screaming Frog SEO Spider installed
- Java 8+ available (`java -version`)
- Python package `screamingfrog` (`pip install screamingfrog`)

## Tools (11)

| Tool | What it does |
|---|---|
| `sf_full_audit` | Full SEO audit in one call: HTTP errors, redirects, titles, meta descriptions, H1s, indexability, images, thin content — counts + sample URLs + a health score |
| `sf_issues_summary` | Instant counts of every issue type (no memory load, works on huge crawls) |
| `sf_get_issue` | Affected URLs for one specific issue, with pagination (`limit`/`offset`) |
| `sf_export` | Export all URLs for an issue to CSV (no row limit) |
| `sf_list_tabs` | List all available report tabs in the crawl, optionally filtered |
| `sf_crawl_summary` | General crawl summary: total URLs, status code distribution |
| `sf_sql` | Run a SQL query directly against the crawl (DuckDB, table `APP.URLS`) |
| `sf_search_pages` | Search a term across URL, title and meta description |
| `sf_inlinks` | All internal links pointing to a specific URL |
| `sf_redirect_chains` | Redirect chains with N or more hops |
| `sf_compare_crawls` | Diff two crawls: status, title and redirect changes |

`sf_get_issue`/`sf_export` accept either a friendly alias (`sin_titulo`,
`errores_4xx`, `h1_missing`...) or the exact Screaming Frog tab name — use
`sf_list_tabs` to discover what's available in your crawl.

## Workflow

1. Open Screaming Frog and crawl the site.
2. `File → Save Crawl` → save as `.seospider` anywhere on disk.
3. Point the agent at it:
   > "Run a full audit on `/path/to/crawl-2026-08.seospider`"

## Supported file formats

| Extension | Notes |
|---|---|
| `.seospider` | Native Screaming Frog format — needs internal conversion (slower on first load) |
| `.dbseospider` | Same, with a DuckDB cache included — instant on second load |
| `.duckdb` | Exported DuckDB database |
| Folder of CSVs | Manual Screaming Frog exports |

## Setup

No credentials needed.

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## MCP client configuration

```json
{
  "mcpServers": {
    "screaming-frog": {
      "command": "/path/to/.venv/bin/python3",
      "args": ["/path/to/screaming-frog-mcp/server.py"]
    }
  }
}
```

## Security

- Every tool carries MCP Tool Annotations, and every single one is marked
  `readOnlyHint=true` — this server only reads a local crawl file, it never
  writes to it or to anything else.
- Bad input (missing crawl file, invalid SQL) raises an exception and
  surfaces as a real MCP protocol error (`isError=true`). A couple of read
  helpers return `None` or an empty list instead when an optional report tab
  simply isn't present in a given crawl — Screaming Frog can generate ~628
  possible tabs, but not every crawl includes every one of them, so a missing
  tab isn't treated as a failure.

## Notes

- First load of a `.seospider` file can take 30–60s (converts Derby → DuckDB
  internally). Subsequent loads of the same file are fast thanks to the
  on-disk `.duckdb` cache.
- Results are capped (100–500 rows depending on the tool) to avoid flooding
  the agent's context — use `sf_export` when you need the full list.
- If you have a JDK 21 installed at `~/.local/jdk21`, it's picked up
  automatically via `JAVA_HOME`.

## License

MIT — see [LICENSE](LICENSE).