seohead

SpiderHead is a working environment for an SEO specialist and their AI agent. Crawl a site, keep every scan, find and prioritise issues, hand tasks to developers, recheck the fixes and report — all on your own machine, in a native desktop app, in the terminal or through Claude over MCP.

• One core, three faces. The desktop app, the seohead CLI and the MCP server share the same Python core, projects and retained SQLite scans.

• Projects, not one-off crawls. Scans, tasks, sources (Search Console, Metrika, Webmaster, GA4) and history live together, so the next audit starts where the last one ended.

• Local-first. No hosted account and no web dashboard; paid providers run only with your key and your confirmation.

Three ways to use it

All three call the same shared handlers (crawl-site in the CLI is seo_crawl_site over MCP) and work on the same local projects and retained scans.

Related MCP server: Agent SEO Engine

Install

One command (pipx)

pipx install seohead-seotools
spiderhead --version

spiderhead is the primary command; seohead is a fully equivalent alias, and sf-analyzer ships alongside. The seohead-seotools distribution is not published to PyPI yet, so that exact command starts working once the upload lands. Today the same one-command install already works from the GitHub release, which pipx puts into the same kind of isolated environment:

pipx install https://github.com/PavloSEO/spiderhead/releases/download/v3.2.0/seohead_seotools-3.2.0.tar.gz

Both routes install the base dependencies only. Optional extras (mcp, reports, render, …) can be added with pipx inject seohead-seotools <extra-packages>; the extras table in SETUP.md names what each one unlocks.

Homebrew (planned)

The maintained formula source lives in packaging/homebrew/spiderhead.rb — a pypi-style formula pinned to the v3.2.0 release sdist with the full dependency closure; it passes brew style and brew audit. The tap repository itself (PavloSEO/homebrew-tap) does not exist yet, so until it is published the command below is planned, not working:

brew install PavloSEO/tap/spiderhead   # planned — no tap published yet

SpiderHead Desktop

Ready-made desktop installers (macOS .dmg, Windows .exe setup, Linux .AppImage/.deb) are attached to every release tag: look for SpiderHead-<version>-<platform> files and their desktop-SHA256SUMS.txt. Until release certificates are configured the builds are unsigned, so macOS Gatekeeper and Windows SmartScreen warn on first launch.

The app lives in desktop/ and installs together with the core (Python 3.10 or newer):

git clone https://github.com/PavloSEO/spiderhead.git
cd spiderhead
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all]" -e desktop
seohead-desktop --project /path/to/existing/project

Packaging (PyInstaller bundles with the core and the seohead CLI) and the source install are described in desktop/docs/.

Core toolkit (CLI and MCP)

Python 3.11 or newer. Clone the repository and install it into a virtual environment:

git clone https://github.com/PavloSEO/spiderhead.git
cd spiderhead
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[all]"

# Confirm the installed interface.
seohead --help

With uv, uv sync --all-extras creates the same environment from the committed uv.lock, and uv run seohead --help runs it. On Windows PowerShell, activate the virtual environment with .venv\Scripts\Activate.ps1.

The repository is named spiderhead; the Python distribution is seohead-seotools. spiderhead is the primary CLI name and seohead is a fully equivalent alias; the import package remains seohead.

all installs every optional dependency; smaller extras (mcp, render, reports, pdf, cluster, gsc, sitemap, tui, remote) are listed in SETUP.md.

Provider credentials and browser binaries are separate and never required for the core crawl. Setup from zero covers versions, environment variables, Docker (CONTAINERS.md) and Linux servers without a display (LINUX_VPS.md).

Quick start

1. Crawl a site, audit it, build a report

# Crawl locally. The URL cap is an explicit budget, not a claim about site size.
seohead crawl-site \
  --url https://example.com \
  --max-urls 500 \
  --scan-out ./scans/audit.sqlite

# Format the retained audit as a working spreadsheet or a client document.
seohead report-build --audit ./scans/audit.sqlite --format xlsx --out audit.xlsx
seohead report-build --audit ./scans/audit.sqlite --format docx --out audit.docx

crawl-site is the primary collector: free, local, polite by default (about two requests per second per host), resumable (RECOVERY.md) and configurable through --config/--set (see seohead crawl-site --config-help and the configuration inventory). It sends only bounded, read-only requests to the site you name. report-build formats existing evidence as XLSX, DOCX, CSV, Markdown, JSON or a bounded PDF overview (PDF_OVERVIEW.md); it never invents findings.

2. Analyse existing Screaming Frog exports

# Offline: no SF installation, licence or request to the site is needed.
seohead sf run --exports-dir docs/examples/exports --out ./report --tasks

You get audit.json, audit.md and a prioritized tasks.md/tasks.json backlog with reproduction steps, DOM positions and fixes. A separately installed, licensed SF CLI can also be driven directly with sf run --crawl. Other crawlers' CSV exports are not drop-in compatible; they are read through a versioned manifest (THIRD_PARTY_CRAWL_IMPORT.md).

3. Run it as a project with an agent

# These calls only create or inspect local project state; they do not crawl.
seohead project new --directory ./shop --target https://example.com/
seohead project checklist-init --directory ./shop
seohead project progress --directory ./shop

Register the MCP server (below), then ask the agent for the outcome you need, for example:

Audit this site for the agreed scope. Give the developers an Excel workbook of tasks, the underlying exports and evidence links, proposed fixes, and acceptance/recheck criteria.

The agent starts from the control skill, records scope and crawl policy in the project, reuses retained evidence and follows the developer handoff scenario. See Projects and agent handoff.

Desktop app

SpiderHead Desktop is a native PyQt5 application for macOS, Windows and Linux. It is a presentation layer over the same core: it opens the same local projects and retained scans as the CLI and MCP server, so a crawl started in the terminal or by an agent shows up in the window and the other way round.

• Crawler — enter a URL and press Start, no project needed; save the result as a project later.

• Project work — goal, tasks, inbox and run history in a With agent display, or the same projects, scans and reports without AI features in a Simple display.

• URLs, issues, link graph, compare — browse retained evidence, rechecks and before/after scans.

• Tools — a built-in toolbox: every core tool with a ready-made form, plus a one-click image pipeline — compress, convert to WebP/AVIF, generate 301 redirects and hand the task to your developer or agent.

• Methods and reports — the core's method skills and report builders behind ready-made screens.

• MCP switch — one shared state with the CLI (seohead mcp status); the app registers the server in Claude Code, Claude Desktop, Codex or Cursor only after a permission dialog that lists the files and lines it will change, with backups.

• Installers — macOS .pkg, Windows setup and Linux tarball put the app and the seohead command on the machine together. Light, dark and high-contrast themes; English UI.

What it can do

Each area links to the page that documents its method, inputs and limits. The capability map answers the same question by engineering task.

MCP server for Claude and other agents

Claude Code — register the installed CLI (use the absolute path to your virtual environment):

claude mcp add seohead -- /absolute/path/to/spiderhead/.venv/bin/seohead mcp

Inside this repository the committed .mcp.json already registers seohead for Claude Code when the virtual environment is active.

Claude Desktop and other stdio clients — add the server to the client's MCP configuration (claude_desktop_config.json for Claude Desktop):

{
  "mcpServers": {
    "seohead": {
      "command": "/absolute/path/to/spiderhead/.venv/bin/seohead",
      "args": ["mcp", "--profile", "full"]
    }
  }
}

--profile full exposes every tool; audit, infra, quick-check and router expose smaller schema sets for focused sessions. seohead mcp status shows the shared on/off state that the desktop app uses too. Progress notifications are sent only when the client supplies a progress token. See MCP profiles and progress.

HTTP-native MCP clients can reach the same registry over streamable HTTP: seohead mcp --transport http binds 127.0.0.1:8321 by default; stdio stays the default, and a non-loopback --host is an explicit, warned-about exposure decision (transports in docs/CLI.md).

An agent discovers routes with seo_tool_catalog, loads a method with seo_skill_show and an end-to-end chain with seo_scenario_show. A catalogue entry describes a capability; it does not execute it. The CLI shows the same catalogue:

seohead tool-catalog --query redirect --limit 5

Projects and agent handoff

A project directory keeps the site, agreed scope, crawl policy, checklist coverage, retained scans, saved finding views and run history, so a new agent continues from the saved project instead of chat history: define the goal and scope, collect once and analyse again, compare, create developer tasks, track repairs and report.

seohead watch --project DIRECTORY opens an optional terminal observer beside the chat (TERMINAL.md). The inbox stores the specialist's notes and proposed goals; workflow runs checkpoint each registered step with hashed evidence so another agent can resume exactly where the first stopped (WORKFLOWS.md). Task completion and site health stay separate: a prepared project or a completed task is not proof that a site error was fixed. Details: Projects, project-control scenario, remediation ledger.

What is measured and what is not

Every audit separates four outcomes:

• Finding: the available evidence supports a specific problem.

• Ran without findings: the check was evaluated and found nothing.

• Skipped: required evidence was unavailable or incomplete; the report names the check and reason.

• Failed or unavailable tool: the boundary failure is recorded instead of being treated as a pass.

Partial crawls withhold conclusions that need complete evidence, such as link-graph claims, and health scores are withheld or marked not comparable when coverage is low. Native crawls and Screaming Frog are different collectors and may discover different URL populations; compare them only with compatible scope, configuration and provenance.

Known limits: native crawl and browser results are lab evidence, not field Core Web Vitals (crux-report is credential-gated); site-audit is a bounded sitemap pass, not a link-graph crawl; backlinks-check verifies donor pages you supply rather than a web-scale index; full parity with commercial crawlers on very large or JavaScript-heavy sites is a direction, not a verified claim (COMPARISON.md, MILLION_CRAWL_ACCEPTANCE.md).

Safety boundaries: network tools block private targets unless explicitly allowed; file mutation, service-path probes, bot DNS verification, provider production mode and paid calls require explicit inputs. DataForSEO defaults to sandbox, and paid calls are journalled for spend-report. Image optimization writes to a separate directory unless in-place mode is requested, which keeps backups. Secrets and client crawl data never belong in this repository or a client report. See the audit guideline and GOTCHAS.md.

Repository map

Development

python -m pip install -e ".[dev,mcp,cluster,reports]"
ruff check .
ruff format --check .
pytest -q
seohead sf run --exports-dir docs/examples/exports --out /tmp/seohead-report --tasks
python -m build

Public commands, counts and generated references are checked in CI: a command shown in the docs must still run, and generated pages must match the registries. Keep public prose in English, use synthetic examples and reserved domains only, and add a focused offline test whenever behaviour changes. See CONTRIBUTING.md, SECURITY.md, architecture and testing.

Licence and provenance

The core toolkit, documentation and SpiderHead Desktop are released under the MIT License (desktop/LICENSE carries the same text). The bundled Qt runtime (PySide6/Qt 6) remains under LGPL v3 on its own terms — see the notices and Qt source offer in desktop/THIRD_PARTY_NOTICES.md. The bundled Schema.org vocabulary keeps its original CC BY-SA terms. See THIRD_PARTY_NOTICES.md, PROVENANCE.md, TRADEMARKS.md and CITATION.cff.

Screaming Frog and SEO Spider are trademarks of Screaming Frog Ltd. SpiderHead is independent and not affiliated with or endorsed by Screaming Frog Ltd; the name is used only to describe compatibility.