seohead

SpiderHead is a working environment for an SEO specialist and their AI agent. Crawl a site, keep every scan, find and prioritise issues, hand tasks to developers, recheck the fixes and report — all on your own machine, in a native desktop app, in the terminal or through Claude over MCP.

• One core, three faces. The desktop app, the seohead CLI and the MCP server share the same Python core, projects and retained SQLite scans.

• Projects, not one-off crawls. Scans, tasks, sources (Search Console, Metrika, Webmaster, GA4) and history live together, so the next audit starts where the last one ended.

• Local-first. No hosted account and no web dashboard; paid providers run only with your key and your confirmation.

Three ways to use it

All three call the same shared handlers (crawl-site in the CLI is seo_crawl_site over MCP) and work on the same local projects and retained scans.

Related MCP server: Agent SEO Engine

Install

SpiderHead Desktop

The app lives in desktop/ and installs together with the core (Python 3.10 or newer):

git clone https://github.com/PavloSEO/spiderhead.git
cd spiderhead
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all]" -e desktop
seohead-desktop --project /path/to/existing/project

Packaging (PyInstaller bundles with the core and the seohead CLI) and the source install are described in desktop/docs/.

Core toolkit (CLI and MCP)

Python 3.10 or newer. Clone the repository and install it into a virtual environment:

git clone https://github.com/PavloSEO/spiderhead.git
cd spiderhead
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[all]"

# Confirm the installed interface.
seohead --help

With uv, uv sync --all-extras creates the same environment from the committed uv.lock, and uv run seohead --help runs it. On Windows PowerShell, activate the virtual environment with .venv\Scripts\Activate.ps1.

The repository is named spiderhead; the Python distribution is seohead-seotools and the installed command and import package are seohead.

all installs every optional dependency; smaller extras (mcp, render, reports, pdf, cluster, gsc, sitemap, tui, remote) are listed in SETUP.md.

Provider credentials and browser binaries are separate and never required for the core crawl. Setup from zero covers versions, environment variables, Docker (CONTAINERS.md) and Linux servers without a display (LINUX_VPS.md).

Quick start

1. Crawl a site, audit it, build a report

# Crawl locally. The URL cap is an explicit budget, not a claim about site size.
seohead crawl-site \
  --url https://example.com \
  --max-urls 500 \
  --scan-out ./scans/audit.sqlite

# Format the retained audit as a working spreadsheet or a client document.
seohead report-build --audit ./scans/audit.sqlite --format xlsx --out audit.xlsx
seohead report-build --audit ./scans/audit.sqlite --format docx --out audit.docx

crawl-site is the primary collector: free, local, polite by default (about two requests per second per host), resumable (RECOVERY.md) and configurable through --config/--set (see seohead crawl-site --config-help and the configuration inventory). It sends only bounded, read-only requests to the site you name. report-build formats existing evidence as XLSX, DOCX, CSV, Markdown, JSON or a bounded PDF overview (PDF_OVERVIEW.md); it never invents findings.

2. Analyse existing Screaming Frog exports

# Offline: no SF installation, licence or request to the site is needed.
seohead sf run --exports-dir docs/examples/exports --out ./report --tasks

You get audit.json, audit.md and a prioritized tasks.md/tasks.json backlog with reproduction steps, DOM positions and fixes. A separately installed, licensed SF CLI can also be driven directly with sf run --crawl. Other crawlers' CSV exports are not drop-in compatible; they are read through a versioned manifest (THIRD_PARTY_CRAWL_IMPORT.md).

3. Run it as a project with an agent

# These calls only create or inspect local project state; they do not crawl.
seohead project new --directory ./shop --target https://example.com/
seohead project checklist-init --directory ./shop
seohead project progress --directory ./shop

Register the MCP server (below), then ask the agent for the outcome you need, for example:

Audit this site for the agreed scope. Give the developers an Excel workbook of tasks, the underlying exports and evidence links, proposed fixes, and acceptance/recheck criteria.

The agent starts from the control skill, records scope and crawl policy in the project, reuses retained evidence and follows the developer handoff scenario. See Projects and agent handoff.

Desktop app

SpiderHead Desktop is a native PyQt5 application for macOS, Windows and Linux. It is a presentation layer over the same core: it opens the same local projects and retained scans as the CLI and MCP server, so a crawl started in the terminal or by an agent shows up in the window and the other way round.

• Crawler — enter a URL and press Start, no project needed; save the result as a project later.

• Project work — goal, tasks, inbox and run history in a With agent display, or the same projects, scans and reports without AI features in a Simple display.

• URLs, issues, link graph, compare — browse retained evidence, rechecks and before/after scans.

• Tools — a built-in toolbox: every core tool with a ready-made form, plus a one-click image pipeline — compress, convert to WebP/AVIF, generate 301 redirects and hand the task to your developer or agent.

• Methods and reports — the core's method skills and report builders behind ready-made screens.

• MCP switch — one shared state with the CLI (seohead mcp status); the app registers the server in Claude Code, Claude Desktop, Codex or Cursor only after a permission dialog that lists the files and lines it will change, with backups.

• Installers — macOS .pkg, Windows setup and Linux tarball put the app and the seohead command on the machine together. Light, dark and high-contrast themes; English UI.

What it can do

Each area links to the page that documents its method, inputs and limits. The capability map answers the same question by engineering task.

MCP server for Claude and other agents

Claude Code — register the installed CLI (use the absolute path to your virtual environment):

claude mcp add seohead -- /absolute/path/to/spiderhead/.venv/bin/seohead mcp

Inside this repository the committed .mcp.json already registers seohead for Claude Code when the virtual environment is active.

Claude Desktop and other stdio clients — add the server to the client's MCP configuration (claude_desktop_config.json for Claude Desktop):

{
  "mcpServers": {
    "seohead": {
      "command": "/absolute/path/to/spiderhead/.venv/bin/seohead",
      "args": ["mcp", "--profile", "full"]
    }
  }
}

--profile full exposes every tool; audit, infra, quick-check and router expose smaller schema sets for focused sessions. seohead mcp status shows the shared on/off state that the desktop app uses too. Progress notifications are sent only when the client supplies a progress token. See MCP profiles and progress.

An agent discovers routes with seo_tool_catalog, loads a method with seo_skill_show and an end-to-end chain with seo_scenario_show. A catalogue entry describes a capability; it does not execute it. The CLI shows the same catalogue:

seohead tool-catalog --query redirect --limit 5

Projects and agent handoff

A project directory keeps the site, agreed scope, crawl policy, checklist coverage, retained scans, saved finding views and run history, so a new agent continues from the saved project instead of chat history: define the goal and scope, collect once and analyse again, compare, create developer tasks, track repairs and report.

seohead watch --project DIRECTORY opens an optional terminal observer beside the chat (TERMINAL.md). The inbox stores the specialist's notes and proposed goals; workflow runs checkpoint each registered step with hashed evidence so another agent can resume exactly where the first stopped (WORKFLOWS.md). Task completion and site health stay separate: a prepared project or a completed task is not proof that a site error was fixed. Details: Projects, project-control scenario, remediation ledger.

What is measured and what is not

Every audit separates four outcomes:

• Finding: the available evidence supports a specific problem.

• Ran without findings: the check was evaluated and found nothing.

• Skipped: required evidence was unavailable or incomplete; the report names the check and reason.

• Failed or unavailable tool: the boundary failure is recorded instead of being treated as a pass.

Partial crawls withhold conclusions that need complete evidence, such as link-graph claims, and health scores are withheld or marked not comparable when coverage is low. Native crawls and Screaming Frog are different collectors and may discover different URL populations; compare them only with compatible scope, configuration and provenance.

Known limits: native crawl and browser results are lab evidence, not field Core Web Vitals (crux-report is credential-gated); site-audit is a bounded sitemap pass, not a link-graph crawl; backlinks-check verifies donor pages you supply rather than a web-scale index; full parity with commercial crawlers on very large or JavaScript-heavy sites is a direction, not a verified claim (COMPARISON.md, MILLION_CRAWL_ACCEPTANCE.md).

Safety boundaries: network tools block private targets unless explicitly allowed; file mutation, service-path probes, bot DNS verification, provider production mode and paid calls require explicit inputs. DataForSEO defaults to sandbox, and paid calls are journalled for spend-report. Image optimization writes to a separate directory unless in-place mode is requested, which keeps backups. Secrets and client crawl data never belong in this repository or a client report. See the audit guideline and GOTCHAS.md.

Repository map

Development

python -m pip install -e ".[dev,mcp,cluster,reports]"
ruff check .
ruff format --check .
pytest -q
seohead sf run --exports-dir docs/examples/exports --out /tmp/seohead-report --tasks
python -m build

Public commands, counts and generated references are checked in CI: a command shown in the docs must still run, and generated pages must match the registries. Keep public prose in English, use synthetic examples and reserved domains only, and add a focused offline test whenever behaviour changes. See CONTRIBUTING.md, SECURITY.md, architecture and testing.

Licence and provenance

The core toolkit and documentation are released under the MIT License. SpiderHead Desktop in desktop/ is GPL-3.0-or-later, as required by PyQt5 (see desktop/LICENSE). The bundled Schema.org vocabulary keeps its original CC BY-SA terms. See THIRD_PARTY_NOTICES.md, PROVENANCE.md, TRADEMARKS.md and CITATION.cff.