# seo-audit-mcp

An MCP server that gives Claude (or any MCP client) the ability to audit a live
website's technical SEO: sitemap coverage, per-page issues, and redirect chains.

Ask in plain language — *"audit mortgagecalculatortools.com and tell me which
pages Google is never told about"* — and the model calls the tools, crawls the
site, and answers with specifics.

---

## The problem it solves

A site's `sitemap.xml` is how you tell Google which pages exist. When a page is
missing from it, nothing errors and nothing warns — the page simply never
accumulates impressions. Checking it by hand means diffing a filesystem
listing against an XML file, so in practice nobody does it.

### Case study: a 25-page gap that turned out to be correct

The first site this was pointed at had **125 HTML files on disk and 100 URLs in
its sitemap**. A 25-page gap — the kind of finding that gets written up as a
bug and assigned to someone.

One `sitemap_coverage` call surfaced the gap, and one `audit_urls` call on a
sample explained it: every one of the 25 carried `<meta name="robots"
content="noindex, follow">`. They were two deliberately de-indexed content
clusters, and the sitemap was **exactly right** to omit them. Verified against
the filesystem afterwards: 25 `noindex` pages on disk, the same 25 absent from
the sitemap, zero `noindex` pages wrongly included. Perfect consistency.

That is the useful result. A coverage number alone ("125 vs 100") reads as a
defect and buys a day of somebody's time; coverage plus per-page `noindex`
status closes the question in a minute. **This tool is as valuable for the
false alarms it kills as for the real gaps it finds** — which is why
`audit_urls` reports `noindex` per page rather than only counting URLs.

---

## Tools

| Tool | What it does |
|---|---|
| `fetch_sitemap` | Fetches `sitemap.xml`, follows sitemap-index nesting (max depth 3), returns every declared URL, deduped |
| `audit_urls` | Crawls URLs concurrently and reports per-page issues: broken status, redirect chains, missing/over-length `<title>` and meta description, missing or duplicate `<h1>`, missing canonical, `noindex`, thin content |
| `sitemap_coverage` | Diffs a sitemap against a list of URLs you know exist → what's **missing from the sitemap**, what's declared but **dead** |
| `check_redirects` | Traces redirect chains, flags multi-hop chains and chains ending in 4xx/5xx — use after a URL-structure change |

Every tool returns structured JSON with an `issues` list per page and an
aggregated `issue_summary`, so the model can reason over counts instead of
re-reading raw HTML.

---

## Install

```bash
pip install -e .
```

Requires Python 3.10+. Dependencies: `mcp>=2.0.0`, `httpx`.

## Connect it to Claude Code

Add to `.mcp.json` in your project (or `~/.claude.json` for global use):

```json
{
  "mcpServers": {
    "seo-audit": {
      "command": "python",
      "args": ["-m", "seo_audit_mcp"]
    }
  }
}
```

For Claude Desktop, the same block goes in `claude_desktop_config.json`.

Then just ask:

> Fetch the sitemap for https://example.com/sitemap.xml, audit the first 20
> URLs, and summarise the issues by frequency.

## Run it directly

```bash
python -m seo_audit_mcp        # stdio transport
```

---

## Design notes

Three decisions worth calling out, because they are the difference between a
demo and something you can point at a client's production site:

**Crawling is rate-limited by construction.** `fetch_many` runs behind an
`asyncio.Semaphore` capped at 16 concurrent requests, and every tool clamps
its input. A 500-URL sitemap without that ceiling would open 500 sockets at
once and read as an attack to the target host. The crawl is a cost the *target*
site pays, so the ceiling is not configurable upward from the tool surface.

**No fetch failure aborts a run.** `fetch_one` catches `httpx.HTTPError` and
records it on the returned `PageAudit` rather than raising. One dead host in a
200-URL crawl degrades one row instead of losing 199 good results.

**Parsing is deliberately lenient.** Real-world HTML is malformed often enough
that a strict parser raising mid-crawl is a liability. The extractors are
permissive regexes that return `None` rather than throw — but with the traps
handled: `<script>` and `<style>` bodies are stripped before word-counting and
heading extraction, so a `<h1>` inside a JS string literal is not counted as a
heading, and relative canonicals are resolved against the page URL.

`normalize_url` deliberately does **not** strip trailing slashes: `/a` and `/a/`
can be genuinely different pages, and collapsing them would hide duplicate-content
problems this tool exists to surface.

---

## Tests

```bash
pip install -e ".[dev]"
pytest
```

The suite is network-free — HTTP is exercised through `httpx.MockTransport`,
so it runs in CI and on a plane. It covers the parsing edge cases that bite in
production: script-embedded headings, namespace-less sitemaps, relative
canonicals, sitemap URLs that return a styled HTML 404 with a 200 status, and
non-HTML content types being wrongly reported as pages "missing a title".

---

## License

MIT