# Super Spider

A crawling and technical SEO auditing tool with a local web dashboard, built in
Python. It crawls a site, extracts the signals that decide whether and how pages
can rank, raises findings at four severities, and reports everything to CSV,
Excel or JSON — or to a browser dashboard organised the way a technical SEO
auditor reads a crawl.

It also speaks the Model Context Protocol, so an agent can query a crawl, verify
a fix, and diff two crawls instead of guessing about a site it never fetched.

```bash
skl example.com --go
```

The dashboard runs at `http://127.0.0.1:8765` and is served by the standard
library only, so there is no extra runtime to install.

---

## What this is, and what it is not

Worth reading before you judge the numbers it produces.

**What it is.** An independent implementation of the crawl-and-audit loop. It
checks a real site over the network, stores what it measured, diffs two crawls,
and can hand the whole thing to an agent over MCP. Everything it reports traces
to a request it actually made.

**What it is not.** It is not Screaming Frog, and it does not try to be. The
feature surface was designed by reading the reference product's own resource
bundles (see below), but the code is its own, the metric formulas are its own
and are documented, and a few areas are deliberately narrower:

- The **link score** is a documented approximation, not PageRank and not the
  reference product's metric.
- **Title and description pixel widths** are estimates from a character-advance
  model, not measurements from a search engine's font stack.
- The **built-in accessibility rules** are a documented subset, not axe-core,
  unless you supply `axe.min.js` — and the report says which engine ran.
- **Spelling** needs a word list you provide. Without one it reports "not
  measured", never zero errors.
- **PageSpeed, Search Console, GA4 and AI columns** need credentials. Without
  them the affected field records why, and the crawl prints which modules could
  not run.

Where a question cannot be answered from the data, the field says so. That is a
deliberate rule rather than a limitation to work around, because an empty column
that looks like a clean result is the most expensive kind of wrong.

---

## Where the feature set came from

The scope was not guessed. A copy of Screaming Frog SEO Spider 22.2 was present
on the development machine, and its shipped `ScreamingFrogSEOSpider.jar` was
inspected read-only (`open_binary` → `inspect_artifact` → inventory paging),
which established:

| Reference evidence | What it determined here |
| --- | --- |
| `lib/` dependency inventory | That a serious crawler needs a parsing stack, real browser rendering, RDF handling for structured data, language tooling, and embedded storage |
| `translations/SeoSpider_en.properties` (`table.column.*`, 206 keys) | The report column taxonomy |
| the same bundle (`issue.*`, ~350 keys across 14 families) | The issue vocabulary and severity classes |
| the same bundle (`tab.*`, `mode.list.*`) | The 28-tab navigation structure and its per-tab filters |
| the same bundle (`configuration.spider.*`, 294 keys) | The crawl/extraction/limit/threshold settings surface |
| the same bundle (`cli.*`) | The command line option surface |

**No code was copied.** The extracted evidence describes a *feature surface* —
which columns exist, which settings exist. Every rule, parser, formula and report
here is an independent implementation written against that surface, and the
user-facing wording is original. The parsing, extraction, indexability, dedup,
planning and MCP layers are all first-party code.

---

## Install

```bash
pip install -e .              # core: requests, lxml, openpyxl
pip install -e ".[dev]"       # adds pytest
pip install -e ".[render]"    # adds Playwright for JS rendering and axe audits
playwright install chromium   # the browser binary Playwright drives
```

| Module | Needs | Without it |
| --- | --- | --- |
| JS rendering | `playwright` + Chromium | the JavaScript columns record `render_error` |
| Accessibility audit | `playwright`; real axe-core if you supply `axe.min.js` | falls back to a documented built-in rule subset |
| PageSpeed | a PageSpeed Insights API key | `psi_error` on every HTML record |
| Search Console / GA4 | an access token, or a service-account file plus `google-auth` | `gsc_error` / `ga_error` |
| Spelling | a word list (`--word-list`) | reported as not measured, not as zero errors |
| PDF metadata | `pypdf` or `PyPDF2` | PDF columns stay empty |
| Jev decisions | a `TYPESAFE_API_KEY` | `jev_error` on every page, and no decision is claimed |

A crawl finishes by printing every enabled module that could not run and why, so
a missing key shows up as a warning rather than as a clean-looking empty column.

---

## Why this exists in an agentic era

A language model cannot fetch ten thousand URLs, cannot follow a redirect chain
hop by hop, cannot verify that every page in an hreflang cluster annotates the
others, and cannot prove a claim about a page it never downloaded. It is a
reasoner without instruments.

So the crawler does not compete with the agent, it equips it. Three things a
model on its own cannot provide:

1. **Ground truth.** Status codes, redirect chains, byte sizes, link counts and
   the exact matching robots.txt line are *measured*, not recalled. Every number
   in this tool traces to a request that was made.
2. **Scale and patience.** Sixty URLs politely, respecting `crawl-delay`; or
   sixty thousand. An agent would burn its context window on the first five.
3. **A stable target.** Crawls are stored, so two runs can be diffed. That is
   what turns "I changed the template" into "the finding is gone", without
   trusting anyone's summary — including the model's.

The AI layer sits on top and is deliberately narrow. It clusters findings that
share a root cause, scores them with a formula you can audit, and drafts the
markup the fix needs. It is not asked to find problems; it is handed the
measurements and asked to explain them. When no model is configured, the plan is
still produced — smaller, but never fabricated.

An earlier version of the scoring divided by an "effort" term. On a real crawl
that ranked a 26-page server outage *below* a missing alt attribute on 17 pages,
because "fix the server" scored as hard work. The formula no longer contains
effort, and the incident in that example now ranks first at 79/100. That is the
kind of correction that only shows up when the tool is pointed at a live site.

---

## Agent interface (MCP)

`superspider mcp` runs a Model Context Protocol server on stdio, exposing the
crawl as twelve inspectable tools. Point any MCP client at it:

```json
{
  "mcpServers": {
    "superspider": {
      "command": "python",
      "args": ["-m", "superspider", "mcp", "--load-crawl", "crawls/example.com/crawl.db"]
    }
  }
}
```

| Tool | What the agent gets |
| --- | --- |
| `crawl_summary` | Counts: URLs, status codes, indexability, findings by severity |
| `issue_summary` | Every finding by code with its tab, filter, URL count and examples |
| `query_urls` | The dashboard's own tab + filter query, paged and sortable |
| `get_url` | Everything measured about one page |
| `cluster_issues` | Root-cause clusters with scores (labelled inferred) |
| `action_plan` | The ordered plan with generated markup and verification steps |
| `recheck_url` | Fetches one URL **live** now, to confirm a fix landed |
| `compare_crawls` | Findings resolved, introduced or changed between two crawls |
| `crawl_site` | Runs a bounded live crawl (robots-respecting, rate-limited) |
| `list_reports`, `export_report` | The 67 named reports and 196 columns |

Three design rules the server holds to:

- **Measured and inferred are labelled differently.** Clusters and priority
  scores say so in their own output.
- **Ambiguity is refused, not guessed.** Asking for `/layanan/lokasi/bali` when
  `/layanan/lokasi/balikpapan` also matches returns both and asks which.
- **Tool failures are results, not protocol errors.** A bad argument comes back
  as a readable message with the valid values, so the agent can correct itself
  in one turn instead of seeing a transport failure.

---

## AI action plan

```bash
skl example.com --all --save-crawl     # crawl first
skl ai --limit 25 --format both        # then plan
skl ai --narrate --provider openai     # optionally narrated
```

`skl ai` folds findings into root-cause clusters using two signals from the crawl
itself — the URL shape (with variable segments inferred from the data, so
`/blog/<slug>` is one template and not forty) and the shared detail — then scores
them:

```text
impact   = severity x critical_multiplier x sqrt(reach) x importance
priority = 100 x impact, clamped to 1-100
```

`critical_multiplier` is 1.4 for findings that make a URL unusable. Effort is
reported beside the score and drives a separate **Quick wins** ranking
(impact per unit of effort), because effort belongs in planning, not in severity.

Each plan item states what is wrong, why it matters, how to fix it, the generated
markup where the fix is mechanical (a corrected `<title>`, a JSON-LD skeleton
with the missing properties filled, the sitemap lines to append), the exact URLs,
and the command that verifies it afterwards.

Sample from the live run against `www.superkilat.com` (60 URLs, 312 findings):

| # | Score | When | Finding | URLs | Effort |
| --- | --- | --- | --- | --- | --- |
| 1 | 79.3 | now | Server error 500 (`internal_server_error`) | 40 | 5/5 |
| 2 | 37.2 | now | Images with empty alt text (`image_missing_alt_text`) | 17 | 1/5 |
| 3 | 25.0 | next | Non-indexable URL listed in a sitemap | 26 | 3/5 |
| 4 | 25.0 | next | Orphan page (`orphan_page`) | 26 | 2/5 |

The 40 URLs were verified as a real outage, not a crawler artefact, by requesting
them independently outside this tool.

---

## Calibrated decisions with Jev

```bash
export TYPESAFE_API_KEY=...          # from https://console.typesafe.ai/keys
skl jev --check                      # confirms the key and lists the models
skl jev --questions                  # prints every question and threshold, no key needed
skl example.com --jev                # crawl and decide
```

Super Spider uses [Jev](https://docs.typesafe.ai), TypeSafe's System One model, where
its own heuristics are only proxies. Jev does not generate text: it evaluates a
*state* against typed *questions* and returns a *choice*, a *score*, or the
*probability* that a yes/no answer is yes, each with the distribution behind it.
There is nothing to parse, and the model reports when it is unsure.

**What it replaces.** Two findings in this crawler were always proxies for a
judgement:

| Finding | Before | With Jev |
| --- | --- | --- |
| Soft 404 | A phrase list. A page mentioning "not found" in a footer, or an article *about* 404 pages, gets flagged | Asked directly, and a confident "no" **withdraws** the phrase-list finding, with the probability recorded |
| Thin content | A word-count threshold | Asked whether the text would satisfy a visitor arriving from a search result |

It also fills the `Search Intent` and `Page Kind` columns, and flags pages whose
visible text is nothing but site furniture.

**Uncertainty is a result, not a failure.** Three paths, following the model's own
documentation:

- **confident yes** — raise the finding, or confirm one the heuristics raised
- **confident no** — raise nothing, and withdraw the proxy finding if there was one
- **in the middle band, or under the confidence floor** — raise nothing, set no
  column, and record that the question went unanswered, with its probability

That last path is the point. A soft 404 guessed from a keyword list is worse than no
soft 404 at all, because somebody has to disprove it. Every threshold lives in
`superspider/jev/questions.py` — one file, so the questions and the cut-offs are the
thing you review rather than something buried in the plumbing.

**One request per page, not one per question.** All five questions go in a single
call and are evaluated in parallel against the same page, so adding a question adds
no round trip. Each response reports the versioned model that answered
(`jev-1.13.0`, not the `jev-latest` alias that was sent) and the crawl stores it, so
you can tell which version produced each decision, and pin the version once you have
tuned the thresholds.

**Two honest caveats.** The model's documentation states that English is its primary
training language and that other languages are handled but not equally well, so on a
non-English site validate these thresholds against your own content first; the
confidence band is how the crawler declines to guess. And a typed answer guarantees
the interface, not the truth: a decision is a judgement to review, not a measurement.

---

## Quick start

Two commands do everything you normally want:

```bash
skl example.com --go     # crawl the site, save everything, open the dashboard
skl                      # reopen the dashboard for the newest crawl
```

`--go` is shorthand for `--all --open`: every report, both sitemaps, the crawl
database and the data modules, written into `crawls/<host>/`, then the dashboard
opens in your browser.

### Making `skl` runnable

`skl.bat` is a Windows launcher that needs no installation, but it has to be
found first. Three ways, shortest setup last:

| How | Command | Notes |
| --- | --- | --- |
| From the project folder | `.\skl example.com --go` | works immediately; the `.\` is required because PowerShell does not search the current directory for executables |
| Install the package | `superspider example.com --go` | after `pip install -e .`, a real console script goes on your PATH and works from anywhere |
| Add the folder to PATH | `skl example.com --go` | run `setup-path.bat`, then open a new terminal |

Two launchers do the same thing: `skl.bat` works from cmd.exe and PowerShell
alike, and `skl.ps1` is the PowerShell-native form that skips the cmd.exe round
trip. Both set `PYTHONPATH` themselves, so neither needs an install.

`setup-path.bat` adds this folder to your **user** PATH. It prints the folder, the
current value and the proposed new value, then asks before writing. It never
touches the system PATH, and it reads only the user value — the usual
`setx PATH "%PATH%;..."` form is avoided because `%PATH%` is the *combined*
system and user value, so it duplicates every system entry into your user PATH.
Running it again is safe: it reports that the folder is already present and
changes nothing. To remove the entry, edit your user PATH in
*System Properties → Environment Variables*.

On macOS and Linux there is no launcher; `python -m superspider ...` always
works, and `pip install -e .` gives you `superspider` on your PATH.

> **Why `skl` and not `sp`?** PowerShell already defines `sp` as an alias for
> `Set-ItemProperty`, so a command named `sp` is never reached — the alias wins.
> `skl` is free, and `skl.bat` explains this in its own header.

| Shortcut | Meaning |
| --- | --- |
| `skl example.com` | the `crawl` subcommand is implied |
| `skl urls.txt` | an existing file of URLs means `list urls.txt` |
| `skl example.com --go` | `--all --open`, with output in `crawls/example.com/` |
| `skl example.com --all` | every report, both sitemaps, the database, the data modules |
| `skl example.com --open` | open the dashboard when the crawl finishes |
| `skl example.com --xlsx` | one consolidated Excel workbook instead of CSV files |
| `skl serve` | dashboard for the newest `crawl.db` found nearby |
| `skl ai` | a prioritised repair plan from the newest crawl |
| `skl mcp` | MCP server so an agent can drive the crawl |

A bare host gets `https://`. Two exceptions, because they are almost always
development servers: a local host (`localhost`, `127.0.0.1`) or any explicit
non-443 port gets `http://`. So `skl 127.0.0.1:3000 --go` just works.

---

## Command line

```bash
skl example.com --go                    # crawl everything, then open the dashboard
skl https://example.com                 # crawl and save the summary reports
skl example.com --all --xlsx            # full reports as one workbook
skl urls.txt --all                      # crawl every URL in a list file
skl sitemap https://example.com/sitemap.xml
skl reports                             # list the named reports
skl serve                               # dashboard for the newest crawl
skl serve --load-crawl out/crawl.db --host 0.0.0.0   # reachable from other machines
skl export --load-crawl out/crawl.db --save-report canonicals --format xlsx
```

`superspider` works everywhere the examples write `skl`.

Common options:

```bash
--save-report issues_all,redirects      # comma separated named reports
--save-all-reports                      # every report
--export-tabs response_codes:client_error,images:missing_alt_attribute
--export-format csv|xlsx|json
--export-custom-summary address,title,status_code   # choose the summary columns
--consolidate-spreadsheets              # one workbook, a sheet per report
--create-sitemap --create-images-sitemap --create-hreflang-sitemap
--max-depth 3 --max-urls 5000 --exclude '/tag/,/page/' --include '/produk/'
--render --screenshots --render-mobile
--structured-data --accessibility --carbon --page-speed <KEY>
--config mycrawl.json --project-name "Q4 audit" --save-crawl
```

`superspider crawl --help` lists the full surface.

### Configuration files

`--config` reads a JSON document with any subset of the settings fields:

```bash
superspider crawl https://example.com --config mycrawl.json
```

```json
{
  "crawl_all_subdomains": true,
  "max_workers": 20,
  "crawl_images": false,
  "title_max_chars": 65,
  "near_duplicates_threshold": 0.85,
  "low_content_word_count": 300,
  "segments": [
    {"name": "Produk", "match_type": "starts", "pattern": "https://example.com/produk/"},
    {"name": "Blog", "match_type": "regex", "pattern": "/blog/"}
  ],
  "custom_extractors": [
    {"name": "Price", "kind": "css", "expression": ".price", "extract": "text"}
  ],
  "custom_searches": [
    {"name": "GTM", "pattern": "googletagmanager", "scope": "source"}
  ]
}
```

---

## What it reports

**28 tabs**, each with its own filters: Internal, External, Response Codes, Page
Titles, Meta Description, Meta Keywords, H1, H2, Images, Canonicals, hreflang,
Directives, Security, Sitemaps, Content, URL, Validation, JavaScript, Structured
Data, Accessibility, PageSpeed, Mobile, Pagination, Links, AMP, Search Console,
Analytics, Custom.

**Report columns** are grouped as Address, Indexability, Graph, Content, Images,
Annotations, Directives, Security, Sitemaps, JavaScript, Structured Data,
Accessibility, Issues, PageSpeed, Search Console, Analytics, Link Metrics, AI and
Carbon. The dashboard's column picker exposes all of them.

**Findings** are graded Error, Warning, Opportunity and Info. Every finding maps
to exactly one tab and filter, because the rule is declared once and the tab
registry is read from the same source — a filter can never disagree with the rule
that produces it.

### Crawl behaviour

- Breadth-first frontier with a bounded worker pool, per-host rate limiting and
  `crawl-delay` support.
- `robots.txt` is parsed with longest-match semantics (`*` and `$` supported) and
  reports the exact matching line that blocked a URL.
- Redirects are followed hop by hop, so the chain, the loop and the first-hop
  status are all preserved. A redirect response's body is reused for the
  destination record instead of costing a second request.
- XML sitemaps are discovered from `robots.txt` and conventional paths, followed
  through sitemap indexes, and their members **are crawled** — which is what makes
  orphan detection possible.
- Limits: depth, total URLs, per-depth, folder depth, query parameters, URL
  length, page size, links per URL, redirect hops.
- Segments split results by URL pattern.

### Analysis

- **Indexability** is decided in the order search engines apply it: response code,
  robots block, meta refresh, then noindex, then canonicalisation.
- **Duplicates**: exact matching on the normalised body hash, and near duplicates
  via 64-bit SimHash over word shingles compared by Hamming distance, indexed with
  banded LSH so the comparison stays near-linear.
- **Link graph**: inlinks, unique inlinks, outlinks, crawl depth and a link score
  computed as `100 × Σ(1 / (1 + source depth)) / max` over unique follow inlinks.
  The formula is stated here because it is an approximation, not a reproduction of
  a proprietary metric.
- **hreflang**: language-code validation against ISO 639-1, self-reference, the
  `x-default` requirement, reciprocal return links across the crawl, and whether
  each target is 200 and canonical.
- **Content**: word counts, text-to-code ratio, Flesch reading ease with its named
  bands, sentence statistics, deterministic grammar checks (repeated words, double
  spaces, space before punctuation, lowercase sentence starts, unbalanced
  brackets), lorem-ipsum and soft-404 detection.
- **Structured data**: JSON-LD, Microdata and RDFa extraction with two independent
  validation passes — schema.org required properties, and Google's additional
  requirements for rich result eligibility.
- **AMP**: structural conformance (AMP attribute, runtime script, boilerplate,
  required charset/viewport/canonical, disallowed tags, inline CSS size) plus the
  two-way link between a page and its `amphtml` variant.
- **Title and description pixel widths** are estimated from a character-advance
  model of an Arial-like face. The estimator intentionally errs towards "too
  long", and the report says it is an estimate.

### Optional modules

| Module | What it adds |
| --- | --- |
| **Rendering** | Chromium loads each page; the rendered DOM is re-parsed and diffed against the raw HTML. Records Rendered HTML Title/Meta/H1/Word Count, word-count change, JS errors/warnings/info, blocked resources, and the "only in rendered HTML" / "updated by JavaScript" findings. Optional full-page screenshots. |
| **Accessibility** | Runs axe-core when you supply `axe.min.js` (`SUPERSPIDER_AXE_PATH`); otherwise a 20-rule built-in set (image alt, labels, lang, title, heading order, link/button names, frame titles, duplicate ids, tabindex, viewport zoom, a simplified contrast check, landmarks). The record says which engine produced the result. |
| **PageSpeed** | PageSpeed Insights v5: category scores, LCP/CLS/TBT/FCP/SI/TTFB/DOM size/byte weight, opportunities, and field-data categories. Bounded by `page_speed_max_urls` (default 50). |
| **Carbon** | Sustainable Web Design energy split (0.15 data centre + 0.60 network + 1.05 device kWh/GB) × grid intensity, with the model's per-view A+ … F bands. Green hosting discounts only the data-centre share. |
| **Search Console** | Search analytics by page (clicks, impressions, CTR, position) and optional URL Inspection (coverage, indexing, robots, canonical, rich results, mobile, AMP). |
| **Analytics** | GA4 sessions, users, pageviews, bounce rate, session duration and conversions, joined by page path. |
| **AI** | Intent, sentiment, language and alt-text columns via OpenAI, Anthropic, Gemini or a local Ollama; embeddings for semantic similarity and near-duplicate detection. Bounded per crawl. |
| **Custom** | XPath, CSS or regex extraction as extra columns; saved regex searches with match counts. |

---

## Architecture

```
superspider/
  config.py        settings surface (crawl, extraction, limits, thresholds, integrations)
  urlutil.py       URL normalisation, classification, path analysis
  model.py         UrlRecord (the report row) and the issue vocabulary
  columns.py       column registry + tab/filter tree (the single source of truth)
  fetch.py         HTTP with explicit redirect chains, retries, rate limiting
  robots.py        robots.txt parsing and RFC 9309 matching
  sitemap.py       sitemap parsing, generation and limits
  extract.py       HTML extraction (head, headings, text, links, images, forms, security)
  crawl.py         phased orchestrator: fetch → graph → indexability → issues
  store.py         SQLite persistence
  export.py        named reports, CSV/XLSX/JSON, sitemap output
  cli.py           argparse interface
  analysis/
    indexability.py  indexability precedence
    issues.py        per-URL rules
    site_rules.py    aggregate rules (duplicates, orphans, canonicals, hreflang, sitemaps)
    duplicates.py    exact + SimHash/LSH near-duplicate detection
    content.py       readability, grammar, spelling
    structured.py    JSON-LD/Microdata/RDFa validation
    amp.py           AMP conformance
  ai/
    clusters.py      root-cause clustering (URL shape inferred from the data)
    priority.py      the documented scoring formula and quick-wins ordering
    plan.py          fix catalogue, markup generation, markdown/JSON plan
    narrate.py       optional model-written summary over measured findings
  mcp/
    server.py        MCP stdio server, twelve tools over one crawl
  modules/
    render.py  a11y.py  pagespeed.py  carbon.py  integrations.py  language.py
  web/
    server.py  static/   local dashboard (stdlib HTTP server + vanilla JS)
```

The crawl runs in four phases so that cheap per-page facts and site-wide
conclusions never contradict each other: fetch and extract, then graph and
identity, then indexability, then issues. Site-wide rules run last because a
duplicate or an orphan cannot be decided from one page.

---

## Testing

```bash
pip install -e ".[dev]"     # or: pip install pytest
python -m pytest            # 441 tests, about 7 seconds
```

The suite runs against a local fixture site (`tests/fixture_site.py`) that serves
deliberately broken pages — a two-hop redirect chain, 404s, a robots.txt block, a
sitemap with an orphan, byte-identical pages, a page canonicalised elsewhere,
broken JSON-LD, an AMP page — so the crawl tests exercise the real network path
rather than a stubbed fetcher. The MCP tests drive the server over in-memory
streams, including the UTF-8 wire check.

Writing it paid for itself immediately: it found bugs that no amount of reading
the code had surfaced.

| Bug the tests found | Consequence before the fix |
| --- | --- |
| An empty `Disallow:` in robots.txt produced the pattern `^`, matching every path | Any site with a bare `Disallow:` would have been crawled as fully blocked |
| `is_canonical_target` was set by a *self*-referencing canonical | The sitemap-orphan rule was dead for every correctly canonicalised site. On the live crawl it went from 0 to 21 real findings |
| `_orphans` ran before sitemap membership was populated | Sitemap orphans were misclassified as plain orphans |
| A truncated `.xml.gz` raised `EOFError`, which was not caught | A cut sitemap file would have killed the crawl |
| `in_sitemap` was unset when the per-URL rules ran | The "non-indexable URL in a sitemap" per-URL rule never fired |
| `load_records` never restored outlinks from the `link` table | `export --load-crawl` produced an empty `inlinks.csv`, and the MCP link graph was empty |
| `<div>` inside `<head>` was undetectable from the parsed tree | The invalid-head-elements rule could never fire |
| Title-only-in-rendered-HTML was reported as "title missing" | A vaguer finding replaced the precise one |
| URL rules were skipped for non-200 responses | The URL tab silently omitted every broken URL |
| A crawled hreflang target with no annotation back was called "unlinked" | The missing-return-link rule rarely fired on real sites |
| Sorting put missing values first when the order was reversed | `get_url` exposed no outlinks at all |

---

## Verification

The test fixture (a local HTTP server with deliberately broken pages) exercises
robots blocking, a two-hop redirect chain, 404s, sitemap membership, orphans,
exact and near duplicates, canonicalisation to another URL, missing/duplicate/multiple
titles and H1s, missing and empty image alt, JSON-LD validation (including a parse
error and a missing `mainEntity`), AMP conformance, and carbon estimation. It
crawled 23 URLs and produced 70 reports plus a sitemap and a crawl database; the
sample output is in `example-output/`.

The tool was then pointed at a live site, which produced the corrections listed
under Testing above — including a 40-URL server outage that was verified as real
by requesting those URLs independently, outside this tool.

---

## Limitations

- Title and description pixel widths are estimates from a character-advance
  model, not measurements from the engine's own font stack.
- The link score is a documented approximation, not the reference tool's metric.
- Grammar checks are deterministic pattern rules, not a full grammar engine;
  spelling needs a word list you supply.
- The built-in accessibility rules are a documented subset of axe-core, and the
  contrast check is a simplification of the WCAG algorithm.
- JavaScript rendering diffs the DOM; it does not attribute individual findings
  to specific scripts.
- URL Inspection is capped at Google's documented daily quota, PageSpeed and AI
  calls are capped per crawl, and both are recorded on the record when skipped.
- No user-agent rotation, no proxy support, and no distributed crawling.