Super Spider

A crawling and technical SEO auditing tool with a local web dashboard, built in Python. It crawls a site, extracts the signals that decide whether and how pages can rank, raises findings at four severities, and reports everything to CSV, Excel or JSON — or to a browser dashboard organised the way a technical SEO auditor reads a crawl.

It also speaks the Model Context Protocol, so an agent can query a crawl, verify a fix, and diff two crawls instead of guessing about a site it never fetched.

skl example.com --go

The dashboard runs at http://127.0.0.1:8765 and is served by the standard library only, so there is no extra runtime to install.

What this is, and what it is not

Worth reading before you judge the numbers it produces.

What it is. An independent implementation of the crawl-and-audit loop. It checks a real site over the network, stores what it measured, diffs two crawls, and can hand the whole thing to an agent over MCP. Everything it reports traces to a request it actually made.

What it is not. It is not Screaming Frog, and it does not try to be. The feature surface was designed by reading the reference product's own resource bundles (see below), but the code is its own, the metric formulas are its own and are documented, and a few areas are deliberately narrower:

• The link score is a documented approximation, not PageRank and not the reference product's metric.

• Title and description pixel widths are estimates from a character-advance model, not measurements from a search engine's font stack.

• The built-in accessibility rules are a documented subset, not axe-core, unless you supply axe.min.js — and the report says which engine ran.

• Spelling needs a word list you provide. Without one it reports "not measured", never zero errors.

• PageSpeed, Search Console, GA4 and AI columns need credentials. Without them the affected field records why, and the crawl prints which modules could not run.

Where a question cannot be answered from the data, the field says so. That is a deliberate rule rather than a limitation to work around, because an empty column that looks like a clean result is the most expensive kind of wrong.

Related MCP server: crawlcove-mcp

Where the feature set came from

The scope was not guessed. A copy of Screaming Frog SEO Spider 22.2 was present on the development machine, and its shipped ScreamingFrogSEOSpider.jar was inspected read-only (open_binary → inspect_artifact → inventory paging), which established:

No code was copied. The extracted evidence describes a feature surface — which columns exist, which settings exist. Every rule, parser, formula and report here is an independent implementation written against that surface, and the user-facing wording is original. The parsing, extraction, indexability, dedup, planning and MCP layers are all first-party code.

Install

pip install -e .              # core: requests, lxml, openpyxl
pip install -e ".[dev]"       # adds pytest
pip install -e ".[render]"    # adds Playwright for JS rendering and axe audits
playwright install chromium   # the browser binary Playwright drives

A crawl finishes by printing every enabled module that could not run and why, so a missing key shows up as a warning rather than as a clean-looking empty column.

Why this exists in an agentic era

A language model cannot fetch ten thousand URLs, cannot follow a redirect chain hop by hop, cannot verify that every page in an hreflang cluster annotates the others, and cannot prove a claim about a page it never downloaded. It is a reasoner without instruments.

So the crawler does not compete with the agent, it equips it. Three things a model on its own cannot provide:

1. Ground truth. Status codes, redirect chains, byte sizes, link counts and the exact matching robots.txt line are measured, not recalled. Every number in this tool traces to a request that was made.

2. Scale and patience. Sixty URLs politely, respecting crawl-delay; or sixty thousand. An agent would burn its context window on the first five.

3. A stable target. Crawls are stored, so two runs can be diffed. That is what turns "I changed the template" into "the finding is gone", without trusting anyone's summary — including the model's.

The AI layer sits on top and is deliberately narrow. It clusters findings that share a root cause, scores them with a formula you can audit, and drafts the markup the fix needs. It is not asked to find problems; it is handed the measurements and asked to explain them. When no model is configured, the plan is still produced — smaller, but never fabricated.

An earlier version of the scoring divided by an "effort" term. On a real crawl that ranked a 26-page server outage below a missing alt attribute on 17 pages, because "fix the server" scored as hard work. The formula no longer contains effort, and the incident in that example now ranks first at 79/100. That is the kind of correction that only shows up when the tool is pointed at a live site.

Agent interface (MCP)

superspider mcp runs a Model Context Protocol server on stdio, exposing the crawl as twelve inspectable tools. Point any MCP client at it:

{
  "mcpServers": {
    "superspider": {
      "command": "python",
      "args": ["-m", "superspider", "mcp", "--load-crawl", "crawls/example.com/crawl.db"]
    }
  }
}

Three design rules the server holds to:

• Measured and inferred are labelled differently. Clusters and priority scores say so in their own output.

• Ambiguity is refused, not guessed. Asking for /layanan/lokasi/bali when /layanan/lokasi/balikpapan also matches returns both and asks which.

• Tool failures are results, not protocol errors. A bad argument comes back as a readable message with the valid values, so the agent can correct itself in one turn instead of seeing a transport failure.

AI action plan

skl example.com --all --save-crawl     # crawl first
skl ai --limit 25 --format both        # then plan
skl ai --narrate --provider openai     # optionally narrated

skl ai folds findings into root-cause clusters using two signals from the crawl itself — the URL shape (with variable segments inferred from the data, so /blog/<slug> is one template and not forty) and the shared detail — then scores them:

impact   = severity x critical_multiplier x sqrt(reach) x importance
priority = 100 x impact, clamped to 1-100

critical_multiplier is 1.4 for findings that make a URL unusable. Effort is reported beside the score and drives a separate Quick wins ranking (impact per unit of effort), because effort belongs in planning, not in severity.

Each plan item states what is wrong, why it matters, how to fix it, the generated markup where the fix is mechanical (a corrected <title>, a JSON-LD skeleton with the missing properties filled, the sitemap lines to append), the exact URLs, and the command that verifies it afterwards.

Sample from the live run against www.superkilat.com (60 URLs, 312 findings):

The 40 URLs were verified as a real outage, not a crawler artefact, by requesting them independently outside this tool.

Calibrated decisions with Jev

export TYPESAFE_API_KEY=...          # from https://console.typesafe.ai/keys
skl jev --check                      # confirms the key and lists the models
skl jev --questions                  # prints every question and threshold, no key needed
skl example.com --jev                # crawl and decide

Super Spider uses Jev, TypeSafe's System One model, where its own heuristics are only proxies. Jev does not generate text: it evaluates a state against typed questions and returns a choice, a score, or the probability that a yes/no answer is yes, each with the distribution behind it. There is nothing to parse, and the model reports when it is unsure.

What it replaces. Two findings in this crawler were always proxies for a judgement:

It also fills the Search Intent and Page Kind columns, and flags pages whose visible text is nothing but site furniture.

Uncertainty is a result, not a failure. Three paths, following the model's own documentation:

• confident yes — raise the finding, or confirm one the heuristics raised

• confident no — raise nothing, and withdraw the proxy finding if there was one

• in the middle band, or under the confidence floor — raise nothing, set no column, and record that the question went unanswered, with its probability

That last path is the point. A soft 404 guessed from a keyword list is worse than no soft 404 at all, because somebody has to disprove it. Every threshold lives in superspider/jev/questions.py — one file, so the questions and the cut-offs are the thing you review rather than something buried in the plumbing.

One request per page, not one per question. All five questions go in a single call and are evaluated in parallel against the same page, so adding a question adds no round trip. Each response reports the versioned model that answered (jev-1.13.0, not the jev-latest alias that was sent) and the crawl stores it, so you can tell which version produced each decision, and pin the version once you have tuned the thresholds.

Two honest caveats. The model's documentation states that English is its primary training language and that other languages are handled but not equally well, so on a non-English site validate these thresholds against your own content first; the confidence band is how the crawler declines to guess. And a typed answer guarantees the interface, not the truth: a decision is a judgement to review, not a measurement.

Quick start

Two commands do everything you normally want:

skl example.com --go     # crawl the site, save everything, open the dashboard
skl                      # reopen the dashboard for the newest crawl

--go is shorthand for --all --open: every report, both sitemaps, the crawl database and the data modules, written into crawls/<host>/, then the dashboard opens in your browser.

Making skl runnable

skl.bat is a Windows launcher that needs no installation, but it has to be found first. Three ways, shortest setup last:

Two launchers do the same thing: skl.bat works from cmd.exe and PowerShell alike, and skl.ps1 is the PowerShell-native form that skips the cmd.exe round trip. Both set PYTHONPATH themselves, so neither needs an install.

setup-path.bat adds this folder to your user PATH. It prints the folder, the current value and the proposed new value, then asks before writing. It never touches the system PATH, and it reads only the user value — the usual setx PATH "%PATH%;..." form is avoided because %PATH% is the combined system and user value, so it duplicates every system entry into your user PATH. Running it again is safe: it reports that the folder is already present and changes nothing. To remove the entry, edit your user PATH in System Properties → Environment Variables.

On macOS and Linux there is no launcher; python -m superspider ... always works, and pip install -e . gives you superspider on your PATH.

Why skl and not sp? PowerShell already defines sp as an alias for Set-ItemProperty, so a command named sp is never reached — the alias wins. skl is free, and skl.bat explains this in its own header.

A bare host gets https://. Two exceptions, because they are almost always development servers: a local host (localhost, 127.0.0.1) or any explicit non-443 port gets http://. So skl 127.0.0.1:3000 --go just works.

Command line

skl example.com --go                    # crawl everything, then open the dashboard
skl https://example.com                 # crawl and save the summary reports
skl example.com --all --xlsx            # full reports as one workbook
skl urls.txt --all                      # crawl every URL in a list file
skl sitemap https://example.com/sitemap.xml
skl reports                             # list the named reports
skl serve                               # dashboard for the newest crawl
skl serve --load-crawl out/crawl.db --host 0.0.0.0   # reachable from other machines
skl export --load-crawl out/crawl.db --save-report canonicals --format xlsx

superspider works everywhere the examples write skl.

Common options:

--save-report issues_all,redirects      # comma separated named reports
--save-all-reports                      # every report
--export-tabs response_codes:client_error,images:missing_alt_attribute
--export-format csv|xlsx|json
--export-custom-summary address,title,status_code   # choose the summary columns
--consolidate-spreadsheets              # one workbook, a sheet per report
--create-sitemap --create-images-sitemap --create-hreflang-sitemap
--max-depth 3 --max-urls 5000 --exclude '/tag/,/page/' --include '/produk/'
--render --screenshots --render-mobile
--structured-data --accessibility --carbon --page-speed <KEY>
--config mycrawl.json --project-name "Q4 audit" --save-crawl

superspider crawl --help lists the full surface.

Configuration files

--config reads a JSON document with any subset of the settings fields:

superspider crawl https://example.com --config mycrawl.json

{
  "crawl_all_subdomains": true,
  "max_workers": 20,
  "crawl_images": false,
  "title_max_chars": 65,
  "near_duplicates_threshold": 0.85,
  "low_content_word_count": 300,
  "segments": [
    {"name": "Produk", "match_type": "starts", "pattern": "https://example.com/produk/"},
    {"name": "Blog", "match_type": "regex", "pattern": "/blog/"}
  ],
  "custom_extractors": [
    {"name": "Price", "kind": "css", "expression": ".price", "extract": "text"}
  ],
  "custom_searches": [
    {"name": "GTM", "pattern": "googletagmanager", "scope": "source"}
  ]
}

What it reports

28 tabs, each with its own filters: Internal, External, Response Codes, Page Titles, Meta Description, Meta Keywords, H1, H2, Images, Canonicals, hreflang, Directives, Security, Sitemaps, Content, URL, Validation, JavaScript, Structured Data, Accessibility, PageSpeed, Mobile, Pagination, Links, AMP, Search Console, Analytics, Custom.

Report columns are grouped as Address, Indexability, Graph, Content, Images, Annotations, Directives, Security, Sitemaps, JavaScript, Structured Data, Accessibility, Issues, PageSpeed, Search Console, Analytics, Link Metrics, AI and Carbon. The dashboard's column picker exposes all of them.

Findings are graded Error, Warning, Opportunity and Info. Every finding maps to exactly one tab and filter, because the rule is declared once and the tab registry is read from the same source — a filter can never disagree with the rule that produces it.

Crawl behaviour

• Breadth-first frontier with a bounded worker pool, per-host rate limiting and crawl-delay support.

• robots.txt is parsed with longest-match semantics (* and $ supported) and reports the exact matching line that blocked a URL.

• Redirects are followed hop by hop, so the chain, the loop and the first-hop status are all preserved. A redirect response's body is reused for the destination record instead of costing a second request.

• XML sitemaps are discovered from robots.txt and conventional paths, followed through sitemap indexes, and their members are crawled — which is what makes orphan detection possible.

• Limits: depth, total URLs, per-depth, folder depth, query parameters, URL length, page size, links per URL, redirect hops.

• Segments split results by URL pattern.

Analysis

• Indexability is decided in the order search engines apply it: response code, robots block, meta refresh, then noindex, then canonicalisation.

• Duplicates: exact matching on the normalised body hash, and near duplicates via 64-bit SimHash over word shingles compared by Hamming distance, indexed with banded LSH so the comparison stays near-linear.

• Link graph: inlinks, unique inlinks, outlinks, crawl depth and a link score computed as 100 × Σ(1 / (1 + source depth)) / max over unique follow inlinks. The formula is stated here because it is an approximation, not a reproduction of a proprietary metric.

• hreflang: language-code validation against ISO 639-1, self-reference, the x-default requirement, reciprocal return links across the crawl, and whether each target is 200 and canonical.

• Content: word counts, text-to-code ratio, Flesch reading ease with its named bands, sentence statistics, deterministic grammar checks (repeated words, double spaces, space before punctuation, lowercase sentence starts, unbalanced brackets), lorem-ipsum and soft-404 detection.

• Structured data: JSON-LD, Microdata and RDFa extraction with two independent validation passes — schema.org required properties, and Google's additional requirements for rich result eligibility.

• AMP: structural conformance (AMP attribute, runtime script, boilerplate, required charset/viewport/canonical, disallowed tags, inline CSS size) plus the two-way link between a page and its amphtml variant.

• Title and description pixel widths are estimated from a character-advance model of an Arial-like face. The estimator intentionally errs towards "too long", and the report says it is an estimate.

Optional modules

Architecture

superspider/
  config.py        settings surface (crawl, extraction, limits, thresholds, integrations)
  urlutil.py       URL normalisation, classification, path analysis
  model.py         UrlRecord (the report row) and the issue vocabulary
  columns.py       column registry + tab/filter tree (the single source of truth)
  fetch.py         HTTP with explicit redirect chains, retries, rate limiting
  robots.py        robots.txt parsing and RFC 9309 matching
  sitemap.py       sitemap parsing, generation and limits
  extract.py       HTML extraction (head, headings, text, links, images, forms, security)
  crawl.py         phased orchestrator: fetch → graph → indexability → issues
  store.py         SQLite persistence
  export.py        named reports, CSV/XLSX/JSON, sitemap output
  cli.py           argparse interface
  analysis/
    indexability.py  indexability precedence
    issues.py        per-URL rules
    site_rules.py    aggregate rules (duplicates, orphans, canonicals, hreflang, sitemaps)
    duplicates.py    exact + SimHash/LSH near-duplicate detection
    content.py       readability, grammar, spelling
    structured.py    JSON-LD/Microdata/RDFa validation
    amp.py           AMP conformance
  ai/
    clusters.py      root-cause clustering (URL shape inferred from the data)
    priority.py      the documented scoring formula and quick-wins ordering
    plan.py          fix catalogue, markup generation, markdown/JSON plan
    narrate.py       optional model-written summary over measured findings
  mcp/
    server.py        MCP stdio server, twelve tools over one crawl
  modules/
    render.py  a11y.py  pagespeed.py  carbon.py  integrations.py  language.py
  web/
    server.py  static/   local dashboard (stdlib HTTP server + vanilla JS)

The crawl runs in four phases so that cheap per-page facts and site-wide conclusions never contradict each other: fetch and extract, then graph and identity, then indexability, then issues. Site-wide rules run last because a duplicate or an orphan cannot be decided from one page.

Testing

pip install -e ".[dev]"     # or: pip install pytest
python -m pytest            # 441 tests, about 7 seconds

The suite runs against a local fixture site (tests/fixture_site.py) that serves deliberately broken pages — a two-hop redirect chain, 404s, a robots.txt block, a sitemap with an orphan, byte-identical pages, a page canonicalised elsewhere, broken JSON-LD, an AMP page — so the crawl tests exercise the real network path rather than a stubbed fetcher. The MCP tests drive the server over in-memory streams, including the UTF-8 wire check.

Writing it paid for itself immediately: it found bugs that no amount of reading the code had surfaced.

Verification

The test fixture (a local HTTP server with deliberately broken pages) exercises robots blocking, a two-hop redirect chain, 404s, sitemap membership, orphans, exact and near duplicates, canonicalisation to another URL, missing/duplicate/multiple titles and H1s, missing and empty image alt, JSON-LD validation (including a parse error and a missing mainEntity), AMP conformance, and carbon estimation. It crawled 23 URLs and produced 70 reports plus a sitemap and a crawl database; the sample output is in example-output/.

The tool was then pointed at a live site, which produced the corrections listed under Testing above — including a 40-URL server outage that was verified as real by requesting those URLs independently, outside this tool.

Limitations

• Title and description pixel widths are estimates from a character-advance model, not measurements from the engine's own font stack.

• The link score is a documented approximation, not the reference tool's metric.

• Grammar checks are deterministic pattern rules, not a full grammar engine; spelling needs a word list you supply.

• The built-in accessibility rules are a documented subset of axe-core, and the contrast check is a simplification of the WCAG algorithm.

• JavaScript rendering diffs the DOM; it does not attribute individual findings to specific scripts.

• URL Inspection is capped at Google's documented daily quota, PageSpeed and AI calls are capped per crawl, and both are recorded on the record when skipped.

• No user-agent rotation, no proxy support, and no distributed crawling.