Toadstone

A toadstone was a gem people wore to detect poison. It never detected anything. This one is named after that, on purpose.

▶  Open the live demo

The real interface, frozen over a real crawl. Click the tabs. Open a finding. No sign-up.

🎬  Watch the tour

Two minutes. Why it exists, what it found, and every view in the app.

Why this exists · Quick start · What it finds · Screenshots · How it compares · For agents · Architecture · Security · FAQ

Why this exists

I run an SEO team of 22 people. Screaming Frog is licensed per machine.

That is not a complaint about the price — it is a good tool and worth its licence. It is that the shape is wrong for how we work. We are in Ahrefs every day. We need a full technical crawl maybe five or six times a month. Paying a seat for every executive so each of them can run a crawl a handful of times a month is paying for the wrong thing.

What I wanted was one instance I host myself, on my own Mac, that the whole team can reach — and that Claude Code can talk to directly over MCP, so a crawl comes back as a list of issues to fix rather than a CSV someone has to open, read and turn into tickets.

Nothing available had that shape. So I forked an open-source crawler and started building.

claude mcp add toadstone -- python -m src.mcp

That is the whole integration. Ten tools, read-only by default, unmetered — because they are your crawls, on your machine. "Crawl freshlycited.com and tell me what to fix" comes back as findings grouped by check, each with its evidence and a one-line fix, ready to become tickets. No credits, no units, no per-seat anything. See the tools →

Then I found something I was not looking for

Most SEO crawlers advertise how many checks they have. That number is almost meaningless, and here is why.

I counted how many of the ones I had inherited had ever been observed to fire on a real page. There were about 225. Eighty-four of them had ever fired. The rest passed their unit tests and sat inert:

• eight AMP checks that could not fire on any site, because the markup they read was discarded before they ran;

• every PDF check, dead from the moment JavaScript rendering was switched on;

• a raw-versus-rendered comparison that had been comparing a document against itself;

• a failed PageSpeed run that reported as a clean bill of performance — the API returned a quota error, the integration swallowed it, and the report came back with no performance issues at all;

• three "checks" that were not checks, only sample data left in a test file.

None of that looked broken. Every report came back clean and confident, and clean was the wrong answer.

So the project built a site that is wrong in as many ways as possible, crawls it with the real crawler, and records which check slugs actually appear.

The last sweep, run while this README was being written, fired 180 distinct checks across the fixture's six modes — 162 in the default run alone, on 77 crawled pages. Here is that sweep, in full:

python tools/coverage_sweep.py

--- default                 77 pages crawled, 162 checks fired
--- sitemap-limits-1         3 pages crawled,  76 checks fired   50,001 URLs, 51 children
--- sitemap-limits-2         3 pages crawled,  70 checks fired   a sitemap listing nothing
--- sitemap-limits-3         3 pages crawled,  75 checks fired   52MB of sitemap
--- silent-image            85 pages crawled, 166 checks fired   a handler that answers nothing
--- https                    4 pages crawled,  30 checks fired   the checks that need TLS

180 distinct checks fired across 6 mode(s)

Each mode needs a crawl of its own — a sitemap of 50,001 URLs cannot share a run with one that works — and the sweep keeps a separate slug log per mode, because the harness truncates its log at the start of every run. (Six runs pointed at one file gives you the sixth mode's slugs and a confidently wrong total. That mistake was made while writing this, and it reported 30.)

BUILD-LOG.md carries the rest of the accounting day by day — including the thirteen checks that rest on a stubbed transport with real parsing either side, the one that rests on a model's verdict being supplied rather than asked for, the two that cannot honestly be reached at all, and every wrong turn taken on the way. It is written in the first person and it does not flatter the work.

Those commands are the whole argument. Run them before you trust anything above.

Related MCP server: seo-agent

What it is

A self-hosted SEO site crawler with a browser UI, an HTTP API, and an MCP server so an AI agent can query crawls directly. It renders JavaScript, keeps crawls so it can tell you what changed since the last one, and reports findings with evidence a developer can act on without re-fetching the page.

It is a fork of LibreCrawl by Phiality, with roughly a hundred commits of checks, integrations, API, agent support and UI work on top. See FORK.md for exactly what the fork adds.

No per-URL pricing, no seats, no cloud. Point it at a site you have permission to crawl and it runs on your machine.

What a finding looks like

Every finding, wherever it surfaces — UI, API, export, MCP — carries the same shape. This one is real, from the crawl in the screenshot above:

{
  "id": "69230cd05f9f7ca9ec7a66ee0f91a85ccf9f0d90",
  "check": "links_to_broken_urls",
  "type": "error",
  "category": "Technical",
  "issue": "Page Links to Broken URLs",
  "url": "http://toadstone-demo.localtest.me:8977/",
  "details": "1 link(s) on this page are broken: http://toadstone-demo.localtest.me:8977/missing-entirely.png (404)",
  "evidence": {
    "found": "1 broken links",
    "related_urls": ["http://toadstone-demo.localtest.me:8977/missing-entirely.png"]
  },
  "how_to_fix": "Update or remove the link; a visitor following it hits an error.",
  "docs_url": "/checks#links_to_broken_urls",
  "status": "open",
  "first_seen_crawl": 2
}

Two things in there do more work than they look like they do:

• id is sha1(url + check + offending value). The same defect keeps the same identity between crawls. A fixed issue never comes back as "new", an accepted one stays accepted, and first_seen_crawl tells you when it appeared. Most tools re-number their findings every run, which makes "what changed?" unanswerable.

• check is a slug, never free text. You can filter on it, alert on it, and diff on it. docs_url points at the generated reference page for it.

Watch it work

Open this crawl in the live demo → — the real interface, not a report. Click anything; starting a new crawl is the one thing it cannot do. It is the actual interface, frozen over a real crawl — the same templates, stylesheet and JavaScript, with every API response captured and replayed. Tabs switch, filters filter, findings expand, the command palette opens. Starting a new crawl is the one thing it cannot do. Built by tools/build_demo_app.py.

What it finds

222 checks, 180 of them observed firing in the last sweep. Every one has a slug, a severity, a category, an evidence block and a one-line fix. The running install generates its own reference page at /checks, built from what it has actually reported rather than from a hand-kept list that drifts.

Full reference, slug by slug, with the ✅ against every one that has been watched to fire: docs/checks.md. It is generated from the running app's own catalogue by tools/generate_checks_doc.py, so it cannot drift from the wording the code uses — and it lists the 59 that fired in the harness but have no description yet, rather than quietly leaving them out.

See it

Every screenshot below is the real app on a clean install, crawling a real site — freshlycited.com, which we own, so the URLs and the findings are genuine rather than staged. They are regenerated by a script, and each one is checked for personal data before it is written.

What is wrong

Every check that fired, with its slug and how many pages it hit. Collapse the groups and the whole shape of a site's problems fits on one screen.

This one is the bundled fixture — the site in tests/ built to be wrong in as many ways as possible — because a healthy site makes a dull screenshot and an honest one. 66 pages, 488 findings. The hero above is the same screen on a real site that is in good shape; this is the same screen on one that is not.

Grouped by check, because forty pages missing a canonical is one job, not forty. Filter by severity, response code, content type, section of the site, or what is new since any previous crawl.

What was found

How it fits together

Getting around

And one tab that is not about your site

About sits at the end of the strip and tells the story — what a toadstone was, why a crawler is named after a gem that detected nothing, and what the audit found on the day this fork got its name. The last panel is live: it asks your install how many distinct checks it has actually reported, so the claim on screen is about your crawls rather than about somebody's marketing.

It is a plugin (web/static/plugins/about.js, ~180 lines) — the shortest worked example of the plugin API, which is documented in docs/plugins.md if you want a tab of your own.

And the licence, in plain words

A source-available licence has terms people need to know before they build on it, and "it's in LICENSE in the repo" is not where somebody running the app will look. Licence sits next to About: what you may do, what you may not, why the upstream MIT grant is what makes the relicence possible, and every third-party licence that travels with the software — with the two copyleft ones marked, because one of them surprised us.

Dark mode, properly

Quick start

Docker Compose (recommended)

git clone https://github.com/spcaeo/toadstone
cd toadstone
cp .env.example .env            # set SECRET_KEY, and a port if 5000 is taken
docker compose up -d
open http://localhost:5000

Docker, without Compose

docker build -t toadstone .
docker run -d --name toadstone -p 5000:5000 \
  -v "$PWD/data:/app/data" \
  -e SECRET_KEY="$(python3 -c 'import secrets;print(secrets.token_hex(32))')" \
  -e LOCAL_MODE=true \
  --shm-size=2g toadstone

--shm-size=2g matters: Chromium crashes in a container with the default 64MB of shared memory, and the crawl fails in ways that look like network problems.

Python, no container

git clone https://github.com/spcaeo/toadstone
cd toadstone
pip install -r requirements.txt
playwright install chromium
python main.py --local --port 5057

On a Mac, pass --port. macOS gives port 5000 to AirPlay Receiver, which answers every request with 403 AirTunes. Everything looks like it started and nothing works. Either pass another port or turn off System Settings → General → AirDrop & Handoff → AirPlay Receiver.

One command, no UI (for CI)

python main.py --crawl https://example.com --fail-on error --report issues.json

Exits non-zero when the crawl finds an open issue at that severity or worse, so a pipeline can block a deploy on it.

What you need

Integrations

Every one of these is optional, off by default, and says when it could not run rather than returning nothing and letting that read as health.

For agents: MCP, API and webhooks

Toadstone was built to be read by software as well as people.

MCP server

// Claude Code, Cursor, or any MCP client
{
  "mcpServers": {
    "toadstone": {
      "command": "python",
      "args": ["-m", "src.mcp", "--url", "http://localhost:5000"],
      "env": { "TOADSTONE_MCP_URL": "http://localhost:5000" }
    }
  }
}

It is read-only by default, restricted to hosts this install has already crawled, and capped at 2,000 URLs per crawl — defaults chosen on the assumption that nobody will ever re-read the config file they pasted it into. Writes need an explicit opt-in. See docs/mcp.md.

HTTP API

curl 'localhost:5000/api/crawls/133/issues?check=links_to_broken_urls&status=open'
curl 'localhost:5000/api/crawls/133/diff/132'           # new, fixed, still open
curl 'localhost:5000/api/checks'                        # the whole catalogue
curl -X POST 'localhost:5000/api/issues/<id>/accept' -d '{"reason":"by design"}'

Filters, formats (JSON, CSV, XLSX), crawl-to-crawl diffs, and issue acceptance with a reason — an accepted issue stays accepted across crawls, because the id is stable.

Webhooks and scheduling

A crawl can POST its summary when it finishes, and the scheduler runs standing crawls without anyone opening the UI.

How Toadstone compares

Twenty-six tools, every cell traced to a page the vendor owns, all fetched 2026-10-20. nc means not checked — I could not confirm it from the vendor's own material, so it is left blank rather than guessed. A gap is more useful than a wrong tick, and corrections are welcome as a pull request.

No prices in the capability tables, on purpose. Published prices move, third-party aggregators contradict each other and the vendors, and a README full of 2026 figures is misinformation by 2027. Where a vendor publishes a headline price on its own site it is noted in the pricing row; everything else links out.

✓ documented · ◐ partial or conditional · ✗ the vendor's own material indicates absence · nc not confirmable

Desktop and self-hosted crawlers

Cloud and enterprise platforms

The audit module inside an SEO suite

Two rows deserve a caveat rather than a victory lap. No vendor anywhere documents persistent per-finding identity, and webhooks are nc across all twenty-six. The honest claim is "no competitor documents equivalent behaviour" — not that they lack it. And moz.com could not be fetched from the environment this was compiled in, while every Bing Webmaster Tools help URL returned 404, so both are nc throughout and want a human with a browser.

On check counts, including ours

Only Screaming Frog and Sitebulb back their number with a public library you can count against — Sitebulb even enumerates its 98 accessibility hints. Ahrefs' "170+", Semrush's "140+", SE Ranking's "115+" and Lumar's "250+/350+" are marketing-defined with nothing public to count. JetOctopus, Oncrawl, Botify, Conductor, seoClarity, Serpstat and Sitechecker publish no count at all, so any claim that they have fewer checks than this would be indefensible.

Counts are not a common unit either: 222 here and Sitebulb's "300+" do not measure the same thing. The unusual claim is not the count — it is the coverage harness. No vendor publishes anything comparable to "here is the command, run it, watch which checks fire."

An MCP server is no longer a differentiator

Ten of the twenty-six document one, Screaming Frog's being the deepest at 29 tools. What differs is what it may do and what it costs:

Trademarks. Screaming Frog, Sitebulb, Sitemoth, Netpeak Spider, Visual SEO Studio, Greenflare, SEO Macroscope, Botify, Lumar, Oncrawl, JetOctopus, Conductor, seoClarity, Ryte, Ahrefs, Semrush, Moz, SE Ranking, Serpstat, Sitechecker, Google, Search Console, Google Analytics, Chrome, Bing, axe, axe DevTools, WAVE, Pa11y and LibreCrawl are trademarks or registered trademarks of their respective owners. Toadstone is not affiliated with, endorsed by, sponsored by or supported by any of them. Those names are used only to identify the products being compared. The comparisons are drawn from each vendor's own published documentation on the date stated, are offered as fair comment, and may be out of date — a correction is a welcome pull request.

The free baseline

Google Search Console is not a crawler, and that is the point. It tells you what Google chose to crawl, on Google's schedule, capped at 1,000 example rows, with validation that takes about two weeks. You cannot crawl a URL set of your choosing, and it can only ever see production, after the fact — no staging, no pre-release, no diff you control. Toadstone reads Search Console and crawls; the interesting findings are where the two disagree.

Among paid tools the free tiers cluster hard at 500 URLs — Screaming Frog, Netpeak Spider, Visual SEO Studio and SiteAnalyzer all stop there. Sitemoth (unlimited, free, no account) and the GPL projects are the only ones that do not. Toadstone is in that company, not alone in it.

The single-purpose tools

For log files alone, Screaming Frog's Log File Analyser is mature and cheap, free up to 1,000 events. For accessibility alone, axe DevTools, WAVE and Pa11y are better than any crawler's accessibility module, this one included — Pa11y in particular does the CI half properly. Note that Toadstone, Screaming Frog, Sitebulb and Sitemoth all wrap the same engine: axe-core. Claiming accessibility as a differentiator against crawlers is fair; claiming it against accessibility tools is not.

Where others lead

• Screaming Frog — a decade of edge cases, the deepest MCP of anyone, spelling in 25+ languages, and unlimited URLs for one annual licence. Hard to beat on value, and its configuration depth is unmatched when a site is being strange.

• Lumar — stops regressions shipping: 8 CI systems and the ability to block a deploy that fails a health threshold. This finds things after they ship.

• Conductor — latency. Continuous monitoring, alerts the moment something changes, 60 months of page-level changelog. A scheduled crawl finds yesterday's regression.

• Botify — acts rather than reports: its agents and PageWorkers deploy the fix to the live site. The LLM analysts here score pages; they do not change them.

• JetOctopus — unlimited seats on every plan and millions of log lines ingested from a dozen CDN formats, at a price that needs no procurement process.

• Oncrawl — OQL across crawl, log and Search Console data, with exports into BI tools. A different product class from a crawler's filters.

• Sitebulb — explains why a finding matters better than anyone, with a public hint library that is a genuine teaching resource, and ticketing integrations.

• Ahrefs and Semrush — a backlink index and keyword data no self-hosted tool can reproduce. Toadstone has to rent that from Moz, Majestic or Ahrefs.

• Sitemoth — the uncomfortable one. Desktop, free today, 190 checks, axe-core, logs with fake-bot detection, LanguageTool, crawl comparison, and an MCP that can crawl. Whatever the case for Toadstone is, "self-hosted with no per-URL pricing" is not it against this.

• All of them — a company, a roadmap and support. This is a fork maintained by a small team, with no SLA and no phone number. The two dormant GPL crawlers in this survey are the honest illustration of how that can end.

Who should use what

• Crawl data must not leave your infrastructure — NDAs, regulated sectors, staging sites that are not public → Toadstone.

• You want an agent driving the crawler end to end, unmetered, over your own data → Toadstone, or Sitemoth if a desktop app suits you better.

• One or a few practitioners wanting the most capability per pound → Screaming Frog, plus its Log File Analyser.

• The deliverable is a prioritised report a client reads → Sitebulb.

• Millions of URLs, many seats, someone else running the pipeline → JetOctopus, Oncrawl, Botify or seoClarity.

• You need to know within minutes → Conductor. You need to block the deploy → Lumar.

• Site audit beside rank tracking, keywords and backlinks → Ahrefs, Semrush or SE Ranking.

• Accessibility is the actual job → Pa11y in CI plus axe DevTools.

• Toadstone is the wrong choice if you need a backlink index, deploy-time gating, sub-minute alerting, a support contract, millions of URLs, or if nobody will operate Docker and a browser pool. And if the appeal was "free, self-hosted, lots of checks" — evaluate Sitemoth first. It is free today and covers much of the same ground.

Architecture

flowchart LR
    subgraph you["Your machine"]
        UI["Browser UI<br/>11 views + plugins, ⌘K palette"]
        API["HTTP API<br/>JSON · CSV · XLSX"]
        MCP["MCP server<br/>read-only by default"]
    end

    subgraph app["Toadstone"]
        APP["Flask + Waitress"]
        SCHED["Scheduler<br/>standing crawls"]
        CRAWL["Crawl loop<br/>threaded HTTP, or<br/>Playwright for JS"]
        CHECKS["222 checks<br/>every finding via one factory:<br/>stable id, slug, evidence, fix"]
        DB[("SQLite<br/>crawls, issues,<br/>encrypted tokens")]
    end

    subgraph ext["Outside services, all optional, all off by default"]
        G["Search Console · GA4<br/>PageSpeed · CrUX"]
        LT["LanguageTool"]
        LM["Moz · Majestic · Ahrefs"]
        AI["Claude CLI<br/>via a loopback bridge"]
    end

    SITE["The site being crawled"]
    LOG["Your access log"]
    HOOK["A webhook, anywhere you like"]

    UI --> APP
    API --> APP
    MCP --> APP
    APP --> CRAWL
    SCHED --> CRAWL
    CRAWL --> CHECKS
    CHECKS --> DB
    APP --> DB
    CRAWL -->|fetch| SITE
    LOG -.-> CHECKS
    CHECKS -.-> G
    CHECKS -.-> LT
    CHECKS -.-> LM
    CHECKS -.-> AI
    APP -->|on finish| HOOK

Three things in that diagram are worth saying out loud:

1. Everything outside the box is optional and off by default. A fresh install contacts nothing but the site you point it at.

2. Every finding goes through one factory (src/utils/issue_factory.py), which is why they all carry the same shape and a stable id. A check cannot invent its own format.

3. The Claude bridge keeps the credential on the host. The CLI ships as a native binary that will not run in a Linux container, so rather than bake a long-lived credential into the image, the container asks the host over a loopback-only bridge with a shared secret.

Reproduce the screenshots

Every image in this README is generated, not curated:

python main.py --local --port 5058 &          # a clean install, empty data/
python tools/screenshots.py --app http://127.0.0.1:5058

With no --site it serves the bundled fixture — the deliberately broken site in tests/ — so anyone can reproduce the shape of these images without having a site to point at. The published ones were taken with --site https://freshlycited.com/, because a fixture on 127.0.0.1 looks like a toy and ours is a real site we are allowed to show.

Either way it crawls through the UI exactly as a person would, captures every view, and refuses to write any screenshot whose page contains a banned string — which is not a theoretical guard. It caught six images carrying a real connected Google account and other people's crawl history while this README was being written, and nothing was written for those.

Security and privacy

• Nothing leaves your machine unless you configure it to. No telemetry, no phone-home, no account.

• Crawled page bodies are stored so that crawl-to-crawl diffing works. Roughly 5–30KB per page; Settings → Crawls to keep bounds it.

• Google tokens are Fernet-encrypted under a key derived from SECRET_KEY, in their own table, and never written into the crawl config (which is snapshotted to the database and included in exports).

• --local disables authentication entirely and grants admin to everyone. Right on a laptop, wrong on anything reachable.

• No third-party security audit has been performed.

The full posture, including what has and has not been reviewed, is in SECURITY.md. Report vulnerabilities privately through a GitHub advisory.

FAQ

No, and it is worth being precise about that. Toadstone is source-available under the Elastic License 2.0: you can read it, run it, modify it, self-host it for yourself or your clients, and redistribute it. What you cannot do is offer it to third parties as a hosted or managed service, or strip the notices. ELv2 is not an OSI-approved open-source licence.

The portions inherited from LibreCrawl remain MIT, and nothing here changes that — anyone can still take LibreCrawl from its own authors under MIT.

Much of what is here is upstreamable and some of it should be upstreamed. But the fork started by auditing the existing checks and finding that most had never been observed to fire, and the work that followed — a coverage harness, a stable issue identity, an evidence block on every finding, an API and an MCP server — changes enough of the shape of the thing that it would have been a very long pull request with no reviewer. Upstream is credited in NOTICE and in FORK.md, and general crawler fixes should go there first.

See the comparison. The short version: Screaming Frog is mature, fast, extremely well understood, and better than this at most of what both do. It also has the deepest MCP server of any vendor here — 29 tools, including crawl control — so "an agent can drive it" is not a difference between us, whatever you may read elsewhere.

What is different: the crawl and the data stay on your own machine, the findings come out of an HTTP API rather than an export, nothing is metered, and there is a published record of which checks have actually been watched to fire. If you want a desktop app, a licence and support, buy Screaming Frog — it is roughly £199 a year and worth it.

The crawl is bounded by max_urls and by memory, and it stores page bodies, so a million-URL crawl is not what this is for. It handles tens of thousands of pages on ordinary hardware. The UI shows a live estimate of disk per URL and what a million would cost, because that is a decision you should make before starting rather than discover afterwards.

Yes, and it reports what robots.txt blocked rather than silently skipping it — including sitemap URLs that are blocked and canonicals that point at a blocked page. You can turn the obedience off for a site you own, which is a setting you have to go and find.

It says so, as a finding. This is the single most important design decision in the project: a failed PageSpeed call used to produce no issues at all, which is indistinguishable from a site with perfect Core Web Vitals. Every integration now reports its own unavailability.

Yes — on your own sites, and on your clients' sites as part of your own work. What ELv2 stops is offering Toadstone itself to third parties as a hosted or managed service. If that is what you want to do, talk to us about a commercial licence.

Roadmap

Honest about what is not here yet.

Documentation

Tools that ship with it

Built by LabsMedia

Toadstone is built and maintained by LabsMedia, the digital marketing division of Space-O Technologies, and led by Rakesh Patel, founder of both.

We did not build this as a side project. We build links and run SEO for other people's sites, which means we spend our days looking at crawl reports and betting client money on whether they are true. The fork started because a report came back clean on a site we already knew was broken — and the reason turned out to be that most of the crawler's checks had never once been observed to fire. Everything in this repository follows from refusing to accept that again.

If you want that same scrutiny applied to your site, or links that are relevant, real and live rather than counted:

We also build Skald — self-hosted web analytics read from your server's own access log, which sees the AI assistants and crawlers that JavaScript analytics cannot. Toadstone tells you what is wrong with a site; Skald tells you who actually came to it.

Credits

LibreCrawl by Phiality, released under the MIT License, is the crawler this is built on. The MIT grant is also what makes it legal to distribute this work under different terms, and it would be poor manners not to say so clearly: a solo developer's work is the reason this project exists at all. Upstream is untouched beneath this fork, and its updates still merge.

axe-core by Deque Systems (MPL-2.0), bundled unmodified, is every accessibility check here.

Toadstone is not affiliated with, endorsed by, or supported by Phiality or LibreCrawl. The MIT licence grants no trademark rights and we claim none. If something here is broken, it is ours, and the issue tracker is here — not theirs.

Every other dependency, with its licence, is listed in NOTICE.

Contributing

One rule above all the others: a check is not finished until it has been seen firing in a real crawl. CONTRIBUTING.md explains what that means in practice, along with how to run the suites (34 of them, no framework, plain scripts) and what a pull request needs to show.

Found a check reporting something that is not true? That is the most valuable issue this project can receive — there is a template for it.

Licence

Elastic License 2.0 — source-available, not OSI open source. Self-host it, modify it, use it on your own and your clients' sites. You may not offer it to others as a hosted or managed service without a commercial licence. Portions inherited from LibreCrawl remain MIT.

Copyright 2026 Space-O Technologies. Author: Rakesh Patel.