Free Open-Source SEO Screaming Toad — Not Frog

A local-first Screaming Frog alternative with an MCP server and theoretical 100M+ URL campaign architecture

Product tour · Quick start · MCP for AI agents · SEO checks · DJAI Academy · Contributing

SEO Screaming Toad, also called DJAI Toad, is a free, open-source technical SEO crawler and website audit tool. It crawls sites locally, preserves evidence in SQLite, audits raw and JavaScript-rendered pages, compares recrawls, exports reports, and exposes a guarded Model Context Protocol (MCP) server so AI agents can operate real SEO audits.

The product is co-created by DJAI Academy trainers and community members, with Siamese Cat Dev as its creator. It brings together practical SEO, product development, AI-agent workflows, and the DJAI community's build-in-public spirit.

If you are searching for an open-source Screaming Frog alternative, a self-hosted SEO spider, a Go website crawler, or an MCP-powered SEO audit tool, this project is built for that workflow—without uploading your crawl database to a hosted service.

Why SEO Screaming Toad?

Related MCP server: Screaming Frog SEO Spider MCP Server

See the dashboard

These are real screenshots from a local 10-page DJAI Academy verification crawl—not mockups or fabricated audit data.

The dense desktop-style workbench keeps projects, bounded crawl profiles, audit evidence, recrawl history, and service links visible without turning a technical audit into a collection of disconnected screens.

Guarded, reusable crawl setup. Define the seed, URL ceiling, raw or JavaScript-rendered mode, compression compatibility, subdomain scope, and path exclusions. Preview scope before starting; robots, DNS/IP, redirect, response-size, and rate controls remain enforced.

Why experienced technical SEOs should evaluate Toad

SEO Screaming Toad is not merely a free crawler with a different mascot. It is built around workflows that matter when an audit must be repeatable, explainable, automatable, and safe:

• Use the whole product without a paid crawler license. The crawler, dashboard, CLI, local API, MCP server, reports, and source code are MIT licensed.

• Give an AI agent purpose-built SEO tools—not a shell. The 23-tool MCP interface covers bounded crawl setup, lifecycle control, pages, links, issues, explanations, comparisons, and managed exports.

• Audit the evidence behind the recommendation. Findings retain versioned rule identities, structured evidence, remediation, and limitations instead of presenting an unexplained score.

• Keep raw and JavaScript-rendered SEO evidence distinct. Diagnose client-side changes without silently replacing what the server originally returned.

• Own the crawl data and automation path. SQLite/WAL state remains local and can be operated through the UI, JSON CLI, authenticated loopback API, or MCP.

• Recover instead of restarting blindly. Durable frontiers, pause/resume/cancel controls, lifecycle events, and crawl history support long-running operational work.

• Build beyond a single desktop crawl. Segmented campaign primitives create a path toward theoretical 100M+ URL work while every individual crawl remains bounded.

If you already use Screaming Frog SEO Spider, test Toad on a representative authorized site and compare coverage, false positives, rendered output, exports, and the workflows your team actually depends on. Teams that value open code, first-class MCP automation, local evidence, and community-driven rules may find a compelling reason to switch; teams dependent on a mature commercial integration should validate that dependency before migrating.

Quick start

Requirements: Go version from .go-version. Node and pnpm are needed only for rendered mode or frontend development.

git clone https://github.com/lovecatisgood-sudo/Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential.git
cd Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential
make bootstrap
go run ./cmd/seo-auditor

Open http://127.0.0.1:7331, create a project and crawl profile, preview its scope, and start an authorized public-site audit.

For a sandboxed development environment:

docker compose up -d dev
docker compose exec dev bash
make test

See DEVELOPMENT.md for container boundaries and the host-toolchain path. Only crawl websites you own or are authorized to test.

MCP server for AI agents

SEO Screaming Toad includes seo-auditor-mcp, a stdio MCP server built with the official Go SDK. An MCP-compatible AI agent can create bounded profiles, preview scope, start or control crawls, inspect SEO issues and page evidence, compare runs, and export managed reports.

AI agent / MCP client
        │ stdio
        ▼
seo-auditor-mcp
        │ authenticated loopback API
        ▼
SEO Screaming Toad → guarded crawler → SQLite evidence

Build the two executables:

mkdir -p bin
go build -o bin/seo-auditor ./cmd/seo-auditor
go build -o bin/seo-auditor-mcp ./cmd/seo-auditor-mcp

Start bin/seo-auditor, configure your MCP client to run the absolute path to bin/seo-auditor-mcp, and call health_get. A connected result reports api_connected: true.

The MCP interface deliberately exposes no generic shell, arbitrary filesystem access, SQL execution, or unrestricted HTTP fetch. Read the complete MCP setup and tool guide or copy the generic MCP client configuration.

SEO audit coverage

The versioned audit engine currently covers:

• HTTP failures, redirects, and broken internal targets;

• titles, meta descriptions, headings, and duplicate metadata;

• canonical tags, canonical chains, conflicts, and invalid targets;

• indexability, robots directives, robots.txt, and sitemap coverage;

• exact and near-duplicate content signals;

• hreflang validity, targets, and reciprocity;

• image alt attributes and failing image resources;

• crawl depth, orphan-like pages, internal links, and nofollow observations;

• mixed content and selected defensive headers;

• JSON-LD structured-data syntax and basic shape validation;

• raw-versus-rendered SEO differences for JavaScript sites.

Every check documents its limitations. Findings are technical observations—not promises about indexing, rankings, traffic, or rich-result eligibility. See the audit rule catalog.

Technical crawler features

• Concurrent Go crawl engine with per-host politeness and bounded retries

• Robots.txt enforcement and sitemap discovery

• URL normalization, deduplication, query controls, and crawler-trap protection

• Prohibited-network and mixed-DNS-answer rejection for SSRF resistance

• Approved-address connection pinning with normal TLS hostname verification

• Response size, header, timeout, redirect, depth, and URL ceilings

• SQLite/WAL durable frontier with serialized writes and recovery

• Optional isolated TypeScript/Chromium rendering worker

• Dense technical React dashboard inspired by desktop SEO crawlers

• Local authenticated API, JSON CLI, MCP server, CSV, NDJSON, and XLSX reports

• Crawl-to-crawl comparison for added, removed, changed, new, and fixed results

About the 100M+ URL architecture

The system has segmented-campaign and distributed-coordination prototypes intended to process bounded batches repeatedly instead of holding an entire enormous crawl in memory. Local synthetic persistence/analysis campaigns have completed at 1 million and 5 million URLs, and a 100,000-URL single-crawl gate has passed.

100M+ is a theoretical architectural potential—not a verified live-network capacity, benchmark, support promise, or guarantee. Actual scale depends on hardware, storage, site behavior, politeness settings, rendering, network conditions, and operational design. Read the scale strategy and benchmark evidence.

Built with the DJAI Academy community

SEO Screaming Toad is a collaborative open-source product created by Siamese Cat Dev together with DJAI Academy trainers and community members. DJAI is where learners, builders, trainers, and working developers turn AI-assisted ideas into useful software—and this crawler is one result of that community working together.

• Looking for web development? Launch a search-ready website with DJAI—from design and implementation to technical SEO foundations.

• Finding a development team that can deliver? Explore DJAI's software development portfolio and build your app, automation, platform, or custom software with the team.

• Want to learn vibe coding? Join the free DJAI online school community and learn alongside people turning ideas into working products.

• Need a break from debugging? Visit Siamese Cat Cafe in Bangna, Bangkok and meet its 16 rescued and adopted cats over a drink.

The service and community links are promotional and open external sites. They never influence crawl results, issue severity, evidence, or recommendations.

Project structure

cmd/                    Go application, CLI, MCP, SBOM, and scale commands
internal/crawler/       crawl engine, robots, sitemaps, links, and traps
internal/fetchpolicy/   target safety, transport, redirects, and scope
internal/rules/         versioned technical SEO audit rules
internal/database/      SQLite/WAL persistence, frontier, and recovery
internal/mcpserver/     bounded Model Context Protocol tools
web/app/                React technical audit dashboard
web/renderer/           isolated optional JavaScript renderer
docs/                   PRD, architecture, operations, security, and evidence

Contributing

Bug reports, new SEO rules, UX improvements, platform testing, documentation, and thoughtful performance work are welcome. Start with CONTRIBUTING.md, review the security model, and choose an issue that matches your experience.

Please give the repository a ⭐ if an open-source, MCP-enabled SEO crawler is useful to you. Stars help more SEO practitioners and AI builders discover the project.

Security and responsible use

Do not put vulnerabilities, private URLs, credentials, or raw client crawl data in public issues. Follow SECURITY.md for responsible disclosure. The crawler must only be used against systems you are authorized to audit.

License and credits

Released under the MIT License. Third-party dependency and source notices are recorded in THIRD_PARTY_NOTICES.md.

Created by Siamese Cat Dev, co-created with trainers and community members from DJAI Academy.