SiteProbe

Open-source autonomous website auditor & fixer for SEO, GEO, performance, accessibility and technical web quality.

Quick Start • Google Update Triage • Workflow • AI Assistants & Skill • MCP Server • Honesty & Status Table

⚡ Short Demo

# 1. Run deep public website audit
siteprobe audit https://example.com --lang en --output ./siteprobe-report

# 2. Automatically apply safe code remediations to local project
siteprobe fix ./siteprobe-report/audit.json --repo /path/to/project

# 3. Verify that changes resolved the issues without regressions
siteprobe verify ./siteprobe-report/audit.json --repo /path/to/project

# 4. Triage Search Console performance around Google algorithm updates
siteprobe update-triage --csv examples/update-impact/sample-input.csv

Related MCP server: OctoBoost SEO MCP Server

💡 Why SiteProbe?

Most auditing tools are passive checklists. They hand you a 40-page PDF of problems and leave the engineering team with all the toil.

SiteProbe connects diagnosis with remediation.

It doesn't just find a missing canonical tag, an unlabelled accessibility input, or an absent /llms.txt file for AI agents—it constructs an atomic, risk-categorized remediation plan, modifies local source code safely, and runs post-change verification to prove the problem is resolved.

Core Guarantees:

• Zero Hallucination: No simulated rankings or fabricated Search Console numbers. If an external credential is missing, SiteProbe clearly marks it as optional/unavailable and explains how to connect it.

• Deterministic Quality Scoring: Transparent, explainable weights across Technical, On-Page, Schema, GEO, Accessibility, and Security.

• Realistic GEO Metrics: The GEO score is strictly a technical readiness index evaluating observable signals (AI crawler permissions in robots.txt, /llms.txt presence, structured entities, and question-answer semantics). It is not a guarantee or predictor of AI citations or rankings.

• Safety First: Every automated fix creates an in-memory file snapshot before applying changes and rolls back automatically if verification fails.

• Multilingual from Day 1: Native localized audits in English (en), Persian (fa), and Turkish (tr).

📋 Feature Status & Honesty Table

🎯 From Measurement To Action: The Remediation Loop

SiteProbe serves as the Autonomous Remediation & Verification Engine for the Molavi AI Visibility Stack, turning empirical findings from GEO-Scope Benchmarks into verified code remediations:

┌─────────────────────────────────┐
│     1. BENCHMARK FINDING        │  Empirical visibility baseline measured via GEO-Scope
│   (e.g., Low AI Recommendation) │  (e.g., missing citations, unquoted brand entities)
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│        2. DEEP AUDIT            │  SiteProbe / SAGE analyzes DOM, headers, AI crawlers,
│  (Technical, Entity, Semantic)  │  passage chunk boundaries, and schema structures
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│     3. RECOMMENDATION PLAN      │  Deterministic, risk-stratified actionable fixes:
│  (SAFE_AUTOFIX / MANUAL_REVIEW) │  robots.txt bot policies, JSON-LD graphs, llms.txt
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│    4. SAFE IMPLEMENTATION       │  SiteProbe safely modifies source code repository
│   (Atomic Snapshot & Patch)     │  with automatic rollback on syntax or rule failure
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│    5. EMPIRICAL RE-MEASURE      │  GEO-Scope re-executes standardized prompt test suite
│  (Measure Δ SoM & Citations)    │  to measure verified lift in observed AI visibility
└─────────────────────────────────┘

See the Benchmark Ecosystem Map for full architectural contracts.

🏛️ Ecosystem

SiteProbe operates as the autonomous remediation component of the Molavi AI Visibility Stack:

• Discovery: AnswerPath GEO

• Measurement: GEO-Scope

• Diagnostics: SAGE Audit

• Action: SiteProbe

• Protocol: MCP GEO Server

📖 Evidence & Remediation Benchmarks

• Audit Findings Payload: examples/audit_example.json

• Verified Fix & Delta Payload: examples/fix_example.json

• Standalone Offline Remediation Demo: examples/public_demo/

• Live Crawl Report Case Study: siteprobe-report-molavi/audit.md

• Remediation Guide & Safety Protocol: docs/remediation_guide.md

• Cross-Repository Evidence Map: Ecosystem Evidence Flow

📊 Report Preview

SiteProbe generates a single-file, interactive HTML report with zero external CDN dependencies:

• Interactive Scorecard: Overall score and individual category health gauges.

• GEO & AI Search Pillar Breakdown: Transparent scoring for bot access, llms.txt, entity clarity, and QA markup.

• Filterable Finding Cards: Browse issues by severity (Critical, High, Medium, Low) and category.

• Diff Viewer: See unified code diffs of all remediations applied.

📦 Installation

From Source (Clean Environment)

git clone https://github.com/tmolavi/siteprobe.git
cd siteprobe
pip install -e .

Optional Dependency Extras

# Headless browser automation (Playwright)
pip install "siteprobe[browser]"
playwright install chromium

# Google Search Console & GA4 APIs
pip install "siteprobe[google]"

# LLM Providers (LiteLLM)
pip install "siteprobe[ai]"

# All optional integrations
pip install "siteprobe[all]"

⏱️ 60-Second Quick Start

# Check installed tools and environment
siteprobe doctor

# Run an audit on a website
siteprobe audit https://example.com

# Inspect generated reports
open ./siteprobe-report/audit.html

🐍 Python Usage

import asyncio
from siteprobe.core.engine import AuditEngine
from siteprobe.core.config import SiteProbeConfig, SiteConfig, CrawlConfig

async def main():
    config = SiteProbeConfig(
        site=SiteConfig(url="https://example.com", language="en"),
        crawl=CrawlConfig(max_pages=25, concurrency=3),
    )
    engine = AuditEngine(config=config)
    result = await engine.run_audit("https://example.com")
    
    print(f"Overall Quality Score: {result.scores.overall_score}/100")
    print(f"GEO Readiness Score: {result.scores.geo_score}/100")
    print(f"Discovered {len(result.findings)} findings across {result.crawl_summary.total_crawled} pages.")

if __name__ == "__main__":
    asyncio.run(main())

💻 CLI Usage

SSR & Rendering Inspection

Inspect initial raw server HTML for SSR vs. client-only SPA rendering, framework hydration payloads (Next.js, Nuxt, Remix, Astro), and missing raw SEO tags:

siteprobe ssr https://example.com

Audit Command

# Standard audit
siteprobe audit https://example.com

# Specify crawl cap and output folder
siteprobe audit https://example.com --max-pages 100 --output ./my-audit

# Multilingual audit in Persian or Turkish
siteprobe audit https://example.com --lang fa
siteprobe audit https://example.com --lang tr

# Include Lighthouse performance lab data (requires Node.js)
siteprobe audit https://example.com --lighthouse

Fix Command

# Apply safe autofixes to a local project repository
siteprobe fix ./my-audit/audit.json --repo /path/to/my-project

Verify Command

# Verify applied changes against the audit report
siteprobe verify ./my-audit/audit.json --repo /path/to/my-project

🔍 Google Update Impact Triage (v0.2.0)

SiteProbe provides an evidence-based, reproducible workflow for examining Google Search Console performance changes around official algorithm updates (e.g. the September 2026 Google Spam Update, rolled out 2026-09-24 through 2026-10-08).

🎯 Objective: Distinguishing 5 Diagnostic States

Rather than guessing or blaming Google updates arbitrarily, SiteProbe strictly classifies observations into one of 5 defensible states:

1. NO_CLEAR_CHANGE_OBSERVED: Observed fluctuations remain within the site's own daily baseline variance envelope ($2 \times \text{CV}$).

2. PERFORMANCE_CHANGE_OBSERVED: Statistically meaningful traffic decline occurred during update rollout with no manual actions or blocking technical errors.

3. TECHNICAL_ISSUE_OBSERVED: Severe technical defects (HTTP 5xx, noindex, broken canonicals, or robots.txt blocks) explain ranking or traffic drop independently.

4. MANUAL_ACTION_REPORTED: Explicit Search Console manual action notice confirmed (never inferred from traffic decline).

5. INSUFFICIENT_DATA / DATA_UNAVAILABLE: Incomplete comparison windows or missing Search Console export.

📅 Frozen Comparison Windows (September 2026 Spam Update)

• PRE Window: 2026-09-10 to 2026-09-23 (14 days, matched weekdays)

• ROLLOUT Window: 2026-09-24 to 2026-10-07 (14 days, matched weekdays)

• COMPLETION Day: 2026-10-08 (1 day recorded separately)

• POST Window: 2026-10-09 to 2026-10-22 (marked INCOMPLETE; no future data extrapolated)

💻 Triage CLI Commands

# Run triage on a Search Console performance export
siteprobe update-triage --csv ./gsc_performance.csv

# Run triage with simultaneous technical site audit
siteprobe update-triage --csv ./gsc_performance.csv --url https://example.com

# Export machine-readable JSON triage report
siteprobe update-triage --csv ./gsc_performance.csv --output ./triage-report.json

# Test with synthetic benchmark dataset
siteprobe update-triage --csv examples/update-impact/sample-input.csv --output examples/update-impact/sample-output.json

• 📖 Research & Methodology: docs/research/google-spam-update-september-2026.md

• 🧪 Synthetic Reproduction Benchmark: examples/update-impact/README.md

• 🛡️ Scientific Disclaimer: Correlation does not prove causation. SiteProbe does not detect internal SpamBrain algorithms, and never labels any website as spam.

System Diagnostics

# Inspect environment and optional integration status
siteprobe doctor

# List configured providers
siteprobe integrations

🤖 AI Assistant & Agent Skill Quickstart (Codex, Antigravity, Claude, Cursor)

SiteProbe includes first-class rules and intake guidelines for all major AI coding environments:

• Canonical Skill: skills/siteprobe/SKILL.md

• Agent Guidelines: AGENTS.md (Codex, Antigravity, Windsurf)

• Claude Guidelines: CLAUDE.md (Claude Code, Claude Desktop)

• Cursor Rules: .cursorrules and .cursor/rules/siteprobe.mdc

📋 What AI Agents Ask For On First Use

Before running any analysis, AI assistants are instructed to check prerequisites and ask for required inputs:

1. Environment Check: Run siteprobe doctor to verify CLI and active tools.

2. Website Audit (siteprobe audit): Requires Target URL (e.g. https://example.com) and optional report language (en, fa, tr).

3. Google Update Triage (siteprobe update-triage): Requires Search Console performance CSV (date,clicks,impressions) and manual action status.

4. Automated Fixes (siteprobe fix): Requires local source code repository path (--repo <path>) and audit.json.

1. Claude Code

Point Claude Code to the skill or copy it into your local skills directory:

# Copy to local skills directory
mkdir -p ~/.claude/skills
cp -r skills/siteprobe ~/.claude/skills/

Tested Invocation:

claude "Read skills/siteprobe/SKILL.md and audit https://example.com, then fix safe issues in this project."

2. OpenAI Codex / CLI Agents

Codex automatically reads repository guidelines from AGENTS.md:

codex "Audit https://example.com using the siteprobe CLI and verify all changes."

3. Google Gemini / Antigravity

In Antigravity or Gemini CLI, reference skills/siteprobe/SKILL.md or copy it to ~/.gemini/config/skills/siteprobe/:

cp -r skills/siteprobe ~/.gemini/config/skills/

Tested Invocation:

"Audit this website and fix all safe issues you can verify following the siteprobe skill workflow."

4. Cursor IDE

Cursor automatically picks up .cursorrules and .cursor/rules/siteprobe.mdc in the repository root. You can also mention the skill directly:

@skills/siteprobe/SKILL.md Audit https://example.com and apply verified fixes.

🔌 Model Context Protocol (MCP)

Launch SiteProbe's MCP server over stdio:

siteprobe serve-mcp

Adding to Claude Desktop or Agent MCP Configuration

Add to your claude_desktop_config.json or agent settings:

{
  "mcpServers": {
    "siteprobe": {
      "command": "siteprobe",
      "args": ["serve-mcp"]
    }
  }
}

Exposed MCP Tools:

• start_audit(url, max_pages, language): Initiates background audit job.

• get_audit_status(job_id): Checks job status.

• get_findings(job_id, category, severity): Returns filtered audit findings.

• get_geo_findings(job_id): Returns GEO score and AI crawler accessibility.

• create_remediation_plan(job_id): Generates actionable fix plan.

• apply_safe_fixes(job_id, repo_path): Applies safe code modifications.

• verify_fixes(job_id, repo_path): Verifies fix results.

• generate_report(job_id, format): Exports report in HTML, JSON, or Markdown.

🔗 External Integrations

All external integrations are strictly optional and degrade gracefully:

• Google Search Console: Set GOOGLE_APPLICATION_CREDENTIALS to correlate audit findings with impressions, CTR, and search queries.

• Google Analytics 4: Set GA4_PROPERTY_ID to connect technical defects with conversion drop-offs.

• DataForSEO: Set DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD for SERP volume and backlink data.

• Lighthouse / Chrome: Available automatically when Node.js and Chrome are detected.

• Playwright: Installed via pip install "siteprobe[browser]".

🐳 Docker

Run SiteProbe via Docker with zero host dependencies:

docker build -t siteprobe .
docker run --rm -v $(pwd)/reports:/app/reports siteprobe audit https://example.com --output /app/reports

Or with docker-compose:

docker-compose up

🏗️ Architecture

siteprobe/
├── src/siteprobe/
│   ├── cli/          # Typer CLI commands
│   ├── core/         # Audit orchestrator, scoring model, YAML loader
│   ├── crawler/      # AsyncIO polite crawler, SQLite storage, robots/sitemaps
│   ├── checks/       # 9 check categories (Technical, Onpage, Schema, GEO, etc.)
│   ├── integrations/ # Modular adapters (GSC, GA4, DataForSEO, Lighthouse, Playwright)
│   ├── remediation/  # Risk-classified planner, atomic file handlers, snapshot/rollback
│   ├── verification/ # Post-fix verification engine
│   ├── reporting/    # Interactive HTML dashboard, JSON, and Markdown reporters
│   ├── updates/      # Google algorithm update triage engine & frozen windows
│   ├── i18n/         # Locales (English, Persian, Turkish) and translation engine
│   ├── models/       # Pydantic data schemas
│   └── mcp/          # FastMCP / Standard MCP server
├── skills/
│   └── siteprobe/    # Canonical Agent Skill (SKILL.md, references, scripts)
├── tests/            # Automated test suite + mock website fixture
└── docs/             # Technical architecture and user guides

🛡️ Remediation Safety & Rollback

SiteProbe is engineered with defensive patterns:

• Pre-Flight Snapshot: Every file about to be edited is captured in an in-memory snapshot.

• Format Preservation: HTML changes use semantic tag injection preserving surrounding structure.

• Automatic Rollback: If a verification check fails after editing, SiteProbe can immediately restore files to their exact pre-fix state.

• Clean Diffs: Unified diffs are printed to the console and saved in remediation reports.

🗺️ Roadmap

• v0.1.0: Core async crawler with SQLite storage

• v0.1.0: 24 deterministic checks across 9 categories

• v0.1.0: GEO / AI-search readiness scoring

• v0.1.0: Remediation framework with safe autofixes

• v0.1.0: Before/after verification engine

• v0.1.0: Standalone interactive HTML reporting

• v0.1.0: Multilingual support (en, fa, tr)

• v0.1.0: FastMCP server & Agent Skill specification

• v0.2.0: Evidence-based Google Update Impact Triage (September 2026 Spam Update)

• v0.2.0: Deep JS rendering via distributed Playwright cluster

• v0.2.0: Automated GitHub Pull Request creation CLI command (siteprobe pr)

• v0.3.0: Native headless CMS plugins for WordPress and Shopify API-based auto-remediation

💬 Community & Contributing

Contributions are welcome! Please read CONTRIBUTING.md and our CODE_OF_CONDUCT.md.

• Discussions: GitHub Discussions across General, Ideas, Help, and Show & Tell.

• First Contribution Guide: docs/FIRST_CONTRIBUTION.md

• Remediation & Safety Guide: docs/remediation_guide.md

• Report Issues: GitHub Issues

# Run test suite
pytest -v

🔒 Security

For security vulnerability disclosures, please review SECURITY.md and contact taqimolavi@gmail.com.

💡 Related Projects & Inspiration

SiteProbe's SSR and rendering analysis draws architectural inspiration from prominent technical web tooling:

• is-ssr & regex/ast hydration analyzers: Inspecting hydration state payloads (__NEXT_DATA__, __NUXT_DATA__, window.__remixContext, astro-island).

• puppeteer-ssr-checker & Rendertron: Comparing initial raw server markup against rendered client DOM.

• Model Context Protocol (@modelcontextprotocol/server-puppeteer & server-fetch): Standardizing browser inspection primitives and raw HTTP fetching for AI agents.

📄 License

SiteProbe is licensed under the MIT License. Copyright © 2026 Taqi Molavi.