SiteProbe

Open-source autonomous website auditor & fixer for SEO, GEO, performance, accessibility and technical web quality.

Quick Start • Workflow • Agent Skill • MCP Server • Honesty & Status Table • Architecture

⚡ Short Demo

# 1. Run deep public website audit
siteprobe audit https://example.com --lang en --output ./siteprobe-report

# 2. Automatically apply safe code remediations to local project
siteprobe fix ./siteprobe-report/audit.json --repo /path/to/project

# 3. Verify that changes resolved the issues without regressions
siteprobe verify ./siteprobe-report/audit.json --repo /path/to/project

Related MCP server: OctoBoost SEO MCP Server

💡 Why SiteProbe?

Most auditing tools are passive checklists. They hand you a 40-page PDF of problems and leave the engineering team with all the toil.

SiteProbe connects diagnosis with remediation.

It doesn't just find a missing canonical tag, an unlabelled accessibility input, or an absent /llms.txt file for AI agents—it constructs an atomic, risk-categorized remediation plan, modifies local source code safely, and runs post-change verification to prove the problem is resolved.

Core Guarantees:

• Zero Hallucination: No simulated rankings or fabricated Search Console numbers. If an external credential is missing, SiteProbe clearly marks it as optional/unavailable and explains how to connect it.

• Deterministic Quality Scoring: Transparent, explainable weights across Technical, On-Page, Schema, GEO, Accessibility, and Security.

• Realistic GEO Metrics: The GEO score is strictly a technical readiness index evaluating observable signals (AI crawler permissions in robots.txt, /llms.txt presence, structured entities, and question-answer semantics). It is not a guarantee or predictor of AI citations or rankings.

• Safety First: Every automated fix creates an in-memory file snapshot before applying changes and rolls back automatically if verification fails.

• Multilingual from Day 1: Native localized audits in English (en), Persian (fa), and Turkish (tr).

📋 Feature Status & Honesty Table

🎯 From Measurement To Action: The Remediation Loop

SiteProbe serves as the Autonomous Remediation & Verification Engine for the Molavi AI Visibility Stack, turning empirical findings from GEO-Scope Benchmarks into verified code remediations:

┌─────────────────────────────────┐
│     1. BENCHMARK FINDING        │  Empirical visibility baseline measured via GEO-Scope
│   (e.g., Low AI Recommendation) │  (e.g., missing citations, unquoted brand entities)
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│        2. DEEP AUDIT            │  SiteProbe / SAGE analyzes DOM, headers, AI crawlers,
│  (Technical, Entity, Semantic)  │  passage chunk boundaries, and schema structures
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│     3. RECOMMENDATION PLAN      │  Deterministic, risk-stratified actionable fixes:
│  (SAFE_AUTOFIX / MANUAL_REVIEW) │  robots.txt bot policies, JSON-LD graphs, llms.txt
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│    4. SAFE IMPLEMENTATION       │  SiteProbe safely modifies source code repository
│   (Atomic Snapshot & Patch)     │  with automatic rollback on syntax or rule failure
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│    5. EMPIRICAL RE-MEASURE      │  GEO-Scope re-executes standardized prompt test suite
│  (Measure Δ SoM & Citations)    │  to measure verified lift in observed AI visibility
└─────────────────────────────────┘

See the Benchmark Ecosystem Map for full architectural contracts.

🏛️ Ecosystem

SiteProbe operates as the autonomous remediation component of the Molavi AI Visibility Stack:

• Discovery: AnswerPath GEO

• Measurement: GEO-Scope

• Diagnostics: SAGE Audit

• Action: SiteProbe

• Protocol: MCP GEO Server

📖 Evidence & Remediation Benchmarks

• Audit Findings Payload: examples/audit_example.json

• Verified Fix & Delta Payload: examples/fix_example.json

• Standalone Offline Remediation Demo: examples/public_demo/

• Live Crawl Report Case Study: siteprobe-report-molavi/audit.md

• Remediation Guide & Safety Protocol: docs/remediation_guide.md

• Cross-Repository Evidence Map: Ecosystem Evidence Flow

📊 Report Preview

SiteProbe generates a single-file, interactive HTML report with zero external CDN dependencies:

• Interactive Scorecard: Overall score and individual category health gauges.

• GEO & AI Search Pillar Breakdown: Transparent scoring for bot access, llms.txt, entity clarity, and QA markup.

• Filterable Finding Cards: Browse issues by severity (Critical, High, Medium, Low) and category.

• Diff Viewer: See unified code diffs of all remediations applied.

📦 Installation

From Source (Clean Environment)

git clone https://github.com/tmolavi/siteprobe.git
cd siteprobe
pip install -e .

Optional Dependency Extras

# Headless browser automation (Playwright)
pip install "siteprobe[browser]"
playwright install chromium

# Google Search Console & GA4 APIs
pip install "siteprobe[google]"

# LLM Providers (LiteLLM)
pip install "siteprobe[ai]"

# All optional integrations
pip install "siteprobe[all]"

⏱️ 60-Second Quick Start

# Check installed tools and environment
siteprobe doctor

# Run an audit on a website
siteprobe audit https://example.com

# Inspect generated reports
open ./siteprobe-report/audit.html

🐍 Python Usage

import asyncio
from siteprobe.core.engine import AuditEngine
from siteprobe.core.config import SiteProbeConfig, SiteConfig, CrawlConfig

async def main():
    config = SiteProbeConfig(
        site=SiteConfig(url="https://example.com", language="en"),
        crawl=CrawlConfig(max_pages=25, concurrency=3),
    )
    engine = AuditEngine(config=config)
    result = await engine.run_audit("https://example.com")
    
    print(f"Overall Quality Score: {result.scores.overall_score}/100")
    print(f"GEO Readiness Score: {result.scores.geo_score}/100")
    print(f"Discovered {len(result.findings)} findings across {result.crawl_summary.total_crawled} pages.")

if __name__ == "__main__":
    asyncio.run(main())

💻 CLI Usage

SSR & Rendering Inspection

Inspect initial raw server HTML for SSR vs. client-only SPA rendering, framework hydration payloads (Next.js, Nuxt, Remix, Astro), and missing raw SEO tags:

siteprobe ssr https://example.com

Audit Command

# Standard audit
siteprobe audit https://example.com

# Specify crawl cap and output folder
siteprobe audit https://example.com --max-pages 100 --output ./my-audit

# Multilingual audit in Persian or Turkish
siteprobe audit https://example.com --lang fa
siteprobe audit https://example.com --lang tr

# Include Lighthouse performance lab data (requires Node.js)
siteprobe audit https://example.com --lighthouse

Fix Command

# Apply safe autofixes to a local project repository
siteprobe fix ./my-audit/audit.json --repo /path/to/my-project

Verify Command

# Verify applied changes against the audit report
siteprobe verify ./my-audit/audit.json --repo /path/to/my-project

System Diagnostics

# Inspect environment and optional integration status
siteprobe doctor

# List configured providers
siteprobe integrations

🤖 Agent Skill Installation & Exact Instructions

SiteProbe provides a canonical Agent Skill at: skills/siteprobe/SKILL.md

1. Claude Code

To equip Claude Code with SiteProbe, point to the skill folder or copy it into your local skill directory:

# Copy to local skills directory
mkdir -p ~/.claude/skills
cp -r skills/siteprobe ~/.claude/skills/

Tested Invocation:

claude "Read skills/siteprobe/SKILL.md and audit https://example.com, then fix safe issues in this project."

2. OpenAI Codex / CLI Agents

Codex reads repository guidelines from AGENTS.md in the project root:

codex "Audit https://example.com using the siteprobe CLI and verify all changes."

3. Google Gemini / Antigravity

In Antigravity or Gemini CLI, link the skill location in your configuration:

{
  "skills": [
    "/path/to/siteprobe/skills/siteprobe/SKILL.md"
  ]
}

Tested Invocation:

"Audit this website and fix all safe issues you can verify following the siteprobe skill workflow."

4. Cursor

In Cursor, reference the skill file in .cursorrules or mention the file directly in the chat prompt:

@skills/siteprobe/SKILL.md Audit https://example.com and apply verified fixes.

🔌 Model Context Protocol (MCP)

Launch SiteProbe's MCP server over stdio:

siteprobe serve-mcp

Adding to Claude Desktop or Agent MCP Configuration

Add to your claude_desktop_config.json or agent settings:

{
  "mcpServers": {
    "siteprobe": {
      "command": "siteprobe",
      "args": ["serve-mcp"]
    }
  }
}

Exposed MCP Tools:

• start_audit(url, max_pages, language): Initiates background audit job.

• get_audit_status(job_id): Checks job status.

• get_findings(job_id, category, severity): Returns filtered audit findings.

• get_geo_findings(job_id): Returns GEO score and AI crawler accessibility.

• create_remediation_plan(job_id): Generates actionable fix plan.

• apply_safe_fixes(job_id, repo_path): Applies safe code modifications.

• verify_fixes(job_id, repo_path): Verifies fix results.

• generate_report(job_id, format): Exports report in HTML, JSON, or Markdown.

🔗 External Integrations

All external integrations are strictly optional and degrade gracefully:

• Google Search Console: Set GOOGLE_APPLICATION_CREDENTIALS to correlate audit findings with impressions, CTR, and search queries.

• Google Analytics 4: Set GA4_PROPERTY_ID to connect technical defects with conversion drop-offs.

• DataForSEO: Set DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD for SERP volume and backlink data.

• Lighthouse / Chrome: Available automatically when Node.js and Chrome are detected.

• Playwright: Installed via pip install "siteprobe[browser]".

🐳 Docker

Run SiteProbe via Docker with zero host dependencies:

docker build -t siteprobe .
docker run --rm -v $(pwd)/reports:/app/reports siteprobe audit https://example.com --output /app/reports

Or with docker-compose:

docker-compose up

🏗️ Architecture

siteprobe/
├── src/siteprobe/
│   ├── cli/          # Typer CLI commands
│   ├── core/         # Audit orchestrator, scoring model, YAML loader
│   ├── crawler/      # AsyncIO polite crawler, SQLite storage, robots/sitemaps
│   ├── checks/       # 9 check categories (Technical, Onpage, Schema, GEO, etc.)
│   ├── integrations/ # Modular adapters (GSC, GA4, DataForSEO, Lighthouse, Playwright)
│   ├── remediation/  # Risk-classified planner, atomic file handlers, snapshot/rollback
│   ├── verification/ # Post-fix verification engine
│   ├── reporting/    # Interactive HTML dashboard, JSON, and Markdown reporters
│   ├── i18n/         # Locales (English, Persian, Turkish) and translation engine
│   ├── models/       # Pydantic data schemas
│   └── mcp/          # FastMCP / Standard MCP server
├── skills/
│   └── siteprobe/    # Canonical Agent Skill (SKILL.md, references, scripts)
├── tests/            # Automated test suite + mock website fixture
└── docs/             # Technical architecture and user guides

🛡️ Remediation Safety & Rollback

SiteProbe is engineered with defensive patterns:

• Pre-Flight Snapshot: Every file about to be edited is captured in an in-memory snapshot.

• Format Preservation: HTML changes use semantic tag injection preserving surrounding structure.

• Automatic Rollback: If a verification check fails after editing, SiteProbe can immediately restore files to their exact pre-fix state.

• Clean Diffs: Unified diffs are printed to the console and saved in remediation reports.

🗺️ Roadmap

• v0.1.0: Core async crawler with SQLite storage

• v0.1.0: 24 deterministic checks across 9 categories

• v0.1.0: GEO / AI-search readiness scoring

• v0.1.0: Remediation framework with safe autofixes

• v0.1.0: Before/after verification engine

• v0.1.0: Standalone interactive HTML reporting

• v0.1.0: Multilingual support (en, fa, tr)

• v0.1.0: FastMCP server & Agent Skill specification

• v0.2.0: Deep JS rendering via distributed Playwright cluster

• v0.2.0: Automated GitHub Pull Request creation CLI command (siteprobe pr)

• v0.3.0: Native headless CMS plugins for WordPress and Shopify API-based auto-remediation

💬 Community & Contributing

Contributions are welcome! Please read CONTRIBUTING.md and our CODE_OF_CONDUCT.md.

• Discussions: GitHub Discussions across General, Ideas, Help, and Show & Tell.

• First Contribution Guide: docs/FIRST_CONTRIBUTION.md

• Remediation & Safety Guide: docs/remediation_guide.md

• Report Issues: GitHub Issues

# Run test suite
pytest -v

🔒 Security

For security vulnerability disclosures, please review SECURITY.md and contact taqimolavi@gmail.com.

💡 Related Projects & Inspiration

SiteProbe's SSR and rendering analysis draws architectural inspiration from prominent technical web tooling:

• is-ssr & regex/ast hydration analyzers: Inspecting hydration state payloads (__NEXT_DATA__, __NUXT_DATA__, window.__remixContext, astro-island).

• puppeteer-ssr-checker & Rendertron: Comparing initial raw server markup against rendered client DOM.

• Model Context Protocol (@modelcontextprotocol/server-puppeteer & server-fetch): Standardizing browser inspection primitives and raw HTTP fetching for AI agents.

📄 License

SiteProbe is licensed under the MIT License. Copyright © 2026 Taqi Molavi.