OpenCode Web Audit MCP

A production-oriented, read-only Python MCP server for authorized website research, structured data extraction, downloadable-document analysis, and content auditing inside OpenCode.

Purpose

The server builds an organized, verifiable, deduplicated representation of publicly accessible website content within an explicitly authorized domain. It processes reachable HTML, PDF, CSV, XLSX, JSON, XML, RSS, Atom, text, and supported images without bypassing restrictions.

Related MCP server: superFetch MCP Server

Activation rules

Use this MCP only when the user explicitly asks OpenCode to inspect, crawl, extract, inventory, or audit a public website and provides an authorized domain. Do not activate it for unrestricted Internet search, private systems, authenticated sessions, paywalls, CAPTCHAs, access-control bypass, vulnerability testing, or extraction from unauthorized external domains.

Scope and constraints

The server uses read-only HTTP GET requests. It respects robots.txt, exact domain authorization, optional subdomain authorization, deterministic crawl limits, bounded concurrency, a minimum request delay, redirect validation, response-size limits, request timeouts, content-type allowlists, and SSRF defenses. External links are recorded but never fetched in the current run.

JavaScript is not executed. Therefore, dynamically loaded content is processed only when it is available through a public JSON/XML endpoint or an ordinary linked resource discovered in the returned source. OCR is optional and disabled by default. It requires the local Tesseract executable and the ocr dependency extra.

Directory tree

opencode-web-audit-mcp-python/
├── .env.example
├── .gitignore
├── AGENTS.md
├── LICENSE
├── README.md
├── SECURITY.md
├── opencode.jsonc
├── pyproject.toml
├── docs/
│   ├── ARCHITECTURE.md
│   ├── COMPLETION-CHECKLIST.md
│   └── TOOL-WORKFLOWS.md
├── examples/
│   ├── prompts.md
│   └── sample_policy.json
├── runs/
│   └── .gitkeep
├── src/opencode_web_audit_mcp/
│   ├── __init__.py
│   ├── config.py
│   ├── models.py
│   ├── server.py
│   ├── core/
│   │   ├── auditor.py
│   │   ├── crawler.py
│   │   ├── http_client.py
│   │   ├── report.py
│   │   └── robots.py
│   ├── extractors/
│   │   ├── document.py
│   │   └── html.py
│   ├── security/
│   │   ├── content_policy.py
│   │   └── url_policy.py
│   └── storage/
│       └── run_store.py
└── tests/
    ├── test_html.py
    ├── test_report.py
    └── test_url_policy.py

Requirements and dependencies

Python 3.11 or newer is required. Runtime dependencies are declared in pyproject.toml: the official mcp Python SDK, HTTPX, Beautiful Soup, lxml, Pydantic, pypdf, openpyxl, python-dateutil, and defusedxml. Development dependencies include pytest, coverage, mypy, and Ruff.

Optional OCR requires Tesseract installed by the operating system plus:

pip install -e ".[ocr]"

Step-by-step installation and configuration

The following procedure installs the MCP server in an isolated Python environment, validates the project, registers it as a local OpenCode MCP server, and confirms that its tools are available. The primary instructions target macOS and Linux. Windows-specific path examples are included where they differ.

Step 1: Install the prerequisites

Install the following software before continuing:

• Python 3.11 or newer.

• OpenCode.

• Git, if the project is being cloned from a repository.

• Tesseract only when optional image OCR is required.

Confirm that Python and OpenCode are available:

python3 --version
opencode --version

The Python version must be 3.11 or newer. If python3 resolves to an older version, use the executable for an installed supported version, such as python3.11, python3.12, or python3.13 in the following commands.

Step 2: Extract or clone the project

Extract the ZIP file or clone the repository into a permanent location. Do not configure OpenCode against a temporary download directory that may later be moved or deleted.

Example location on macOS or Linux:

/Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python

Example location on Windows:

C:\Users\YOUR_USERNAME\Developer\opencode-web-audit-mcp-python

Open a terminal and enter the project directory:

cd /ABSOLUTE/PATH/opencode-web-audit-mcp-python

Confirm that the expected project files are present:

pwd
ls -la

At minimum, the directory must contain pyproject.toml, README.md, opencode.jsonc, src/, and tests/.

Step 3: Create the Python virtual environment

Create a dedicated virtual environment inside the project directory:

python3.11 -m venv .venv

If a different supported Python executable is installed, substitute it directly:

python3.12 -m venv .venv

Activate the environment on macOS or Linux:

source .venv/bin/activate

Activate it on Windows PowerShell:

.venv\Scripts\Activate.ps1

Verify that the active interpreter belongs to the project:

python -c "import sys; print(sys.executable)"

The printed path must point to .venv/bin/python on macOS/Linux or .venv\Scripts\python.exe on Windows.

Step 4: Install the project and dependencies

Upgrade the Python packaging tools:

python -m pip install --upgrade pip setuptools wheel

Install the MCP server in editable mode with its development and validation dependencies:

python -m pip install -e ".[dev]"

This command installs the official MCP Python SDK and all runtime dependencies declared in pyproject.toml.

Confirm that the server package and MCP SDK can be imported:

python -c "import mcp; import opencode_web_audit_mcp; print('Imports successful')"

Step 5: Run the project validation checks

Run the static checks and automated tests before registering the server in OpenCode:

ruff check .
mypy src
pytest

All three commands must complete successfully. A failed check should be corrected before the MCP is enabled in OpenCode.

To run the tests with an explicit coverage report:

pytest --cov=opencode_web_audit_mcp --cov-report=term-missing

Step 6: Create the output directory

The server writes each audit run to a dedicated directory under runs/. Create the directory and verify that the current user can write to it:

mkdir -p runs
python -c "from pathlib import Path; p=Path('runs/.write_test'); p.write_text('ok'); p.unlink(); print('Output directory is writable')"

Use a different output directory only when necessary. The process started by OpenCode must have write access to that location.

Step 7: Determine the absolute paths

OpenCode should be configured with absolute paths so that the MCP can be started from any workspace.

On macOS or Linux, obtain the project path with:

pwd

Obtain the virtual-environment Python path with:

python -c "import sys; print(sys.executable)"

Example resolved values:

Project directory: /Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python
Python executable: /Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python/.venv/bin/python
Output directory: /Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python/runs

Do not leave /ABSOLUTE/PATH placeholders in the final OpenCode configuration.

Step 8: Configure the MCP server in OpenCode

OpenCode starts a local MCP server as a subprocess and communicates with it through the MCP stdio transport. Add the following entry to the applicable opencode.json or opencode.jsonc file and replace every placeholder with the absolute paths determined in Step 7.

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "web_audit": {
      "type": "local",
      "command": [
        "/ABSOLUTE/PATH/opencode-web-audit-mcp-python/.venv/bin/python",
        "-m",
        "opencode_web_audit_mcp.server"
      ],
      "cwd": "/ABSOLUTE/PATH/opencode-web-audit-mcp-python",
      "enabled": true,
      "timeout": 30000,
      "environment": {
        "WEB_AUDIT_OUTPUT_ROOT": "/ABSOLUTE/PATH/opencode-web-audit-mcp-python/runs",
        "WEB_AUDIT_MAX_PAGES": "1000",
        "WEB_AUDIT_MAX_DEPTH": "5",
        "WEB_AUDIT_CONCURRENCY": "3",
        "WEB_AUDIT_MIN_DELAY_MS": "750",
        "WEB_AUDIT_LOG_LEVEL": "info"
      }
    }
  },
  "permission": {
    "web_audit_*": "ask"
  }
}

A completed macOS example would look similar to:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "web_audit": {
      "type": "local",
      "command": [
        "/Users/alice/Developer/opencode-web-audit-mcp-python/.venv/bin/python",
        "-m",
        "opencode_web_audit_mcp.server"
      ],
      "cwd": "/Users/alice/Developer/opencode-web-audit-mcp-python",
      "enabled": true,
      "timeout": 30000,
      "environment": {
        "WEB_AUDIT_OUTPUT_ROOT": "/Users/alice/Developer/opencode-web-audit-mcp-python/runs",
        "WEB_AUDIT_MAX_PAGES": "1000",
        "WEB_AUDIT_MAX_DEPTH": "5",
        "WEB_AUDIT_CONCURRENCY": "3",
        "WEB_AUDIT_MIN_DELAY_MS": "750",
        "WEB_AUDIT_LOG_LEVEL": "info"
      }
    }
  },
  "permission": {
    "web_audit_*": "ask"
  }
}

For Windows, use the virtual-environment interpreter path and escape backslashes in JSON:

"command": [
  "C:\\Users\\YOUR_USERNAME\\Developer\\opencode-web-audit-mcp-python\\.venv\\Scripts\\python.exe",
  "-m",
  "opencode_web_audit_mcp.server"
]

The permission rule is intentionally set to ask. It requires explicit approval before an audit tool is used. Change it only after reviewing OpenCode's permission model and the server's access scope.

Step 9: Verify that OpenCode can start the MCP server

Run:

opencode mcp list

The web_audit server should be listed as enabled and connected. If OpenCode reports a startup failure, run the exact configured command manually from the configured working directory:

cd /ABSOLUTE/PATH/opencode-web-audit-mcp-python
/ABSOLUTE/PATH/opencode-web-audit-mcp-python/.venv/bin/python -m opencode_web_audit_mcp.server

A stdio MCP server normally waits for protocol input and may appear idle when started manually. The important checks are that it does not immediately exit with an import error, path error, configuration error, or traceback. Stop the manual process with Ctrl+C after the check.

Step 10: Confirm tool discovery inside OpenCode

Start OpenCode in a workspace and ask it to list or use the tools provided by the web_audit MCP server. OpenCode should discover these five tools:

web_audit_inspect_site
web_audit_crawl_site
web_audit_extract_resource
web_audit_get_run
web_audit_audit_run

A suitable first request is:

Use the web_audit MCP server to inspect https://example.org with example.org as the authorized domain. Do not crawl beyond depth 1 and do not process more than 20 pages.

OpenCode should request permission before tool execution when the supplied permission configuration is used.

Step 11: Run a bounded first audit

Begin with a small, deterministic crawl before increasing limits:

Use web_audit_inspect_site for https://example.org. The authorized domain is example.org. Allow HTML, PDF, JSON, XML, CSV, and XLSX. Record external links but do not fetch them.

After reviewing the preflight result, run:

Use web_audit_crawl_site for https://example.org. The authorized domain is example.org. Set maximum depth to 2, maximum pages to 50, concurrency to 2, and minimum request delay to 1000 milliseconds. Respect robots.txt and do not fetch external domains.

The response should include a run ID and output directory. Verify the generated artifacts:

find runs -maxdepth 2 -type f -print

Each completed or partially completed run should contain:

run.json
resources.jsonl
report.md

Step 12: Configure optional OCR

OCR is not required for normal HTML, PDF text, CSV, XLSX, JSON, or XML processing. Enable it only when the authorized scope includes images containing text.

Install Tesseract with the operating system package manager. On macOS with Homebrew:

brew install tesseract

On Debian or Ubuntu:

sudo apt-get update
sudo apt-get install tesseract-ocr

Install the Python OCR extra inside the project environment:

python -m pip install -e ".[dev,ocr]"

Confirm that Tesseract is available:

tesseract --version

Only installed Tesseract language packs can be used. OCR must not be treated as authoritative when the source image is unclear; the resulting resource should retain an appropriate verification status.

Step 13: Update or reinstall the server

After changing the source code or pulling a newer project version, reactivate the virtual environment and reinstall the editable package:

cd /ABSOLUTE/PATH/opencode-web-audit-mcp-python
source .venv/bin/activate
python -m pip install -e ".[dev]"
ruff check .
mypy src
pytest

Restart OpenCode after dependency or server-entry-point changes so that it starts a new MCP process.

Step 14: Uninstall or disable the server

To disable the MCP without deleting the project, set:

"enabled": false

Alternatively, remove the web_audit entry from the OpenCode configuration.

To remove the Python environment and generated runs after disabling the server:

rm -rf .venv
rm -rf runs/run_*

Review audit outputs before deleting them because they may contain the only persisted copy of a completed extraction.

Troubleshooting

OpenCode reports that the MCP command does not exist. Verify that the command path is absolute and points to .venv/bin/python or .venv\\Scripts\\python.exe. Recreate the virtual environment if the project directory was moved.

The server exits with ModuleNotFoundError. Activate the project environment and rerun python -m pip install -e ".[dev]". Confirm that the interpreter configured in OpenCode is the same interpreter used for installation.

OpenCode cannot discover the tools before the timeout. Increase timeout from 30000 to 60000, then inspect the server startup manually for import or configuration errors.

The server cannot write audit results. Confirm that WEB_AUDIT_OUTPUT_ROOT exists and is writable by the user running OpenCode. Do not configure a protected system directory.

A URL is rejected as outside the authorized scope. Ensure that authorized_domain matches the URL hostname exactly. Subdomains remain blocked unless the request explicitly authorizes them and the server policy permits them.

A page is skipped because of robots.txt. This is expected behavior. Do not disable or bypass the restriction. The skipped resource must remain recorded as blocked or inaccessible.

A website relies heavily on JavaScript. This server does not run a browser. Use only public HTML, linked resources, feeds, sitemaps, or publicly accessible JSON/XML endpoints exposed by the website.

PDF text or tables are incomplete. Text extraction quality depends on the PDF structure. Scanned PDFs require optional OCR, and visually complex tables may require a specialized downstream parser.

The crawl stops before the entire site is processed. Review report.md and run.json for maximum-page, maximum-depth, response-size, timeout, robots.txt, or format restrictions. Increase limits cautiously and preserve a minimum request delay.

MCP tools

web_audit_inspect_site performs a preflight inspection of the primary page, robots.txt, and sitemap candidates.

web_audit_crawl_site performs the bounded crawl, extraction, normalization, deduplication, persistence, and metric calculation.

web_audit_extract_resource processes one authorized URL.

web_audit_get_run retrieves a persisted run.

web_audit_audit_run checks duplicates, incomplete resources, potentially outdated dates, limitations, objective metrics, and completion criteria.

Inputs

Every acquisition tool requires primary_url or resource_url and authorized_domain. Optional inputs control maximum depth, maximum pages, languages of interest, priority sections, subdomain authorization, and allowed formats. Configuration values impose hard upper bounds even when a tool call requests larger values.

Outputs

A crawl tool response returns the run ID, state, output directory, metrics, and up to 100 errors. Each run directory contains:

runs/run_<uuid>/run.json
runs/run_<uuid>/resources.jsonl
runs/run_<uuid>/report.md

The Markdown report contains all 13 requested output sections and the required completion statement. Each resource record includes its canonical URL, title, type, section, author, publication and update dates, language, extracted content, structured data, tables, related files, related links, verification status, and notes.

Error handling

Input and scope violations fail before crawling. Individual resource failures are recorded without silently terminating a partially successful crawl. Blocked robots.txt routes, unsupported formats, HTTP failures, DNS/IP failures, excessive redirects, oversized responses, parser failures, OCR failures, and configured-limit stops are explicitly represented in records, errors, metrics, or completion criteria.

Deterministic completion criteria

A run is objectively complete only when the queue is empty; discovered routes are processed or recorded; normalization, source linkage, and deduplication checks pass; limitations are documented; and all required coverage metrics exist. Reaching a configured page or depth limit produces completed_with_errors, not an absolute completeness claim.

Three usage examples

See examples/prompts.md for copy-ready OpenCode prompts covering preflight inspection, a bounded institutional-site crawl, and a completed-run audit.

Compatibility checklist

See docs/COMPLETION-CHECKLIST.md. The critical checks are Python 3.11+, successful dependency installation, zero Ruff and mypy errors, passing tests, OpenCode MCP connectivity, discovery of five tools, safe URL rejection tests, run-artifact generation, and all required coverage metrics.

Known limitations

This server cannot guarantee discovery of routes that are neither linked nor present in sitemaps, feeds, public API responses, or returned HTML. It does not execute browser JavaScript. PDF table extraction is limited to text extraction; complex visual tables may require a specialized downstream parser. OCR supports only languages installed in the local Tesseract environment. XLS macro execution is never permitted.

License

This project is licensed under the MIT License. See the LICENSE file for the complete license text.