# OpenCode Web Audit MCP

A production-oriented, read-only Python MCP server for authorized website research, structured data extraction, downloadable-document analysis, and content auditing inside OpenCode.

## Purpose

The server builds an organized, verifiable, deduplicated representation of publicly accessible website content within an explicitly authorized domain. It processes reachable HTML, PDF, CSV, XLSX, JSON, XML, RSS, Atom, text, and supported images without bypassing restrictions.

## Activation rules

Use this MCP only when the user explicitly asks OpenCode to inspect, crawl, extract, inventory, or audit a public website and provides an authorized domain. Do not activate it for unrestricted Internet search, private systems, authenticated sessions, paywalls, CAPTCHAs, access-control bypass, vulnerability testing, or extraction from unauthorized external domains.

## Scope and constraints

The server uses read-only HTTP GET requests. It respects robots.txt, exact domain authorization, optional subdomain authorization, deterministic crawl limits, bounded concurrency, a minimum request delay, redirect validation, response-size limits, request timeouts, content-type allowlists, and SSRF defenses. External links are recorded but never fetched in the current run.

JavaScript is not executed. Therefore, dynamically loaded content is processed only when it is available through a public JSON/XML endpoint or an ordinary linked resource discovered in the returned source. OCR is optional and disabled by default. It requires the local Tesseract executable and the `ocr` dependency extra.

## Directory tree

```text
opencode-web-audit-mcp-python/
├── .env.example
├── .gitignore
├── AGENTS.md
├── LICENSE
├── README.md
├── SECURITY.md
├── opencode.jsonc
├── pyproject.toml
├── docs/
│   ├── ARCHITECTURE.md
│   ├── COMPLETION-CHECKLIST.md
│   └── TOOL-WORKFLOWS.md
├── examples/
│   ├── prompts.md
│   └── sample_policy.json
├── runs/
│   └── .gitkeep
├── src/opencode_web_audit_mcp/
│   ├── __init__.py
│   ├── config.py
│   ├── models.py
│   ├── server.py
│   ├── core/
│   │   ├── auditor.py
│   │   ├── crawler.py
│   │   ├── http_client.py
│   │   ├── report.py
│   │   └── robots.py
│   ├── extractors/
│   │   ├── document.py
│   │   └── html.py
│   ├── security/
│   │   ├── content_policy.py
│   │   └── url_policy.py
│   └── storage/
│       └── run_store.py
└── tests/
    ├── test_html.py
    ├── test_report.py
    └── test_url_policy.py
```

## Requirements and dependencies

Python 3.11 or newer is required. Runtime dependencies are declared in `pyproject.toml`: the official `mcp` Python SDK, HTTPX, Beautiful Soup, lxml, Pydantic, pypdf, openpyxl, python-dateutil, and defusedxml. Development dependencies include pytest, coverage, mypy, and Ruff.

Optional OCR requires Tesseract installed by the operating system plus:

```bash
pip install -e ".[ocr]"
```

## Step-by-step installation and configuration

The following procedure installs the MCP server in an isolated Python environment, validates the project, registers it as a local OpenCode MCP server, and confirms that its tools are available. The primary instructions target macOS and Linux. Windows-specific path examples are included where they differ.

### Step 1: Install the prerequisites

Install the following software before continuing:

- Python 3.11 or newer.
- OpenCode.
- Git, if the project is being cloned from a repository.
- Tesseract only when optional image OCR is required.

Confirm that Python and OpenCode are available:

```bash
python3 --version
opencode --version
```

The Python version must be 3.11 or newer. If `python3` resolves to an older version, use the executable for an installed supported version, such as `python3.11`, `python3.12`, or `python3.13` in the following commands.

### Step 2: Extract or clone the project

Extract the ZIP file or clone the repository into a permanent location. Do not configure OpenCode against a temporary download directory that may later be moved or deleted.

Example location on macOS or Linux:

```text
/Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python
```

Example location on Windows:

```text
C:\Users\YOUR_USERNAME\Developer\opencode-web-audit-mcp-python
```

Open a terminal and enter the project directory:

```bash
cd /ABSOLUTE/PATH/opencode-web-audit-mcp-python
```

Confirm that the expected project files are present:

```bash
pwd
ls -la
```

At minimum, the directory must contain `pyproject.toml`, `README.md`, `opencode.jsonc`, `src/`, and `tests/`.

### Step 3: Create the Python virtual environment

Create a dedicated virtual environment inside the project directory:

```bash
python3.11 -m venv .venv
```

If a different supported Python executable is installed, substitute it directly:

```bash
python3.12 -m venv .venv
```

Activate the environment on macOS or Linux:

```bash
source .venv/bin/activate
```

Activate it on Windows PowerShell:

```powershell
.venv\Scripts\Activate.ps1
```

Verify that the active interpreter belongs to the project:

```bash
python -c "import sys; print(sys.executable)"
```

The printed path must point to `.venv/bin/python` on macOS/Linux or `.venv\Scripts\python.exe` on Windows.

### Step 4: Install the project and dependencies

Upgrade the Python packaging tools:

```bash
python -m pip install --upgrade pip setuptools wheel
```

Install the MCP server in editable mode with its development and validation dependencies:

```bash
python -m pip install -e ".[dev]"
```

This command installs the official MCP Python SDK and all runtime dependencies declared in `pyproject.toml`.

Confirm that the server package and MCP SDK can be imported:

```bash
python -c "import mcp; import opencode_web_audit_mcp; print('Imports successful')"
```

### Step 5: Run the project validation checks

Run the static checks and automated tests before registering the server in OpenCode:

```bash
ruff check .
mypy src
pytest
```

All three commands must complete successfully. A failed check should be corrected before the MCP is enabled in OpenCode.

To run the tests with an explicit coverage report:

```bash
pytest --cov=opencode_web_audit_mcp --cov-report=term-missing
```

### Step 6: Create the output directory

The server writes each audit run to a dedicated directory under `runs/`. Create the directory and verify that the current user can write to it:

```bash
mkdir -p runs
python -c "from pathlib import Path; p=Path('runs/.write_test'); p.write_text('ok'); p.unlink(); print('Output directory is writable')"
```

Use a different output directory only when necessary. The process started by OpenCode must have write access to that location.

### Step 7: Determine the absolute paths

OpenCode should be configured with absolute paths so that the MCP can be started from any workspace.

On macOS or Linux, obtain the project path with:

```bash
pwd
```

Obtain the virtual-environment Python path with:

```bash
python -c "import sys; print(sys.executable)"
```

Example resolved values:

```text
Project directory: /Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python
Python executable: /Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python/.venv/bin/python
Output directory: /Users/YOUR_USERNAME/Developer/opencode-web-audit-mcp-python/runs
```

Do not leave `/ABSOLUTE/PATH` placeholders in the final OpenCode configuration.

### Step 8: Configure the MCP server in OpenCode

OpenCode starts a local MCP server as a subprocess and communicates with it through the MCP `stdio` transport. Add the following entry to the applicable `opencode.json` or `opencode.jsonc` file and replace every placeholder with the absolute paths determined in Step 7.

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "web_audit": {
      "type": "local",
      "command": [
        "/ABSOLUTE/PATH/opencode-web-audit-mcp-python/.venv/bin/python",
        "-m",
        "opencode_web_audit_mcp.server"
      ],
      "cwd": "/ABSOLUTE/PATH/opencode-web-audit-mcp-python",
      "enabled": true,
      "timeout": 30000,
      "environment": {
        "WEB_AUDIT_OUTPUT_ROOT": "/ABSOLUTE/PATH/opencode-web-audit-mcp-python/runs",
        "WEB_AUDIT_MAX_PAGES": "1000",
        "WEB_AUDIT_MAX_DEPTH": "5",
        "WEB_AUDIT_CONCURRENCY": "3",
        "WEB_AUDIT_MIN_DELAY_MS": "750",
        "WEB_AUDIT_LOG_LEVEL": "info"
      }
    }
  },
  "permission": {
    "web_audit_*": "ask"
  }
}
```

A completed macOS example would look similar to:

```jsonc
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "web_audit": {
      "type": "local",
      "command": [
        "/Users/alice/Developer/opencode-web-audit-mcp-python/.venv/bin/python",
        "-m",
        "opencode_web_audit_mcp.server"
      ],
      "cwd": "/Users/alice/Developer/opencode-web-audit-mcp-python",
      "enabled": true,
      "timeout": 30000,
      "environment": {
        "WEB_AUDIT_OUTPUT_ROOT": "/Users/alice/Developer/opencode-web-audit-mcp-python/runs",
        "WEB_AUDIT_MAX_PAGES": "1000",
        "WEB_AUDIT_MAX_DEPTH": "5",
        "WEB_AUDIT_CONCURRENCY": "3",
        "WEB_AUDIT_MIN_DELAY_MS": "750",
        "WEB_AUDIT_LOG_LEVEL": "info"
      }
    }
  },
  "permission": {
    "web_audit_*": "ask"
  }
}
```

For Windows, use the virtual-environment interpreter path and escape backslashes in JSON:

```jsonc
"command": [
  "C:\\Users\\YOUR_USERNAME\\Developer\\opencode-web-audit-mcp-python\\.venv\\Scripts\\python.exe",
  "-m",
  "opencode_web_audit_mcp.server"
]
```

The `permission` rule is intentionally set to `ask`. It requires explicit approval before an audit tool is used. Change it only after reviewing OpenCode's permission model and the server's access scope.

### Step 9: Verify that OpenCode can start the MCP server

Run:

```bash
opencode mcp list
```

The `web_audit` server should be listed as enabled and connected. If OpenCode reports a startup failure, run the exact configured command manually from the configured working directory:

```bash
cd /ABSOLUTE/PATH/opencode-web-audit-mcp-python
/ABSOLUTE/PATH/opencode-web-audit-mcp-python/.venv/bin/python -m opencode_web_audit_mcp.server
```

A `stdio` MCP server normally waits for protocol input and may appear idle when started manually. The important checks are that it does not immediately exit with an import error, path error, configuration error, or traceback. Stop the manual process with `Ctrl+C` after the check.

### Step 10: Confirm tool discovery inside OpenCode

Start OpenCode in a workspace and ask it to list or use the tools provided by the `web_audit` MCP server. OpenCode should discover these five tools:

```text
web_audit_inspect_site
web_audit_crawl_site
web_audit_extract_resource
web_audit_get_run
web_audit_audit_run
```

A suitable first request is:

```text
Use the web_audit MCP server to inspect https://example.org with example.org as the authorized domain. Do not crawl beyond depth 1 and do not process more than 20 pages.
```

OpenCode should request permission before tool execution when the supplied `permission` configuration is used.

### Step 11: Run a bounded first audit

Begin with a small, deterministic crawl before increasing limits:

```text
Use web_audit_inspect_site for https://example.org. The authorized domain is example.org. Allow HTML, PDF, JSON, XML, CSV, and XLSX. Record external links but do not fetch them.
```

After reviewing the preflight result, run:

```text
Use web_audit_crawl_site for https://example.org. The authorized domain is example.org. Set maximum depth to 2, maximum pages to 50, concurrency to 2, and minimum request delay to 1000 milliseconds. Respect robots.txt and do not fetch external domains.
```

The response should include a run ID and output directory. Verify the generated artifacts:

```bash
find runs -maxdepth 2 -type f -print
```

Each completed or partially completed run should contain:

```text
run.json
resources.jsonl
report.md
```

### Step 12: Configure optional OCR

OCR is not required for normal HTML, PDF text, CSV, XLSX, JSON, or XML processing. Enable it only when the authorized scope includes images containing text.

Install Tesseract with the operating system package manager. On macOS with Homebrew:

```bash
brew install tesseract
```

On Debian or Ubuntu:

```bash
sudo apt-get update
sudo apt-get install tesseract-ocr
```

Install the Python OCR extra inside the project environment:

```bash
python -m pip install -e ".[dev,ocr]"
```

Confirm that Tesseract is available:

```bash
tesseract --version
```

Only installed Tesseract language packs can be used. OCR must not be treated as authoritative when the source image is unclear; the resulting resource should retain an appropriate verification status.

### Step 13: Update or reinstall the server

After changing the source code or pulling a newer project version, reactivate the virtual environment and reinstall the editable package:

```bash
cd /ABSOLUTE/PATH/opencode-web-audit-mcp-python
source .venv/bin/activate
python -m pip install -e ".[dev]"
ruff check .
mypy src
pytest
```

Restart OpenCode after dependency or server-entry-point changes so that it starts a new MCP process.

### Step 14: Uninstall or disable the server

To disable the MCP without deleting the project, set:

```jsonc
"enabled": false
```

Alternatively, remove the `web_audit` entry from the OpenCode configuration.

To remove the Python environment and generated runs after disabling the server:

```bash
rm -rf .venv
rm -rf runs/run_*
```

Review audit outputs before deleting them because they may contain the only persisted copy of a completed extraction.

### Troubleshooting

**OpenCode reports that the MCP command does not exist.** Verify that the `command` path is absolute and points to `.venv/bin/python` or `.venv\\Scripts\\python.exe`. Recreate the virtual environment if the project directory was moved.

**The server exits with `ModuleNotFoundError`.** Activate the project environment and rerun `python -m pip install -e ".[dev]"`. Confirm that the interpreter configured in OpenCode is the same interpreter used for installation.

**OpenCode cannot discover the tools before the timeout.** Increase `timeout` from `30000` to `60000`, then inspect the server startup manually for import or configuration errors.

**The server cannot write audit results.** Confirm that `WEB_AUDIT_OUTPUT_ROOT` exists and is writable by the user running OpenCode. Do not configure a protected system directory.

**A URL is rejected as outside the authorized scope.** Ensure that `authorized_domain` matches the URL hostname exactly. Subdomains remain blocked unless the request explicitly authorizes them and the server policy permits them.

**A page is skipped because of `robots.txt`.** This is expected behavior. Do not disable or bypass the restriction. The skipped resource must remain recorded as blocked or inaccessible.

**A website relies heavily on JavaScript.** This server does not run a browser. Use only public HTML, linked resources, feeds, sitemaps, or publicly accessible JSON/XML endpoints exposed by the website.

**PDF text or tables are incomplete.** Text extraction quality depends on the PDF structure. Scanned PDFs require optional OCR, and visually complex tables may require a specialized downstream parser.

**The crawl stops before the entire site is processed.** Review `report.md` and `run.json` for maximum-page, maximum-depth, response-size, timeout, robots.txt, or format restrictions. Increase limits cautiously and preserve a minimum request delay.

## MCP tools

`web_audit_inspect_site` performs a preflight inspection of the primary page, robots.txt, and sitemap candidates.

`web_audit_crawl_site` performs the bounded crawl, extraction, normalization, deduplication, persistence, and metric calculation.

`web_audit_extract_resource` processes one authorized URL.

`web_audit_get_run` retrieves a persisted run.

`web_audit_audit_run` checks duplicates, incomplete resources, potentially outdated dates, limitations, objective metrics, and completion criteria.

## Inputs

Every acquisition tool requires `primary_url` or `resource_url` and `authorized_domain`. Optional inputs control maximum depth, maximum pages, languages of interest, priority sections, subdomain authorization, and allowed formats. Configuration values impose hard upper bounds even when a tool call requests larger values.

## Outputs

A crawl tool response returns the run ID, state, output directory, metrics, and up to 100 errors. Each run directory contains:

```text
runs/run_<uuid>/run.json
runs/run_<uuid>/resources.jsonl
runs/run_<uuid>/report.md
```

The Markdown report contains all 13 requested output sections and the required completion statement. Each resource record includes its canonical URL, title, type, section, author, publication and update dates, language, extracted content, structured data, tables, related files, related links, verification status, and notes.

## Error handling

Input and scope violations fail before crawling. Individual resource failures are recorded without silently terminating a partially successful crawl. Blocked robots.txt routes, unsupported formats, HTTP failures, DNS/IP failures, excessive redirects, oversized responses, parser failures, OCR failures, and configured-limit stops are explicitly represented in records, errors, metrics, or completion criteria.

## Deterministic completion criteria

A run is objectively complete only when the queue is empty; discovered routes are processed or recorded; normalization, source linkage, and deduplication checks pass; limitations are documented; and all required coverage metrics exist. Reaching a configured page or depth limit produces `completed_with_errors`, not an absolute completeness claim.

## Three usage examples

See `examples/prompts.md` for copy-ready OpenCode prompts covering preflight inspection, a bounded institutional-site crawl, and a completed-run audit.

## Compatibility checklist

See `docs/COMPLETION-CHECKLIST.md`. The critical checks are Python 3.11+, successful dependency installation, zero Ruff and mypy errors, passing tests, OpenCode MCP connectivity, discovery of five tools, safe URL rejection tests, run-artifact generation, and all required coverage metrics.

## Known limitations

This server cannot guarantee discovery of routes that are neither linked nor present in sitemaps, feeds, public API responses, or returned HTML. It does not execute browser JavaScript. PDF table extraction is limited to text extraction; complex visual tables may require a specialized downstream parser. OCR supports only languages installed in the local Tesseract environment. XLS macro execution is never permitted.

## License

This project is licensed under the MIT License. See the [`LICENSE`](LICENSE) file for the complete license text.