# pentest-guide-kb

A read-only, versioned knowledge base for OWASP testing guides, served over
the Model Context Protocol (MCP) so AI agents and IDEs can query them
precisely instead of relying on training-data memory.

> **This project is not an official OWASP project and is not endorsed by
> OWASP.** It compiles and re-serves content from independent OWASP testing
> guides under their own licenses -- see [Licensing](#licensing). No content
> from this project's retrieval or recommendation logic is official OWASP
> guidance; always cite the guide name, version, and source commit a result
> traces back to.

## What it does

- Precise lookup of a specific OWASP test by id (versioned or canonical).
- Natural-language search across test procedures (full-text + semantic).
- Browsing a guide by version, category, platform, or feature keyword.
- Cross-guide relationship traversal, with provenance on every edge.
- Version comparison for a test across two compiled guide releases.
- Candidate-test generation for a target profile -- explicitly labeled
  unverified candidates, never a completed test plan.

Every result carries full provenance (repository, commit, path, content hash,
license), so a citation always traces back to the exact upstream source.

**It is a knowledge *retrieval* service, not an offensive security execution
platform.** It does not — and will not — execute scanning/exploitation tools,
expose a shell, reach a caller-specified target, act as an autonomous pentest
agent, or mutate data over MCP (source sync and ingestion are offline CLI-only
steps). See [`SECURITY.md`](SECURITY.md) for the full threat model.

## Architecture

```text
Official OWASP Git Repositories (wstg, mastg, owasp-istg, www-project-ai-testing-guide)
        |
        v
Source Mirror and Version Lock        sources/sources.lock.yaml, sources/.cache/
        |
        v
Deterministic Guide Compilers         src/pentest_guide_kb/ingestion/*
        |
        v
Canonical Registry (PostgreSQL)       src/pentest_guide_kb/storage/*
        |
        +-- metadata + JSONB (test_cases, test_case_sections, ...)
        +-- full-text search (tsvector + GIN)
        +-- pgvector semantic search (embeddings)
        +-- Generated Markdown Wiki    src/pentest_guide_kb/wiki/*
        +-- Optional Neo4j projection  src/pentest_guide_kb/graph/*  (--profile graph)
        |
        v
Read-only MCP Server                  src/pentest_guide_kb/mcp/*
        |
        v
AI Agents and IDE Clients
```

**The PostgreSQL registry is the single source of truth.** The Wiki, the
vector index, and the Neo4j graph are derived views compiled *from* it --
none independently store content that could drift out of sync. See
[`docs/architecture.md`](docs/architecture.md) and
[`docs/data-model.md`](docs/data-model.md).

## Supported guides

| Guide | Short name | Upstream repository | Pinned |
|---|---|---|---|
| OWASP Web Security Testing Guide | WSTG | `OWASP/wstg` | tag `v4.2` |
| OWASP Mobile App Security Testing Guide | MASTG | `OWASP/mastg` | tag `v2.0.0` |
| OWASP IoT Security Testing Guide | ISTG | `OWASP/owasp-istg` | tag `v1.0.1` |
| OWASP AI Testing Guide | AITG | `OWASP/www-project-ai-testing-guide` | `main` commit (no tagged release yet) |

Exact pins live in `sources/sources.lock.yaml` (see
[`docs/versioning.md`](docs/versioning.md)). `latest` is never a valid version
anywhere -- it's rejected by a validator.

## Quickstart

Requires Python 3.12+, [`uv`](https://docs.astral.sh/uv/), and Docker.

```bash
uv sync --all-extras
cp .env.example .env                    # local-dev defaults, no real secrets
docker compose up -d postgres           # PostgreSQL + pgvector
uv run alembic upgrade head             # apply the schema

uv run pentest-guide source sync --guide wstg   # fetch a pinned source
uv run pentest-guide ingest --all               # compile into the registry
uv run pentest-guide index embeddings           # build the semantic index

uv run pentest-guide query search "credentials over http" --mode hybrid
uv run pentest-guide mcp serve                  # stdio; --transport streamable-http for HTTP
```

Prefer a container? Pull the prebuilt multi-arch image instead of building:
`docker pull ghcr.io/sunick2009/mcp-owasp-pentesting-guide:latest` -- full
run/serve and MCP-client wiring in [`docs/deployment.md`](docs/deployment.md).

## Documentation

| Topic | Doc |
|---|---|
| Deploy & connect a client (Compose, GHCR image, MCP config) | [`docs/deployment.md`](docs/deployment.md) |
| Local dev loop (setup, sync, ingest, tests, adding a guide) | [`docs/development.md`](docs/development.md) |
| MCP API — 11 tools, 10 resource templates, 5 prompts | [`docs/mcp-api.md`](docs/mcp-api.md) |
| Architecture & the single-source-of-truth model | [`docs/architecture.md`](docs/architecture.md) |
| Data model & schema | [`docs/data-model.md`](docs/data-model.md) |
| Ingestion — per-guide parsers & determinism | [`docs/ingestion.md`](docs/ingestion.md) |
| Retrieval — classify → exact → FTS/vector → RRF → rerank | [`docs/retrieval.md`](docs/retrieval.md) |
| Versioning — canonical vs versioned ids, pinning | [`docs/versioning.md`](docs/versioning.md) |
| Licensing & attribution chain | [`docs/licensing.md`](docs/licensing.md) |
| Source pinning workflow | [`sources/README.md`](sources/README.md) |
| Security threat model | [`SECURITY.md`](SECURITY.md) |
| What's implemented vs. known limitations | [`IMPLEMENTATION_STATUS.md`](IMPLEMENTATION_STATUS.md) |
| Glossary | [`docs/glossary.md`](docs/glossary.md) |

## Licensing

- This project's own code: **Apache-2.0** (`LICENSE`).
- Each upstream guide's content: **CC BY-SA 4.0**, per `sources/licenses/` --
  carried through on every compiled `TestCase.source.license`.

This repository does not bulk-copy OWASP guide text: source sync + local
compilation + provenance reference is the model; committed fixtures under
`tests/fixtures/` are short, attributed excerpts. See `NOTICE` and
[`docs/licensing.md`](docs/licensing.md) for the full attribution chain.

## Key file paths

| What | Where |
|---|---|
| Domain models | `src/pentest_guide_kb/domain/models.py` |
| Guide parsers | `src/pentest_guide_kb/ingestion/{wstg,mastg,istg,aitg}.py` |
| Storage / repositories | `src/pentest_guide_kb/storage/` |
| Retrieval pipeline | `src/pentest_guide_kb/retrieval/` |
| MCP server | `src/pentest_guide_kb/mcp/{server,resources,tools,prompts}.py` |
| CLI | `src/pentest_guide_kb/cli.py` |
| Wiki generator + Error Book | `src/pentest_guide_kb/wiki/` |
| Neo4j projection (optional) | `src/pentest_guide_kb/graph/` |
| Source lock | `sources/sources.lock.yaml` |
| Curated relationships | `registry/relationships/cross-guide.yaml` |
| Agent Skill | `skills/owasp-guide-research/SKILL.md` |
| Tests | `tests/{unit,integration,contract,evals}/` |