OSINT Workbench

A self-hostable integration workbench for collecting, preserving and searching open-source information, with a read-only MCP interface for AI assistants. It combines a Python application, PostgreSQL, immutable evidence storage and optional local inference or explicitly governed cloud enrichment.

Development status: collection, evidence reads, local retrieval, enrichment, recovery and supervised service paths are implemented within documented bounds. The complete reference platform remains unfinished. STATUS.md separates implemented behavior, revision-bound verification and remaining acceptance; the roadmap and backlog retain the full intended scope.

What works today

The vulnerability profile currently provides an offline advisory-observation contract and KEV-format document ingestion. A complete applicability/prioritization engine and regulatory-intelligence profile remain roadmap work. Source claims, catalogue membership and model output do not establish target exposure or legal applicability.

Related MCP server: Proofpage MCP Server

Start with the core

Native Ubuntu installation uses the distro Python in a virtual environment: Ubuntu 24.04/Python 3.12 or Ubuntu 26.04/Python 3.14. See runtime support for package prerequisites. From a checkout, this core smoke check needs no database, model, cloud account or configured sources:

uv sync --locked --no-dev --python /usr/bin/python3
.venv/bin/osint-workbench collect --config config/sources.example.json

Expected output: {"status": "unconfigured", "sources": 0}. Dependency installation uses the configured package registry; this command performs no source acquisition. The bundled source list is empty. Keep real source URLs, credentials, prompts, private inventories and deployment configuration outside Git and public CI.

For a working database-backed installation, follow deployment and the supervised profile. Provision PostgreSQL 18, separate migration/collector/writer/reader identities and persistent evidence storage before running migrations or workers. Optional hybrid retrieval adds pgvector and the retrieval dependency extra; it also needs explicitly selected model artifacts, manifests and an activated compatible index generation.

No model or provider is selected or downloaded automatically. Remote enrichment requires an explicit route, data class, private policy and credential. There is no silent cloud fallback. An MCP client can disclose returned content to its own provider, even when this application runs locally. A verified hash establishes byte integrity; retrieved content remains untrusted.

Operator workflows

osint-workbench diagnose reports sanitized runtime and configuration-presence facts without application I/O. Add --check-reader for an explicit reader-authority and schema check; this is not readiness or deployment qualification. See runtime diagnostics.

Use only the credentials needed by each process. Collection never falls back to writer credentials; the MCP server rejects broader reader authority. Enforce read-only evidence access at the OS/mount boundary as well as in Python. Recovery, lifecycle, purge and accounting have separately scoped credentials and effects. Private checkpoints are not consumer exports.

Operator configuration/import paths must name regular files, with no final symlink. Feed configuration is capped at 8 MiB; text import is capped at 4,000,000 UTF-8 bytes and 1,000,000 Unicode code points. See the file-admission boundary.

Development and verification

CPython 3.12–3.14 is permitted. Required installed/native CI qualifies Ubuntu 24.04/Python 3.12 and Ubuntu 26.04/Python 3.14. Python 3.13 has separate optional weekly/manual coverage; Python 3.15 has a manual dependency-only probe and remains outside the package bound. Neither optional profile gates merges or main-build signing.

uv sync --locked --extra retrieval --python /usr/bin/python3
.venv/bin/ruff check .
.venv/bin/ruff format --check .
.venv/bin/python scripts/check_repository.py
.venv/bin/python scripts/check_static.py
.venv/bin/python -m pytest -m 'not integration' --import-mode=importlib
uv build --python /usr/bin/python3 --build-constraints build-constraints.txt --require-hashes
.venv/bin/python scripts/check_publication.py --artifacts dist

These checkout checks are development feedback. CI installs the built wheel into a separate environment and requires the full native suite against disposable PostgreSQL/pgvector, HTTP/TLS/Unix transports and synthetic ONNX/GGUF fixtures. CONTRIBUTING.md documents the exact procedure, pinned native dependency and --require-integration gate. Its database fixture must be named ow_test; recovery tests create/drop their own ow_test_restore_ databases. Never use operational storage or resynchronize an installed-wheel test environment. Separate disposable service tests validate systemd and credential isolation.

Synthetic execution tests establish bounded contracts. They do not establish pretrained semantic quality, representative capacity, GPU support, actual-provider behavior, full-host recovery or complete acceptance of all 51 roadmap scenarios. Current results must identify their tested source and artifact. See STATUS.md, the verification programme and the alignment review.

Engineering and attribution

This is AI-assisted integration work. Human responsibility includes requirements, design decisions, review, validation and operations. Existing libraries supply database, parsing, model-runtime and protocol capabilities; this project does not claim original model research or production assurance.

The engineering baseline uses ADRs, STPA-derived constraints, explicit trust boundaries, least privilege, bounded work, failure/recovery tests, locked builds and publication review. NIST SSDF and selected OWASP ASVS controls are engineering references; the control evidence records bounded coverage and gaps, without a certification or regulatory-conformance claim. Signed main-build evidence binds tested bytes and workflow identity; it does not establish vulnerability freedom or complete redistribution clearance.

Read AGENTS.md, architecture, decisions, the security policy and secure development for the governing contracts. The documentation map distinguishes current procedures, target specifications and historical evidence.