BlackBook MCP v0.8.0

Source-Grounded Cybersecurity and bugbounty Knowledge & Research MCP

The read-only cybersecurity knowledge & research teammate: source-grounded search, exact citations, a knowledge graph, and investigation context, running alongside an execution MCP.

📖 What It Is • 🏗️ Architecture • 🚀 Installation • 🛠️ MCP Tools • 🕸️ Knowledge Graph • 🔒 Security

BlackBook MCP is a source-grounded cybersecurity knowledge & research server that speaks the Model Context Protocol (MCP). It is the research teammate that runs alongside an execution MCP such as HexStrike inside Claude Code, Cursor, VS Code, or any MCP-compatible client.

                CLAUDE / AI AGENT
                       |
     +------------+------------+
     |                         |
          v                         v
      HEXSTRIKE              BlackBook MCP
      EXECUTION                   KNOWLEDGE
          |                         |
   +------+------+         +--------+--------+
   |      |      |         |        |        |
  Nmap  ffuf  nuclei   HackTricks  0xdf   ATT&CK
   |      |      |         |        |        |
   +------+------+         +--------+--------+
          |                         |
          +------------+------------+
                       v
                AI REASONING LOOP

• HexStrike answers: "what can I execute or test?"

• BlackBook answers: "what is documented about this situation, which similar cases exist, which techniques are relevant, and what source material supports that conclusion?"

Claude is the orchestrator.

Read-only by design. BlackBook never runs commands, scans hosts, or exploits targets. It indexes a controlled corpus and retrieves source-grounded knowledge with exact, verifiable citations. Execution belongs to a separate MCP.

Architecture Overview

BlackBook MCP v0.8.0 is a source-grounded knowledge system: every query flows through a hybrid retrieval facade, is enriched (never gated) by a knowledge graph, and returns results that resolve to exact, verifiable citations. Nothing is executed.

%%{init: {"themeVariables": {
  "primaryColor": "#7f1d1d",
  "secondaryColor": "#dc2626",
  "tertiaryColor": "#ef4444",
  "background": "#1a0505",
  "edgeLabelBackground":"#7f1d1d",
  "fontFamily": "monospace",
  "fontSize": "15px",
  "fontColor": "#fee2e2",
  "nodeTextColor": "#fee2e2"
}}}%%
graph TD
    A[AI Agent - Claude / Cursor / VS Code] -->|MCP Protocol over stdio| B[BlackBook MCP Server v0.8.0]

    B --> C[Hybrid Retrieval Facade]
    B --> D[11 Knowledge Tools]
    B --> E[Knowledge Graph]

    C --> F[FTS5 BM25 - always on]
    C --> G[Local Semantic - optional]
    C --> H[Reranker + Source Diversity]

    D --> I[knowledge_search]
    D --> J[knowledge_source]
    D --> K[knowledge_technique]
    D --> L[knowledge_case_search]
    D --> M[knowledge_research]
    D --> N[knowledge_context]

    E --> O[Technique / Tool / Service / OS]
    E --> P[Writeup / Source entities]
    E --> Q[Evidence-linked edges]

    B --> R[Corpus - SQLite FTS5 + JSON1]
    R --> S[HackTricks]
    R --> T[0xdf Writeups]
    R --> U[Local PDFs]
    R --> X["MITRE ATT&CK"]
    R --> Y[GTFOBins + LOLBAS + LOOBins]
    R --> Z[Payloads + Recipes + WADComs]
    R --> AA[InternalAllTheThings + HTB Writeups]

    B --> V[Exact Citations and Provenance]
    V --> W[chunk_id resolves to verifiable excerpt]

    style A fill:#7f1d1d,stroke:#ef4444,stroke-width:3px,color:#fee2e2
    style B fill:#dc2626,stroke:#7f1d1d,stroke-width:4px,color:#ffffff
    style C fill:#ef4444,stroke:#7f1d1d,stroke-width:2px,color:#1a0505
    style D fill:#ef4444,stroke:#7f1d1d,stroke-width:2px,color:#1a0505
    style E fill:#ef4444,stroke:#7f1d1d,stroke-width:2px,color:#1a0505
    style R fill:#ef4444,stroke:#7f1d1d,stroke-width:2px,color:#1a0505
    style V fill:#ef4444,stroke:#7f1d1d,stroke-width:2px,color:#1a0505

How It Works

1. AI Agent Connection: Claude, Cursor, VS Code, or any MCP-compatible client connects over stdio. The server owns stdout for the JSON-RPC protocol; every byte of banner/log chrome goes to stderr, so the stream is never corrupted.

2. Source-Grounded Retrieval: a query flows through metadata filters → FTS5 BM25 (always available) → optional local semantic search → reranking → a per-document cap that enforces source diversity.

3. Graph Enrichment: the knowledge graph annotates technique dossiers and similar-case results with evidence-linked edges. It enhances retrieval and never gates it: everything works with an empty graph.

4. Verifiable Provenance: every result resolves through its chunk_id to the exact indexed excerpt. BlackBook never fabricates a citation.

5. Read-Only by Design: no command execution, host scanning, or arbitrary URL fetching through tool parameters. Execution belongs to a separate MCP such as HexStrike.

Related MCP server: Open Mind

What it is

A hybrid knowledge system, not a query→embedding→dump pipeline:

query → metadata filter → FTS5 (BM25) → [optional semantic] → rerank →
source diversity → provenance → exact citations

Lexical retrieval (SQLite FTS5) is the always-available backbone. Semantic search is optional and local. Nothing is presented as fact unless it traces to an indexed source chunk.

Features (current phase)

• Source-grounded search across 26 configured sources (26 enabled by default): HackTricks, 0xdf writeups, MITRE ATT&CK, GTFOBins, LOLBAS, LOOBins, WADComs, PayloadsAllTheThings, The Hacker Recipes, Internal All The Things, Moamen Basel's HTB writeups, local PDFs, OWASP WSTG, OWASP ASVS, OWASP API Security, the Bug Bounty Cheatsheet, PortSwigger Academy, Google Bug Hunters, Bugcrowd VRT, GitHub Security Lab research, Hacker101, and four public HackerOne report/index repositories, plus WebHackList. Report archives are labeled with unknown authority and should be treated as local research data, not official HackerOne guidance.

• Exact, verifiable citations: every reference resolves to real indexed text

• Structure-preserving chunking: heading breadcrumbs and code blocks intact

• Hybrid retrieval facade with reranking + source diversity: lexical (FTS5 BM25) always on, local semantic embeddings merged in when enabled

• Local semantic search (sentence-transformers, offline): paraphrased queries with no keyword overlap still find the right chunk; degrades gracefully to lexical when the extra isn't installed

• Source filtering & platform/category filters

• Knowledge graph (Technique/Tool/Service/OS/Writeup/Source) built from the index; evidence-linked edges enrich technique dossiers and case search without ever gating retrieval

• Modular ingestion via a SourceAdapter interface (add sources without a rewrite)

• CLI for ingestion, search, graph, stats, sources, diagnostics

• MCP server over stdio (default) for Claude Code / Cursor / VS Code, with an optional streamable-http transport (serve --http) for a shared network server

Installation

Requires Python ≥ 3.10.

# with uv (recommended)
uv pip install -e .

# or with pip
pip install -e .

# optional: semantic/embedding search (Phase 3)
uv pip install -e ".[semantic]"

# development / tests
uv pip install -e ".[dev]"

This installs two CLI entry points: blackbook and cyber-knowledge (alias).

Configuration

BlackBook reads, in increasing priority: built-in defaults → a YAML config file → BLACKBOOK_* environment variables.

cp config.example.yaml ~/.blackbook/config.yaml
# edit paths/sources; see config.example.yaml for every option

Key settings:

home: ~/.blackbook                 # data dir (db, caches, raw checkouts)
sources:
  - id: hacktricks
    enabled: true
  - id: "0xdf"                     # quote hex-like ids (YAML parses 0xdf as 223)
    enabled: true
  - id: local_pdfs
    enabled: true
    directory: ~/knowledge/pdfs
    authority: user                # NOT assumed authoritative
  - id: attack                     # MITRE ATT&CK STIX bundle (authority: official)
    enabled: true
embeddings:
  enabled: false                   # set true + install [semantic] for local semantic search
  model: sentence-transformers/all-MiniLM-L6-v2
  device: cpu
retrieval:
  default_limit: 8
  per_document_cap: 2              # source diversity

Twenty-six sources are configured and enabled by default (run blackbook sources to list them). Website sources are bounded to their configured origin and optional path_prefix; they skip non-HTML assets and respect max_files, max_document_bytes, and request_delay. GitHub-backed sources accept a few extra keys: ref (branch), include_glob (which files to index), exclude_glob (skip repo plumbing / link indexes), content_root (restrict to a subtree), and site_url (map citations to the published site instead of the GitHub blob URL; a Jekyll permalink in a page's front matter wins over the path-derived URL). See config.example.yaml.

Initial ingestion

blackbook ingest --source hacktricks   # markdown book (tarball over HTTPS)
blackbook ingest --source 0xdf         # HTB/CTF writeups
blackbook ingest --source attack       # MITRE ATT&CK STIX bundle (~54 MB download)
blackbook ingest --source gtfobins     # Unix binary abuse (YAML corpus)
blackbook ingest --source lolbas       # Windows living-off-the-land binaries
blackbook ingest --source loobins       # macOS living-off-the-land binaries
blackbook ingest --source wadcoms       # offensive Windows/AD command cheat sheets
blackbook ingest --source payloads     # PayloadsAllTheThings
blackbook ingest --source hacker_recipes   # The Hacker Recipes
blackbook ingest --source internal_all_the_things   # AD / internal network cheat sheets
blackbook ingest --source htb_writeups     # Moamen Basel's HTB writeups + cheatsheets
blackbook ingest --source owasp_wstg        # OWASP web testing methodology
blackbook ingest --source owasp_asvs        # OWASP verification requirements
blackbook ingest --source owasp_api_security # OWASP API security guidance
blackbook ingest --source bugbounty_cheatsheet # practical bug bounty workflow
blackbook ingest --source portswigger       # Web Security Academy guidance
blackbook ingest --source google_bug_hunters # Google bug bounty guidance
blackbook ingest --source bugcrowd_vrt      # vulnerability severity taxonomy
blackbook ingest --source github_security_lab # GitHub Security Lab research
blackbook ingest --source hacker101         # HackerOne educational material
blackbook ingest --source hackerone_reports_index # report index
blackbook ingest --source hackerone_disclosed_reports # report bodies
blackbook ingest --source hackerone_reports_metadata # report metadata
blackbook ingest --source hackerone_bug_bounty_reports # report index
blackbook ingest --source webhacklist      # web hacking technique archive
blackbook ingest                        # all enabled sources

# bound the size during a first run:
#   set `max_files: 25` on a source in config.yaml

ingest is incremental by default. For the normal daily workflow, use blackbook update: it checks every enabled source, downloads only changed GitHub revisions or missing cached pages, skips unchanged documents by content hash, prints status=up-to-date for sources with no changes, and continues if another source has an error. New sources are ingested normally on the same run. Use blackbook ingest --force only when you explicitly need a full refresh. Citation metadata (URLs, titles) is refreshed in place when it drifts, so a URL-mapping fix or a source re-publishing under new permalinks self-heals without re-chunking or new chunk ids.

WebHackList ingests its complete Markdown repository, including yearly lists, archived references, and evaluation notes. It enables exact normalized cross-document deduplication, so repeated chunks keep the first citation while unique material remains searchable.

PDF ingestion

# point the local_pdfs source at your directory in config.yaml, then:
blackbook ingest --source local_pdfs

PDFs are chunked per page with page-number citations. They default to authority: user and are not treated as authoritative.

Searching

blackbook search "kerberoasting"
blackbook search "windows service privilege escalation" --source hacktricks
blackbook search "NTLM relay" --platform windows --limit 5
blackbook search "crack service account passwords" --mode semantic  # paraphrase-friendly
blackbook stats [--json]        # corpus counts (machine-readable with --json)
blackbook sources [--json]
blackbook graph build     # (re)build the knowledge graph from the index
blackbook graph show [--json]   # graph entity/relationship counts
blackbook doctor          # diagnostics: db, index, sources, embeddings
blackbook rebuild-index   # rebuild the FTS5 index
blackbook case export MY-CASE   # export an investigation case as Markdown
blackbook backup          # snapshot the knowledge base (VACUUM INTO)

platform and categories are hard filters: results only come from documents carrying the tag (e.g. windows/linux, htb, Easy/Insane). The MCP tools' techniques parameter resolves through the controlled vocabulary and biases results toward chunks whose heading names the technique; unknown terms are searched as plain keywords and flagged in the response note.

Search modes: hybrid (default, lexical + semantic), keyword (FTS5 only), semantic (embeddings only), plus two intent-biased modes: technique (nudges canonical technique/reference material up) and case_similarity (favours hands-on writeups). The intent modes nudge ranking, they never filter results out. Semantic and hybrid use vectors only when embeddings.enabled and the [semantic] extra is installed; otherwise they fall back to lexical automatically.

Semantic embeddings

With embeddings.enabled: true and the [semantic] extra, ingestion embeds new chunks inline. To (re)build the semantic index without re-ingesting:

blackbook embed                       # embed chunks missing a current-model vector
blackbook embed --source local_pdfs   # only one source
blackbook embed --reembed             # drop existing vectors first, then re-embed

Embeddings are computed locally and never leave the machine. blackbook doctor reports coverage (N/M embedded).

Transport modes

BlackBook speaks MCP over two transports. stdio is the default and is what you almost always want.

• stdio — the client (Claude Code, Cursor, VS Code) owns the process: it spawns blackbook serve on session start, talks over stdin/stdout, and stops it on exit. Nothing to launch by hand. This is the mode all the setup snippets below use.

• streamable-http — a long-lived network server you run yourself, reachable at http://<host>:<port>/mcp with a GET /health check. Handy for a shared instance or when you want the banner in front of you. It is not something a client auto-launches, so don't register an HTTP endpoint that isn't already running or the client will just fail to connect each session.

blackbook serve                       # stdio (default)
blackbook serve --http                # streamable-http on 127.0.0.1:8890/mcp
blackbook serve --http --port 9000    # override port
blackbook serve --http --host 0.0.0.0 # bind all interfaces (see auth note below)

Equivalently, set BLACKBOOK_TRANSPORT=streamable-http (or sse) in the environment. HTTP host/port/path come from the server: block in config.yaml (defaults 127.0.0.1 / 8890 / /mcp); --host and --port override them for a single run.

Bind safety. BlackBook refuses to start on a non-loopback address (e.g. 0.0.0.0) unless a bearer token is set, guarding against an accidentally exposed, unauthenticated server. Set server.auth_token in config.yaml (or BLACKBOOK_SERVER__AUTH_TOKEN) and clients must then send Authorization: Bearer <token>. Only flip require_auth_off_loopback: false if you understand the exposure.

Claude Code setup

claude mcp add blackbook -- blackbook serve            # this project (local scope)
claude mcp add blackbook -s user -- blackbook serve    # every directory (user scope)

Use -s user to register it once for all your projects; Claude Code then auto-launches it everywhere over stdio. Or put it in your MCP config (.mcp.json / ~/.config/claude/...):

{
  "mcpServers": {
    "blackbook": { "command": "blackbook", "args": ["serve"] }
  }
}

Using a virtualenv? Point command at it: "/home/you/venv/bin/blackbook".

Cursor

Settings → MCP → Add server:

{ "mcpServers": { "blackbook": { "command": "blackbook", "args": ["serve"] } } }

VS Code

.vscode/mcp.json (with an MCP-capable extension):

{ "servers": { "blackbook": { "command": "blackbook", "args": ["serve"] } } }

Startup banner

Launching the server prints a banner and then streams status logs. Every byte of this chrome goes to stderr; stdout is reserved for the JSON-RPC protocol, so the banner and logs never corrupt an MCP client's stream.

██████╗ ██╗      █████╗  ██████╗██╗  ██╗██████╗  ██████╗  ██████╗ ██╗  ██╗
██╔══██╗██║     ██╔══██╗██╔════╝██║ ██╔╝██╔══██╗██╔═══██╗██╔═══██╗██║ ██╔╝
██████╔╝██║     ███████║██║     █████╔╝ ██████╔╝██║   ██║██║   ██║█████╔╝ 
██╔══██╗██║     ██╔══██║██║     ██╔═██╗ ██╔══██╗██║   ██║██║   ██║██╔═██╗ 
██████╔╝███████╗██║  ██║╚██████╗██║  ██╗██████╔╝╚██████╔╝╚██████╔╝██║  ██╗
╚═════╝ ╚══════╝╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═════╝  ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
  Source-grounded cybersecurity knowledge & research MCP
  v0.8.0  ·  stdio  ·  read-only · no execution · every claim cited
  corpus  <live database count> sources · <live count> docs · <live count> chunks · <live count> embeddings
  graph   <live count> entities · <live count> relationships · <live count> cases

In a real terminal the wordmark is gradient-lit (cyan→indigo, intentionally distinct from an execution MCP's red). The corpus and graph lines always reflect the live database, so the counts above are placeholders rather than a fixed snapshot. The transport line reflects how you started it (stdio, or streamable-http · http://127.0.0.1:8890/mcp under --http, where the MCP endpoint and /health URL are also printed). Suppress the banner with blackbook serve --no-banner.

Status and log lines use a compact, level-styled prefix, showing the successes and failures at a glance:

[+] Embedded 18630 chunks. Total vectors: 18630     success  (green)
[*] server ready                                    info     (cyan)
[!] Graph rebuild skipped: no chunks changed        warning  (yellow)
[-] hacktricks: fetch failed (offline)              error    (red)

Rich strips the colour automatically when output is piped or redirected, so log files stay clean.

Available MCP tools

Only implemented tools are registered; nothing is stubbed or faked.

Example Claude Code interaction

You:    What does HackTricks document about Kerberoasting, and has 0xdf
        covered a similar HTB machine?

Claude: (calls knowledge_search {query: "kerberoasting", sources: ["hacktricks","0xdf"]})
        HackTricks documents Kerberoasting under Active Directory → Kerberos …
        Similar 0xdf case: HTB: Forest …
        [cites chunk refs]

Claude: (calls knowledge_source {chunk_id: …} to read the exact section)
        Here's the exact HackTricks enumeration procedure …

Knowledge graph

A lightweight graph of Technique / Tool / Service / OS / Writeup / Source entities and their relationships (documented_by, demonstrated_in, uses, targets, runs_on, …), derived from the already-indexed corpus, with no fetching or execution. Every non-structural edge carries the document it was extracted from (evidence_doc_id), a confidence, and an inferred flag; nothing is fabricated, and a citation always resolves to real indexed text.

The graph enhances retrieval, it never gates it: search and both new tools work with an empty graph and simply gain neighbours/annotations once it is built.

blackbook graph build     # (re)build the graph from the index, full and idempotent
blackbook graph show      # current entity/relationship counts, no rebuild

Ingesting also refreshes the graph automatically (skip with ingest --no-graph). Two tools consume it:

• knowledge_technique: returns which sources document a technique, which tools/services/writeups the graph associates with it (each edge with confidence and its backing document), plus real cited excerpts. Works before the graph exists; it always returns indexed references.

• knowledge_case_search: finds hands-on writeups similar to a situation and, when the graph is built, annotates each with the techniques it demonstrates.

Retrieval architecture

See docs/retrieval.md. FTS5 BM25 is always available; semantic search is an optional local backend merged into the same facade. Reranking combines lexical score, source authority, platform/category match, and keyword overlap, then a per-document cap enforces source diversity.

Source provenance

Every claim carries provenance. knowledge_search returns a ref (chunk_id, doc_id, source, url, page, section_path); knowledge_source resolves it to the exact indexed text. BlackBook never fabricates a citation.

Troubleshooting

blackbook doctor --verbose     # full diagnostics

• Unknown or disabled source: check blackbook sources; quote "0xdf" in YAML.

• Empty results: run blackbook ingest first; check blackbook stats.

• PDF dir missing: set sources[].directory for local_pdfs.

• Logs: add --verbose to any command for structured debug output.

Security model

BlackBook is read-only with respect to external systems, confines filesystem reads to configured knowledge directories, validates all tool inputs, and never fetches arbitrary URLs through tool parameters. See docs/security.md.

Development

uv pip install -e ".[dev]"
python -m pytest tests          # run the suite

Layout:

src/blackbook/
  config.py            layered settings
  server.py            FastMCP wiring (stdio)
  mcp/                 tool schemas + implementations
  ingestion/           SourceAdapter + per-source adapters + pipeline
  retrieval/           lexical / hybrid / reranker / chunking
  knowledge/           source resolution (citation -> excerpt)
  storage/             SQLite (FTS5 + JSON1), models, migrations
  cli/                 Typer CLI
  utils/               path-safety helpers
tests/                 unit + fixtures (+ integration)
docs/                  architecture, ingestion, retrieval, mcp, security

Testing

python -m pytest tests -q

Tests cover chunking, storage/FTS5 sync, every source adapter (offline fixtures: HackTricks, 0xdf, GitHub markdown, GTFOBins, LOLBAS, ATT&CK STIX), retrieval & reranking, semantic embeddings & hybrid merge, MCP tools, provenance round-trips, and path safety. Semantic tests use a deterministic model-free embedder so they run offline with no model download; one real-model test skips cleanly when the [semantic] extra isn't installed. PDF tests use a generated PDF and skip if reportlab isn't installed.

Roadmap

• Phase 1: MCP server, SQLite+FTS5, HackTricks + 0xdf ingestion, search, citations

• Phase 2: font-aware PDF adapter (heading/code detection, page-level citations), cross-document near-duplicate detection, structural chunking, CLI

• Phase 3: local embeddings (all-MiniLM-L6-v2), hybrid retrieval, reranking

• Phase 4: knowledge graph, technique relationships, case similarity

• Phase 5: knowledge_research (observation → source-grounded packet), knowledge_context (local investigation state)

• Phase 6: offline evaluation suite (blackbook eval), citation-integrity gate, FTS5 optimize on ingest, adversarial/hardening tests

• Phase 7: generic GitHub source adapter (tarball over HTTPS, config-driven) with PayloadsAllTheThings, The Hacker Recipes, GTFOBins, LOLBAS, LOOBins, WADComs; MITRE ATT&CK STIX source with technique-dossier enrichment

License

MIT. See LICENSE.