🛡️ Super RAG

An offline, agentic AI for penetration testing — your private cybersecurity brain.

Ask pentest questions, run reconnaissance, and exploit-chain over 61,000+ chunks of curated security knowledge — entirely on your own machine. No cloud. No telemetry. No data leaves your box.

💡 Why Super RAG?

Enterprise security copilots (Microsoft Security Copilot, CrowdStrike Charlotte) live in the cloud, cost a fortune, and are built for defenders. Super RAG flips that:

• 🔒 Fully offline — runs against a local LLM in LM Studio. Perfect for air-gapped labs, sensitive engagements, and regions with restricted cloud access.

• ⚔️ Built for offensive reasoning — not just "summarize this alert," but "I see Apache 2.4.49 — what's my next move?" and it chains recon → vuln-ID → exploitation.

• 📚 Grounded in real knowledge — indexes 1,110 hand-curated notes plus HackTricks, PayloadsAllTheThings, the OWASP cheat sheets, and hundreds of CTF write-ups. Every answer is cited back to its source file.

• 🧩 Plugs into your agents — exposed as an MCP server, so Hermes, OpenClaw, Claude Desktop, or any MCP client gains a cybersec_search tool instantly.

⚠️ For authorized use only. This is a tool for pentesters, CTF players, and security researchers operating with explicit written permission. See Responsible Use.

Related MCP server: Security Context MCP Server

✨ Features

🏗️ Architecture

flowchart LR
    subgraph Knowledge["📚 Knowledge (61,552 chunks)"]
        V["1,110 curated notes"]
        D["HackTricks · PayloadsAllTheThings · OWASP"]
        W["CTF / HTB write-ups"]
    end
    subgraph Engine["🧠 Super RAG"]
        I["ingest.py<br/>chunk + embed (parallel)"]
        Q[("Qdrant<br/>hybrid index")]
        R["rag_engine.py<br/>13 RAG strategies"]
    end
    subgraph Local["💻 LM Studio (localhost:1234)"]
        E["nomic-embed-text"]
        L["gpt-oss-20b"]
    end
    A["agent.py<br/>pentest loop"]
    M["rag_mcp.py<br/>MCP server :8765"]

    V & D & W --> I --> Q
    I -.embeddings.-> E
    Q --> R --> L
    R --> A
    R --> M
    M --> Hermes & OpenClaw & Claude["Claude Desktop"]

Three tiers, depth over scale: embedded Qdrant (no Docker) for hybrid search → 13 combined RAG strategies for context assembly → a local LLM for generation.

🔬 The 13 RAG strategies (combined on every query)

Most projects use #1 and wonder why retrieval is mediocre. Super RAG layers 13, each earning its place in a pentest workflow:

🏆 Model benchmark (16 local models, real pentest scenario)

We benchmarked every model in LM Studio on a 2-turn pentest tool-use scenario (recon → exploit chain), measuring speed, accuracy, and valid tool-loops — ejecting each model between runs for clean numbers.

Key finding: raw "accuracy" is misleading for agents — several high-scoring models produced empty output because they reasoned internally without ever emitting an actionable tool call. The metric that matters is valid tool-loops, and gpt-oss-20b wins it. Reproduce with python model_benchmark.py.

🚀 Quick start

Prerequisites

• LM Studio on localhost:1234 with nomic-embed-text-v1.5 (embeddings) + a chat model loaded

• Python 3.10+ (tested on 3.14.5)

• An RTX-class GPU recommended (built on an RTX 5090; the embedder + a 12 GB chat model fit in 24 GB VRAM)

# 1. Install (no torch, no Docker, no HuggingFace needed)
pip install -r requirements.txt

# 2. Point config.py → VAULT_DIR at your knowledge folder, then index it
python main.py ingest          # one-time; resumable; ~minutes with parallel embedding

# 3. Ask anything
python main.py query "how do I exploit Apache 2.4.49 path traversal"
python main.py query "AS-REP roasting — what tool and command?"

# 4. Run the agentic pentest loop (hybrid: auto-recon, manual exploit approval)
python main.py pentest 10.10.10.5 --scope 10.10.10.0/24

# 5. Generate a report from the session findings
python main.py report

# 6. Health check
python main.py status

🤝 Use it from your AI agents (MCP)

Super RAG runs as one shared MCP server (rag_mcp.py, HTTP @ 127.0.0.1:8765/mcp) so multiple agents can query it concurrently:

python rag_mcp.py        # or let Startup\SuperRAG-MCP.cmd auto-start it

Tools exposed: cybersec_search(query, phase) · cybersec_answer(question) · cybersec_status()

Register it in any MCP client:

// Claude Desktop / OpenClaw style
"mcp": { "servers": { "cybersec-rag": {
  "url": "http://127.0.0.1:8765/mcp", "transport": "streamable-http"
}}}

# Hermes style (config.yaml)
mcp_servers:
  cybersec-rag: { url: http://127.0.0.1:8765/mcp, enabled: true }

See INTEGRATION.md for the full Hermes + OpenClaw walkthrough.

🎬 It works — real agent output

Scenario: black-box target, nmap reveals Apache httpd 2.4.49.

STEP 1  recon       → nmap -sS -A 10.10.10.5          ✓ correct first move (2.9s)
STEP 2  enumerate   → gobuster on :80 (Apache live)    ✓ methodical (3.3s)
STEP 3  exploit     → curl --path-as-is "…/cgi-bin/.%2e/…/bin/bash" -d 'reverse shell'
                      ✓ textbook CVE-2021-41773 mod_cgi RCE (4.9s)
        sources: htb-cpts/initial-access-exploitation.md, oswe/file-inclusion-upload.md, HackTricks

The model followed correct methodology, grounded each step in the vault, and produced a working exploit chain — fully offline.

📁 Project layout

super-rag/
├── main.py              # CLI: ingest · query · pentest · report · extract · status
├── config.py            # paths, model IDs, chunking, timeouts, tool registry
├── ingest.py            # vault → chunks → parallel embed → Qdrant  (~14× faster pipeline)
├── rag_engine.py        # the 13 RAG strategies + RRF fusion
├── agent.py             # ReAct+Reflect pentest loop, scope + approval gates
├── detector.py          # honeypot / WAF / firewall detection
├── memory.py            # attack-surface graph, findings, stuck-loop tracking
├── report.py            # pentest report + bug-bounty submission generators
├── llm.py               # one streaming chat helper (works for every model)
├── rag_mcp.py           # MCP server for agent integration
├── model_benchmark.py   # the 16-model benchmark harness
├── extract_training.py  # real write-ups → fine-tuning JSONL
└── tools/               # registry + nmap/gobuster/ffuf parsers

🧰 Tech stack

Python 3.14 · Qdrant (embedded) · LM Studio (OpenAI-compatible local API) · nomic-embed-text-v1.5 · gpt-oss-20b · MCP / FastMCP — zero cloud dependencies.

🔐 Responsible use

• Authorized targets only. Super RAG does not enforce authorization — that is your legal responsibility. Use it on systems you own or have explicit written permission to test (engagements, CTFs, labs).

• Hybrid mode gates exploitation behind a manual [y/N] approval. Don't bypass it.

• Indexed external repos are reference-only and untrusted — never execute code pulled from them.

• This project is for defensive learning, authorized testing, and CTF/education. Don't be a criminal.

🛣️ Roadmap

• Qdrant server mode (Docker) for fully-concurrent multi-agent access

• GRPO fine-tune of a 7–14B specialist on extracted real write-ups

• Web UI (the CLI works today)

• Auto-scoping from engagement rules-of-engagement files

🤝 Contributing

Issues and PRs welcome — new tool parsers, RAG strategies, and detector signatures especially. Keep it defensive, keep it cited.

📄 License

MIT — see LICENSE.