AgentOverFlow — Verified Solutions Layer

Stack Overflow's verified answers, but injected into the AI coding agent your developers already use — and owned by the company whose product the answers are about.

AgentOverFlow captures the real, working fixes for a developer product (from forums, GitHub issues, Discourse, Discord), validates them into a structured solution library, retrieves them with a three-signal hybrid engine (BM25 + dense vector + a verified-aware rerank), compresses each answer to what this question needs (the Re:Compress technique), and injects it into the developer's own AI session (Claude Code, Cursor) over MCP at the moment they're stuck.

See PRD.md for the full design and docs/EXPERIMENT_RESULTS.md for the validation results.

Does it work? (real NixOS experiment)

Same agent (deepseek-chat), three conditions, only the context changes. On 150 real NixOS Discourse solutions + 24 frozen tasks:

• B beats A by +50 pts (CI excludes zero) — the verified layer turns a 1-in-8 agent into a ~2-in-3 agent on niche, training-underrepresented problems.

• B beats C by +21 pts success (CI excludes zero) — validation+compression beats merely handing the model the forum text.

• B uses the fewest tokens. Full write-up + before/after iteration in docs/EXPERIMENT_RESULTS.md.

Why not just a docs chatbot

1. Validated solution pairs, not raw retrieval — a curated "this exact problem → this proven fix" library, so the system never serves a confidently-wrong answer. A trust layer, not just a retrieval layer.

2. Distribution into the dev's own tool — the answer shows up inside the coding agent the developer already uses, via MCP, not a chatbot they must visit.

3. Context-frugal injection — retrieved solutions are query-aware-compressed before they hit the agent's context, so we add signal without blowing the token budget.

Quickstart

uv venv --python 3.11
uv pip install -e .                 # base engine (pure-Python, no ML deps needed)
python scripts/make_seed.py         # build the seed solution DB

# query the hybrid retrieval engine
aof query "infinite recursion when overriding a package" --community seed

# (optional) full stack for real retrieval quality + LLM compression + scraping
uv pip install -e ".[all]"

The engine degrades gracefully: with no ML libraries installed it falls back to a pure-Python BM25 + NumPy cosine + hashing embeddings, so it always runs. Install .[retrieval] for real bm25s + FAISS + bge-m3 + bge-reranker-v2-m3 quality, .[llm] for DeepSeek-backed extraction/compression/grading (set DEEPSEEK_API_KEY), and .[mcp] for the MCP server.

Components (PRD §3)

• Ingest (pipeline/) — scrape → extract (DeepSeek) → normalize → dedupe → validation-score → index.

• Retrieval (retrieval/) — BM25 + dense vector → RRF fusion → verified-aware rerank → confidence gate.

• Compression (compress.py) — query-aware rewrite (Re:Compress technique) before injection.

• MCP (mcp/) — one find_solution tool over stdio, for Claude Code / Cursor.

• Experiment (experiment/) — the A/B/C harness that tests whether validation actually helps.

CLI

aof query "<problem>" [--error "..."] [--code "..."] [--version 24.05] [--community seed] [-k 5]
aof index --community seed          # build/rebuild index artifacts
aof seed                            # (re)generate the seed dataset
aof stats --community seed          # corpus + validation-signal breakdown

Status

MVP per PRD §10 milestones M0–M5. Built for the validation experiment: does a coding agent with a verified-solution MCP solve real problems more often, more correctly, and in fewer tokens than the same agent alone — and than the same agent given raw, unvalidated forum text?

License

MIT