# AgentOverFlow — Verified Solutions Layer

**Stack Overflow's verified answers, but injected into the AI coding agent your developers already use — and owned by the company whose product the answers are about.**

AgentOverFlow captures the real, working fixes for a developer product (from forums, GitHub issues, Discourse, Discord), **validates** them into a structured solution library, retrieves them with a **three-signal hybrid engine** (BM25 + dense vector + a verified-aware rerank), **compresses** each answer to what *this* question needs (the Re:Compress technique), and **injects** it into the developer's own AI session (Claude Code, Cursor) over **MCP** at the moment they're stuck.

See [`PRD.md`](./PRD.md) for the full design and [`docs/EXPERIMENT_RESULTS.md`](./docs/EXPERIMENT_RESULTS.md) for the validation results.

## Does it work? (real NixOS experiment)

Same agent (`deepseek-chat`), three conditions, only the context changes. On 150 real NixOS Discourse solutions + 24 frozen tasks:

| Condition | Success | Output tokens |
|---|---|---|
| **A** — agent alone | 12.5% | 162 |
| **B** — + our verified+compressed tool | **62.5%** | **100** |
| **C** — + raw unvalidated forum text | 41.7% | 118 |

- **B beats A by +50 pts** (CI excludes zero) — the verified layer turns a 1-in-8 agent into a ~2-in-3 agent on niche, training-underrepresented problems.
- **B beats C by +21 pts success** (CI excludes zero) — validation+compression beats merely handing the model the forum text.
- **B uses the fewest tokens.** Full write-up + before/after iteration in [`docs/EXPERIMENT_RESULTS.md`](./docs/EXPERIMENT_RESULTS.md).

## Why not just a docs chatbot

1. **Validated solution pairs, not raw retrieval** — a curated "this exact problem → this proven fix" library, so the system never serves a confidently-wrong answer. A *trust* layer, not just a *retrieval* layer.
2. **Distribution into the dev's own tool** — the answer shows up inside the coding agent the developer already uses, via MCP, not a chatbot they must visit.
3. **Context-frugal injection** — retrieved solutions are query-aware-compressed before they hit the agent's context, so we add signal without blowing the token budget.

## Quickstart

```bash
uv venv --python 3.11
uv pip install -e .                 # base engine (pure-Python, no ML deps needed)
python scripts/make_seed.py         # build the seed solution DB

# query the hybrid retrieval engine
aof query "infinite recursion when overriding a package" --community seed

# (optional) full stack for real retrieval quality + LLM compression + scraping
uv pip install -e ".[all]"
```

The engine **degrades gracefully**: with no ML libraries installed it falls back to a pure-Python BM25 + NumPy cosine + hashing embeddings, so it always runs. Install `.[retrieval]` for real `bm25s` + FAISS + `bge-m3` + `bge-reranker-v2-m3` quality, `.[llm]` for DeepSeek-backed extraction/compression/grading (set `DEEPSEEK_API_KEY`), and `.[mcp]` for the MCP server.

## Components (PRD §3)

- **Ingest** (`pipeline/`) — scrape → extract (DeepSeek) → normalize → dedupe → validation-score → index.
- **Retrieval** (`retrieval/`) — BM25 + dense vector → RRF fusion → verified-aware rerank → confidence gate.
- **Compression** (`compress.py`) — query-aware rewrite (Re:Compress technique) before injection.
- **MCP** (`mcp/`) — one `find_solution` tool over stdio, for Claude Code / Cursor.
- **Experiment** (`experiment/`) — the A/B/C harness that tests whether validation actually helps.

## CLI

```bash
aof query "<problem>" [--error "..."] [--code "..."] [--version 24.05] [--community seed] [-k 5]
aof index --community seed          # build/rebuild index artifacts
aof seed                            # (re)generate the seed dataset
aof stats --community seed          # corpus + validation-signal breakdown
```

## Status

MVP per PRD §10 milestones M0–M5. Built for the validation experiment: does a coding agent with a verified-solution MCP solve real problems more often, more correctly, and in fewer tokens than the same agent alone — and than the same agent given raw, unvalidated forum text?

## License

MIT