# repo-rag

A chat agent over my own GitHub profile — ask it "has Aman used Kotlin?" or "is
CareerSpire finished?" and it answers in synthesized prose with citations, backed by a
tool-calling loop that decides for itself what to look up.

## The interesting decision

Most personal-profile RAG demos batch-embed every repo upfront, then do plain vector
search over the embeddings. This doesn't do that. At the scale of one person's GitHub
account, pre-embedding everything is wasted work — most repos never get asked about,
and the ones that do only need a few files read, not a full-repo index.

Instead, the LLM is given the responsibility of deciding what to explore:

- **Tier 1** (always available, no embeddings needed): every repo's name, language, and
  a short description, loaded once per session.
- **Tier 2** (on demand): the agent calls `search_index` first (checks a Postgres
  `pgvector` cache — free, no API cost). If the cache is empty or thin, it falls back to
  live GitHub calls (`list_repo_files` / `read_repo_file`). Whatever gets read live is
  chunked, embedded, and written into the cache as a deterministic side effect — not
  something the LLM decides to do — so the next question about that repo is cheaper.

This is a cache-augmented, lazy-indexing agent, not classic batch RAG and not pure live
browsing either. Full reasoning: `docs/LEARNINGS.md` (§8-9), diagram in
`docs/ARCHITECTURE.md`.

## Try it

**CLI** (streaming, human-in-the-loop confirmations, model routing all live here):

```bash
docker-compose up -d                          # Postgres + pgvector
uv sync
GROQ_MODEL=openai/gpt-oss-120b uv run repo-rag  # interactive session
```

Renders live as it streams: tool calls as they happen (with args/latency), the
answer's tokens as they generate, and citations — not a wall of JSON. Any live GitHub
API call (`list_repo_files`/`read_repo_file`) pauses for a y/n/always confirmation
first — a human-in-the-loop gate; `search_index`/`get_repo_status` never pause,
they're cache-only. Structured logs go to `repo-rag.log`, not the terminal.

**Web**: `docker-compose up --build` runs the FastAPI app too, serving a minimal chat
UI at `http://localhost:8000` (`POST /ask` non-streaming, `POST /ask/stream` for SSE —
the frontend uses the streaming one). No HITL here — pausing a live HTTP request for
out-of-band approval needs a different mechanism than the CLI's synchronous prompt;
accepted gap, not solved.

**MCP**: `uv run repo-rag-mcp` exposes the same four tools over the Model Context
Protocol (stdio transport) — usable from Claude Desktop, Claude Code, or any other
MCP client, independent of the Groq-specific agent loop above.

## Does it actually work?

`docs/eval/results.md` — a golden set of real questions run against the live agent,
scored for correctness and groundedness (not vibes). Methodology and the deliberate
choice *not* to use an LLM judge: `docs/eval/README.md`.

## What actually broke building this

`docs/INCIDENTS.md` — the real failures hit getting this running: a Postgres type-cast
bug, a Groq client rejecting a parameter it shouldn't have, a model that
*deterministically* mis-formats tool calls (not a flake — three different fix attempts
produced the identical failure before the real cause was found: the model, not the
prompt), and a token-budget collision that took several iterations to actually solve.
One of these — an error class the graceful-degradation handler didn't catch — was
found by running the eval harness, not by manual testing.

## What's built, with a named limit

- **Cache invalidation is SHA-based, not proactive.** Every cached chunk stores the
  GitHub blob SHA it was fetched at. `list_repo_files` reuses the tree listing it
  already has to compare live SHAs against cached ones and drops anything stale — no
  extra API calls, since GitHub returns blob SHAs for free. The limit: this only fires
  when the agent decides to list a repo's files again, not on every cached
  `search_index` hit. A repo that's re-committed but never gets `list_repo_files`
  called on it again keeps serving stale cached chunks. Renamed/deleted files also
  aren't caught — only SHA mismatches on paths that still exist. `docs/INCIDENTS.md`,
  `PgVectorStore.invalidate_stale`.
- **Rate-limit handling is tiered by wait length, not a queue.** Groq returns a real
  `retry-after` header. A short wait (its per-minute limit, resets in seconds) gets one
  bounded automatic retry. A long wait (its per-day quota, resets in minutes to hours —
  see `docs/INCIDENTS.md` #10) fails fast with the *actual* wait time surfaced to the
  user, instead of either hanging the CLI for 20 minutes or pretending "try rephrasing"
  would help. What's still missing: no persistent retry queue across restarts.
- **Multi-provider fallback is an abstraction, not a wired second provider.**
  `FallbackLLMClient` (`infrastructure/llm/fallback_client.py`) tries a list of clients
  in order, falling through only on the long-wait `RateLimitError` case above — no
  second provider's API key was available to actually wire one in, so it's
  ready-to-plug-in, tested against fakes, not proven against a real second provider.
- **Model routing is a two-tier cascade, not a learned router.** The first turn of
  every question uses a cheap/fast model (`llama-3.1-8b-instant`); any turn after that
  (meaning a tool call was actually needed) escalates to the capable model. A real
  question mix showed a good fraction resolve with zero tool calls, i.e. cheaply. No
  complexity classifier, no cost tracking dashboard — just this one cascade point.
- **Human-in-the-loop only exists on the CLI.** Live GitHub tool calls
  (`list_repo_files`/`read_repo_file`) pause for a synchronous y/n/always prompt there.
  The streaming HTTP route has no equivalent — pausing a request mid-stream for an
  out-of-band approval is a different, harder mechanism that wasn't built.
- **Streaming doesn't retry.** `GroqLLMClient.chat_stream()` skips the
  tool-use-failed/short-rate-limit retry logic the non-streaming `chat()` has, since
  retrying would mean discarding partially-streamed text. Accepted gap.

## What's deliberately not built

- **No hybrid search or reranking.** Flat cosine similarity only. Fine at
  single-profile scale; would need revisiting at more repos or more ambiguous queries.
- **No LLM-as-judge eval.** Deterministic fact-presence scoring instead, to keep eval
  runs cheap and reproducible on a free-tier account. Trade-off: can't catch a
  confidently wrong answer that happens to contain the right keywords.

## Docs

- `docs/PRD.md` / `docs/TRD.md` — product and technical requirements
- `docs/ARCHITECTURE.md` — module map, data flow
- `docs/LEARNINGS.md` — design reasoning for the cache-augmented approach
- `docs/INCIDENTS.md` — real bugs, symptom → diagnosis → fix
- `docs/eval/` — eval harness, methodology, results
- `docs/AI_ENGINEER_ROADMAP.md` — what this project covers against the AI Engineer
  curriculum, and what's genuinely left
- `docs/STATUS_AND_NEXT_STEPS.md` — what's working (verified, not assumed), what's
  left to close the zero-to-AI-engineer roadmap, what comes after