# intent-index

**Local-first retrieval over a long design conversation, for auditing an implementation against what was actually decided.**

[![license: MIT](https://img.shields.io/badge/license-MIT-22d3ee.svg)](LICENSE)
[![node: 24](https://img.shields.io/badge/node-24-3c873a.svg)](mise.toml)
[![runs locally](https://img.shields.io/badge/index-stays%20on%20disk-7c3aed.svg)](#requirements)

![The path from a transcript to a citable answer](docs/diagrams/answer-path.svg)

## The problem

A long design conversation is the most accurate record of a system that exists,
and it is the least usable. Decisions are stated once, in the middle of a turn,
in prose. Months later an engineer asks three questions and gets three different
answers:

- Where in the conversation was this decided?
- Why was it decided that way, and what was rejected instead?
- Has it changed since?

A file-based search answers none of them. It finds the word, not the ruling. It
cannot tell you that the ruling you are reading was replaced in turn 26.

## What it does

One pipeline, three mechanisms, each kept in its lane.

| Mechanism | Answers | Why it is the right tool |
|---|---|---|
| Hybrid retrieval | where something was decided | finds passages by meaning and by term, over transcript sections and extracted records together |
| Typed judgment | why, and has it changed | reranks the shortlist and reports whether the corpus answers the question at all, as a probability rather than a vibe |
| Exact search | what the words literally were | reads the authoritative transcript only, so wording can be confirmed without trusting a generated file |

Everything a reviewer receives carries a turn number and a line range. A
citation without coordinates cannot be checked, so an item without coordinates
is not returned as evidence.

## Quick start

Requirements: [mise](https://mise.jdx.dev) and `pnpm`. The toolchain is pinned in
`mise.toml`; dependencies install from the committed `pnpm-lock.yaml`.

```sh
mise install
pnpm install
pnpm build
```

Build the index over the sample conversation in `examples/`. This needs no
credentials:

```sh
node dist/cli.js build --no-generative
pnpm doctor
pnpm eval
```

That gives you sections, a hybrid index, and a manifest. Add a generative pass
to extract decision records and write the intent map:

```sh
export OPENROUTER_API_KEY=...
node dist/cli.js build
```

Point `intent-index.yml` at your own transcript to index a real conversation.
Nothing in the configuration references a path outside the repository.

## Ask it things

```sh
node dist/cli.js retrieve "why was the reorder buffer rejected"
node dist/cli.js grep "quarantine lane"
node dist/cli.js read --turn 6
node dist/cli.js read --turn 6 --context 4
node dist/cli.js trace D0010
node dist/cli.js overview
node dist/cli.js doctor
node dist/cli.js eval
```

`trace` is the one worth understanding. It walks a record's history: what it
supersedes, what supersedes it, and the transcript lines behind each step. When
a ruling changes, the correction is visible rather than implied.

`grep` reads the transcript and nothing else. Generated records and the intent
map are navigation aids, so they are excluded from exact search unless you ask
for them by name. If a generated artifact and the transcript disagree, the
transcript is right.

Add `--json` to any command for machine-readable output, and `--dir <path>` to
operate on another project directory.

## Tools over MCP

`intent-index mcp` serves five tools over stdio:

| Tool | Purpose |
|---|---|
| `overview` | corpora, source hashes, counts, intent map sections |
| `retrieve` | evidence for a question, with an answer-exists probability |
| `grep` | exact search over the authoritative transcript |
| `read` | transcript lines by turn number or line range |
| `trace` | one record's history: what it supersedes and what supersedes it |

Registration for a harness that speaks MCP:

```json
{
  "mcpServers": {
    "intent-index": {
      "command": "node",
      "args": ["/absolute/path/to/intent-index/dist/cli.js", "mcp"],
      "env": { "TYPESAFE_API_KEY": "..." }
    }
  }
}
```

## What gets indexed

Every document is a small markdown file with typed metadata the engine can
filter on.

| Kind | One document per | Carries |
|---|---|---|
| `section` | structural section inside a turn | turn number, speaker, transcript line range |
| `decision` | extracted design record | record kind, status, supersedes, superseded by, turn, line range |
| `intent` | intent map section | section name, and no turn: a summary spans the conversation |
| `document` | file in a `documents` corpus | corpus name |

Sections come from the transcript's own structure: turn headings, pseudo
headings, and paragraph blocks grouped to a target size. No model decides where
a boundary falls, so a rebuild reproduces the same identifiers and line ranges
byte for byte.

## Configuration

```yaml
corpora:
  - name: atlas
    path: examples/atlas-transcript.md
    role: conversation      # conversation | documents
    ledger: true            # extract records and an intent map for this corpus

generative:
  baseUrl: https://openrouter.ai/api/v1
  model: deepseek/deepseek-v4.1-flash
  apiKeyEnv: OPENROUTER_API_KEY
  reasoning: false

judgment:
  model: jev-latest
  apiKeyEnv: TYPESAFE_API_KEY
  rerank: true

retrieval:
  limit: 10
  candidateLimit: 40
```

Generative calls are cached by a hash of their inputs under
`.intent-index/cache/generative`, so a rebuild replays identical artifacts
instead of paying for them twice.

Three settings matter for reliability on a long transcript:

- `concurrency` caps calls in flight. 8 is what the measured throughput supports.
- `reasoning` is off. These passes are mechanical extraction. When the model
  spends output tokens reasoning first, it exhausts its budget before writing
  the structured answer and the provider returns an empty completion.
- Turn text is windowed to 6000 characters per call, so one oversized turn
  cannot fail as a single request.

Structured output is schema-validated, so a reply that does not match the
requested shape fails the call instead of entering the ledger. A window whose
generation fails is reported by name and left out of the ledger; it does not
fail the build silently.

The intent map is written one section at a time and each section is checked for
its heading, its content, and a citation on every bullet. A section that cannot
be written fails the build, because a shorter map that looks complete is worse
than an error.

## Verification

`pnpm eval` runs probes whose evidence location is already known: each probe
names the turns that carry the answer and, where applicable, a literal phrase
that must exist there.

| Corpus | Turns | Documents | Records | Probes | recall@10 | MRR | exact recall |
|---|---|---|---|---|---|---|---|
| `examples/atlas-transcript.md`, shipped here | 13 | 72 | 49 | 6 | 1.00 | 0.92 | 1.00 |
| a 7,971-line private workshop transcript | 57 | 1,141 | 828 | 12 | 1.00 | 0.64 | 0.75 |

The second row is a real corpus that is not shipped with this repository. It is
listed because a sample corpus is easy to be good at, and a 7,971-line
conversation is not.

`doctor` checks the index against its own manifest: source files still hash to
what was recorded, every document on disk matches its digest, no stray documents
exist, every record carries a turn, all ten intent map sections are present, and
the engine sees the same document count the build reported.

```
ok   manifest: 72 artifacts
ok   sources: 1 sources unchanged
ok   documents present: 72 files
ok   no stray documents: none
ok   document digests: all match
ok   decision turn coordinates: 49 records carry a turn
ok   intent sections: 10 sections
ok   index document count: 72 indexed, 72 expected
```

`pnpm check` typechecks and `pnpm lint` lints. A change is done when those
pass, `doctor` is green, and `eval` has not regressed.

## Requirements

- Node 24, pinned through `mise.toml`.
- `OPENROUTER_API_KEY` for the passes that extract records and write the intent
  map.
- `TYPESAFE_API_KEY` for the judgment rerank.

Both are read from the environment. Put them in `.env` beside
`intent-index.yml`; that file is git-ignored and `.env.example` is the committed
template. Retrieval, exact search, reading, tracing, and a structure-only build
work without either key.

The index is a local SQLite file under `.intent-index/`, alongside the rendered
documents and the manifest. It is disposable build output: delete it and rebuild.
Nothing is sent anywhere except the generative and judgment calls you configure,
and those carry the transcript windows they are given.

## Layout

```
src/corpus/       normalization: transcript to turns and sections
src/index/        document rendering and the build pipeline
src/ledger/       record extraction, turn digests, intent map
src/judgment/     typed judgment calls over a retrieval shortlist
src/retrieval/    query planning, retrieval, exact search, reading, tracing
src/mcp/          the five tools over stdio
src/eval/         probe runner
docs/diagrams/    the diagram above, as SVG
docs/design-decisions.md   every decision this system was built from
examples/         the sample conversation and its probe set
```

`docs/design-decisions.md` records each choice behind the architecture with the
question, the criteria, and the answer it received. It is the design authority
for this repository: a change that contradicts a recorded decision is a decision
to revisit, not an implementation detail.

## License

MIT. See [LICENSE](LICENSE).