# VectorMCP — central cross-repo code index over MCP

Gives Claude Code one central, always-warm index over every Ikshana repo. State a
task, get back the repos responsible for it, then search and read their code
without cloning anything.

## The constraint this is built around

Claude Code clones a fresh copy of a repo for each task and throws it away. So
**index identity is git object identity, never filesystem location**: a chunk's
Qdrant point id is `uuid5(NS, "{repo_id}:{blob_sha}:{chunk_idx}")`. Blob shas are
identical in every clone, forever, which means:

- a fresh clone at any commit needs **zero** re-indexing;
- indexing runs server-side against bare mirrors, never in the agent's checkout;
- the agent only ever contributes *deltas* for its uncommitted edits, into a
  disposable `scope=session:<id>` overlay.

Measured on this codebase: a single commit changes **0.5–0.8%** of indexable
files, so an incremental pass re-embeds one or two files rather than 927.

## Layout

| path | what |
|---|---|
| `codeindex/ignore.py` | what not to index — the highest-leverage file here |
| `codeindex/gitmirror.py` | git access; works on bare mirrors *or* existing checkouts |
| `codeindex/scan.py` | commit → manifest, plus the reviewable report |
| `codeindex/db.py` | Postgres: manifests, embedding cache, ACL |
| `codeindex/vectors.py` | Qdrant collections, point ids, visibility filters |
| `codeindex/openrouter.py` | embeddings, rerank, summarisation ("online mode") |
| `codeindex/chunking.py` | syntax-aware chunking; copes with a 593 KB single-class module |
| `codeindex/lexical.py` | code-aware sparse/BM25 tokens (snake_case, camelCase, dotted paths) |
| `codeindex/indexer.py` | blob → chunks → summaries → vectors → Qdrant, batched repo-wide |
| `codeindex/search.py` | hybrid fusion, cross-encoder rerank, clone-free file reads |
| `codeindex/cards.py` | per-repo routing cards, used as a routing prior |
| `codeindex/routing.py` | task → responsible repos, evidence, sparse-checkout plan |
| `codeindex/session.py` | overlays for the agent's uncommitted edits |
| `codeindex/extract.py` | tree-sitter extraction: symbols, imports, routes, env keys, calls, tasks |
| `codeindex/graph.py` | cross-repo coupling edges and context expansion |
| `codeindex/mcp_server.py` | the MCP tools |
| `codeindex/webui.py` | local inspection UI backend (Starlette, no new deps) |
| `codeindex/ui.html` | the UI itself; re-read per request, so edits are live |
| `codeindex/cli.py` | `setup`, `add-repo`, `scan`, `sync-manifests`, `verify-contents`, `index`, `search`, `verify-index`, `serve-*` |
| `sql/01_schema.sql` | manifests, graph slice, sessions |
| `sql/06_org_token.sql` | the token -> organisation mapping |
| `sql/07_control.sql` | repo credentials, webhook secrets, the job queue |
| `codeindex/control.py` | the control API: registration, webhooks, org lifecycle |
| `codeindex/pipeline.py` | one repo brought fully up to date; the job-queue drain |
| `codeindex/credentials.py` | resolving a private repo's clone token, without storing one |
| `sql/08_job_backoff.sql` | when a failed job becomes eligible again |

## Running it

```bash
cp .env.example .env          # then set OPENROUTER_API_KEY
docker compose up -d qdrant postgres
docker compose build indexer
```

Ports are deliberately non-default — **6335/6336 for Qdrant and 5433 for
Postgres** — because the Qdrant on 6333 holds live `person_reid` and face
collections and this is a tool whose own collections get dropped and rebuilt.

The vector dimension is fixed at collection creation and OpenRouter does not
document it per model, so it is measured:

```bash
python3 scripts/probe_embedding.py --rerank   # prints EMBED_DIM -> put it in .env
docker compose run --rm indexer python -m codeindex.cli setup --owner acme
```

`setup` applies the schema and creates one organisation's collections; run it
per org. It is safe to re-run: the base schema is created once and the
migrations after it are idempotent, which is how an existing database picks up
a newly added one.

Bootstrap from the checkouts already on this machine (no Bitbucket credentials
needed — `LOCAL_REPO_ROOT` is bind-mounted read-only at `/repos`):

```bash
docker compose run --rm indexer python -m codeindex.cli add-repo /repos/BACKEND
docker compose run --rm indexer python -m codeindex.cli sync-manifests
```

Check the ignore policy against real repos before spending anything on
embeddings — this needs no Docker, no Postgres and no API key:

```bash
python3 scripts/scan_report.py ~/Documents/Ikshana-V2/*/
```

## Organisations

Everything is scoped to an organisation, and **Qdrant collections are per
organisation**: `code_chunks__<org>` holds that org's code *and* its documents,
`repo_cards__<org>` its routing cards. `CODE_COLLECTION` is a name prefix, not a
collection name.

Per-org rather than per-repo, because a repo is not a unit anything is ever
dropped or rebuilt at, whereas an org is -- `drop_org` is two calls -- and
because it bounds the blast radius of a mistake. Every query still filters on
`owner_id` inside the collection, so a bug that opened the wrong collection
returns nothing rather than another customer's source.

A request names its organisation with a bearer token; the server no longer
infers one from its own config:

```bash
docker compose run --rm indexer python -m codeindex.cli token --mint "acme ci" --owner acme
docker compose run --rm indexer python -m codeindex.cli token            # list
docker compose run --rm indexer python -m codeindex.cli token --revoke vmcp_AbC123
```

Only the sha256 is stored, so the plaintext is shown once and a leaked database
row cannot be replayed. `REQUIRE_AUTH=false` serves an *unauthenticated* caller
as `DEFAULT_OWNER_ID` and exists so a local checkout works with no token; a
*wrong* token is refused either way.

## Registering repos from a product

`add-repo` takes a checkout on the indexer's own disk, which is fine for a
bootstrap and useless from a product. The control plane is the other way in:

| endpoint | what |
|---|---|
| `POST /v1/orgs/{org}/repos` | register (or update) a repo; creates the org's collections on first use and queues a sync |
| `GET /v1/orgs/{org}/repos` | what is registered, with commit, chunk count and job state |
| `DELETE /v1/orgs/{org}/repos/{slug}` | deregister, taking its vectors with it |
| `POST /v1/orgs/{org}/repos/{slug}/reindex` | queue a sync by hand |
| `POST /v1/orgs/{org}/tokens` | mint an MCP bearer token for the org |
| `DELETE /v1/orgs/{org}` | drop the organisation entirely |
| `POST /hooks/{org}/{slug}` | a push at the git host -> a queued sync |

It runs on **8081**, separate from the MCP endpoint on 8080, and takes a
different credential (`CONTROL_TOKEN`) because it can delete an organisation's
whole index. With `CONTROL_TOKEN` unset it refuses every request rather than
running open.

`/hooks/...` is the one public route, so it authenticates per repo on an HMAC of
the raw body against that repo's own secret -- checked before the body is parsed,
and answering 401 for an unknown repo so the endpoint cannot be used to
enumerate which repos exist. The secret is returned on every registration, not
just the first, so a caller that lost it can reconfigure the git host without
re-registering.

Queued work is coalesced: at most one outstanding job per repo, enforced by a
partial unique index. Ten pushes in a minute cause one index run against the
newest commit rather than ten against ten commits.

Credentials for a private repo are stored as a **reference** (`credential_ref`),
not a token. A token copied here would outlive its revocation and would have to
be re-copied on rotation, and there would then be two places it could leak from.

## Keeping it current

`serve-indexer` drains a queue. Three things fill it -- a push webhook, a manual
reindex, and a stale sweep -- and because they all go to the same queue, a push
and a schedule tick cannot race into two concurrent index runs of one repo.

Each job runs the whole sequence, in the order the data depends on it:

```
mirror -> fetch -> scan -> manifest -> index delta -> extract -> card
```

Every step is cheap when nothing changed, which is what makes it safe to run on
every push. A repo whose head has not moved and is already indexed costs one
`git fetch` and two queries; a repo one commit ahead re-embeds the one or two
files that actually changed, because chunk identity is the blob sha and the
embedding cache is keyed on content.

There is deliberately **no partial pipeline**. A half-synced repo -- manifest
current but vectors stale -- answers searches with confident wrong line numbers,
which is worse than being briefly out of date.

```bash
docker compose run --rm indexer python -m codeindex.cli sync --repo cloud-ikshana
docker compose run --rm indexer python -m codeindex.cli sync --force   # ignore the commit check
docker compose run --rm indexer python -m codeindex.cli jobs           # what is queued, running, failed
```

### Jobs, leases and retries

Jobs are claimed with `FOR UPDATE SKIP LOCKED`, so more than one indexer can run
without two of them embedding the same commit -- which costs money, not just
time. A claim takes a **lease**: an indexer that dies mid-run would otherwise
leave its job `running` forever and the repo would silently stop being indexed,
so an expired lease is reclaimable by anyone.

A failed job goes back to the queue with a **backoff** (1 min, 5 min, 15 min).
The backoff is the part that matters: without it the same drain loop re-claims
the job it just failed and spends all three attempts in the same millisecond,
which cannot outlast even a one-second outage -- and a one-second outage is the
only thing retrying is for. Past the attempt limit the job stays `error` so a
genuinely broken repo is visible rather than quietly burning quota.

The **stale sweep** queues any repo not fetched in the last hour. It is the
safety net under webhooks: a delivery that never arrived, or a host we could not
register a hook on, would otherwise leave a repo frozen at an old commit with
nothing to notice.

### Private repos

A private repo is registered with a `secret://` *reference*, and the token is
fetched back from Agent Studio at clone and fetch time
(`CREDENTIAL_CALLBACK_URL`). Storing the token here would fail in two ordinary
ways -- it outlives its own revocation, and it has to be re-sent on every
rotation -- and it would mean a compromise of this database yields credentials
rather than references.

The callback will only resolve a reference **already attached to a registered
repo in that tenant**, so it cannot be used to read the project's secret store
generally; a leaked callback token exposes the repo credentials the index was
given and nothing else.

## Using it from Claude Code

```bash
claude mcp add --transport http codeindex http://localhost:8080/mcp \
  --header "Authorization: Bearer $CODEINDEX_TOKEN"
```

The header is what tells the server which organisation you are; it can be
omitted only against a local stack running `REQUIRE_AUTH=false`.

Tools, in the order an agent should reach for them:

| tool | what it answers |
|---|---|
| `route_task` | which repos own this task, with evidence and a sparse-checkout plan |
| `search_code` | which chunks are relevant, across every repo at once |
| `get_file` | the contents of any file at any commit, without cloning |
| `expand_context` | what else a symbol touches — callers in *other* services, routes, config |
| `sync_workspace` | make your own uncommitted edits searchable |
| `session_status` / `end_session` | inspect and discard an overlay |
| `list_repos` | what is indexed |

## The UI

<http://localhost:8090> — three views over the same data the MCP tools return,
so what you see is what the agent sees.

| view | what it shows |
|---|---|
| **Index** | per-repo files / indexed / chunks / points, summary coverage, commit. Click a repo for its routing card, language mix and skip reasons |
| **Search** | the full hybrid + rerank pipeline, with each hit's summary and code |
| **Route** | ranked repos with the score broken into peak / depth / spread / prior / substance, the evidence, and the checkout plan |

The Index tab also shows the structural graph — extraction totals and the
cross-repo coupling table. Open session overlays appear below it, and entering a session id on
the Search tab includes that session's uncommitted edits — flagged
`uncommitted` in the results. Repo point counts are scoped to `main`, so an
overlay never inflates a repo's committed total.

Clicking any `path:line-line` opens the real file at that commit, read from git
objects with 25 lines of context either side and absolute line numbers — so a
reported range can be checked against the actual file rather than trusted.

Views are linkable: `?q=<query>` runs a search (add `&sess=<id>` to include a
session's edits), `?t=<task>` routes a task, `?tab=index|search|route` switches.

This is served by the stack rather than published anywhere, because it has to
reach Qdrant, Postgres and OpenRouter on localhost.

## Checks

```bash
docker compose run --rm indexer python -m codeindex.cli verify-contents  # reject blobs the bytes disqualify
docker compose run --rm indexer python -m codeindex.cli index            # safe to re-run; unchanged blobs hit the cache
docker compose run --rm indexer python -m codeindex.cli extract          # structural facts (~6s, pure function of the commit)
docker compose run --rm indexer python -m codeindex.cli edges            # derive cross-repo coupling
docker compose run --rm indexer python -m codeindex.cli verify-index     # reconcile manifest -> chunks -> points
docker compose run --rm indexer python scripts/eval_routing.py           # routing regression suite (9 hand-verified cases)
docker compose run --rm indexer python scripts/eval_session.py           # session overlay lifecycle (5 assertions)
docker compose run --rm indexer python scripts/eval_tenancy.py           # org isolation + token auth (31 assertions)
docker compose run --rm indexer python scripts/eval_control.py           # registration, webhook auth, repo state, deletion (41 assertions)
docker compose run --rm indexer python scripts/eval_pipeline.py          # sync pipeline, leases, retries (44 assertions)
```

`verify-index` exits non-zero if any repo is short of points, and reconciles
against `repo_commit.chunk_count` rather than the `blob_chunk` cache -- that
cache is content-addressed and shared across repos and commits, so it
accumulates and cannot serve as a baseline.

## The structural graph

Vector search answers "what looks like this". It cannot answer "what else breaks
if I change this", because that is a question about structure — and structure is
deterministic, so tree-sitter gives it up for free with no model and no
ambiguity. Extraction over all seven repos takes **~6 seconds**:

| | |
|---|---|
| symbols | 6,253 |
| calls | 52,911 |
| routes | 476 (Django `path()`/`re_path()` and FastAPI/Flask decorators) |
| imports | 4,260 |
| env keys | 336 |
| celery tasks | 25 |
| URL literals | 354 |

Calls are matched by **name**, not resolved to a definition. Resolution needs
type inference; a name match over 800 Python files is cheap, has no failure
modes, and answers the two questions that matter: who calls this, and which
other repo is coupled to it. `Model.objects.filter(...)` also records the root
of the dotted chain, so referencing a shared model counts as a usage rather than
being invisible behind `filter`.

### Coupling rules

Chosen for what is genuinely invisible to a reader — an import graph misses all
of these:

| kind | evidence |
|---|---|
| `http_call` | a route registered in one repo, hardcoded as a URL in another |
| `celery_task` | producer and consumer share only the task name |
| `env_key` | two services that must agree on a config key (skipped if >3 repos use it — that is a house convention, not a coupling) |
| `db_model` | the same top-level model class in two repos means a shared schema |



`route_task` reports coupling **filtered to the task**: an edge is only listed
if its evidence appears in the code the search actually surfaced. Every repo
here is coupled to the largest one somehow, so listing a repo's couplings
wholesale says nothing about the task at hand. Counts read `2 here of 9` —
matched versus total — rather than overstating.

## Session overlays

The server indexes commits, which is everything except the file the agent is
currently editing. So the agent sends its dirty files to `sync_workspace` and
they land in the same collection under `scope=session:<id>`:

```
sync_workspace(session_id="fix-alerts", repo="cloud-ikshana",
               files={"path.py": "<contents>"}, deleted=["gone.py"])
search_code(query="...", session_id="fix-alerts")   # sees the edits
end_session("fix-alerts")
```

Shadowing happens **in the Qdrant filter**, not after retrieval: paths the
session has touched are excluded at `main` scope. Post-filtering cannot do this
— a *deleted* file has no session points to shadow with, so it would keep
returning its committed version forever.

Cost is proportional to what the agent touched. The other 5,400 chunks are
already embedded and stay that way, which is the entire point: nothing an agent
does can corrupt or re-cost the committed index. Overlays are dropped by
`end_session` and swept after 24 idle hours by the indexer loop, so a crashed
agent leaks nothing permanent.

## Routing

`route_task` scores each repo on four signals and returns the ones that survive
a relative floor:

| signal | weight | why |
|---|---|---|
| peak | 0.48 | best chunk after cross-encoder rerank |
| depth | 0.28 | mean of the top 5, so one lucky match is not enough |
| spread | 0.10 | distinct files, saturating -- 35 files is not 4x the ownership of 8 |
| prior | 0.14 | task vs the repo's card, for vocabulary matches with no standout chunk |

Reranking happens **before** grouping, because RRF fusion scores are rank-based
and not comparable across repos. The candidate pool is capped per repo, since
one repo holds 78% of all chunks and would otherwise crowd the alternatives out
of the reranker entirely. A final `substance` factor demotes repos whose
evidence is all fixtures and docs -- as a threshold, not a ratio, so a small
repo whose README genuinely explains its behaviour is not punished for that.

## Retrieval design

Small corpus (~26k chunks), so quality wins over every efficiency trade:

- **no quantization**, and `exact=True` while under 250k points — brute force
  with perfect recall instead of an HNSW approximation;
- **two dense vectors per point**: the code itself, and an LLM-written
  description of it. Task text is natural language and matches the latter far
  better than it matches Django internals;
- **a sparse `lex` vector** with server-side IDF, because tasks name exact
  identifiers that dense vectors miss;
- **cross-encoder rerank** of the fused top ~150 down to ~20. Largest single
  precision gain in the pipeline.

## Status

Phases 0-4 done and verified end to end. **5,467 chunks across 7 repos**, 100%
with an LLM summary, `verify-index` clean, routing **9/9** at rank 1, session
overlays **5/5**, and a structural graph of 6,253 symbols / 52,911 calls / 476
routes with 90 cross-repo coupling edges. Measured: hybrid fusion ~900 ms
(including the query embedding round trip), cross-encoder rerank ~1.8 s,
extraction ~6 s for everything.

Multi-org is in: collections are per organisation, and every MCP tool resolves
its org from the caller's token instead of server config -- **31/31** in
`eval_tenancy`, covering collection isolation, token lifecycle, and the fact
that a grant on one org's repo cannot widen what another org sees. The tool
schemas are unchanged, so existing clients see the same surface.

Registration is in too: Agent Studio registers a repo over the control API,
which creates the org's collections, records the credential *reference*, and
queues a sync -- **41/41** in `eval_control`, and verified end to end from the
Agent Studio API through to Qdrant.

And the queue now drains: `serve-indexer` runs the whole pipeline per job, with
leases, backoff and a stale sweep -- **44/44** in `eval_pipeline`, run against a
real git repository with the embedding API stubbed, so the incremental path is
checked rather than assumed. Private repos resolve their clone token back
through Agent Studio, verified end to end across both services.

Next: documents alongside code, under `kind=doc` in the same per-org
collection, and the workflow node that turns a task into the repos that own it.

### Credential material is excluded, and there is some to deal with

Indexing a secret copies it into a vector store and then into agent context on
every loosely-related search, so credential material is rejected on the path
where possible (`serviceAccountKey.json`, `.env`, `*.pem`, `*.key`) and on the
bytes where not. Source files are handled at **chunk** granularity instead --
dropping all of `settings.py` or 25 chunks of `backup_manager.py` would cost
real code, so the file is indexed and only the offending chunk is held back.
Verified: 0 of 5,467 points contain credential markers.

What that turned up in the repos themselves is listed under Known gaps.