VectorMCP — central cross-repo code index over MCP

Gives Claude Code one central, always-warm index over every Ikshana repo. State a task, get back the repos responsible for it, then search and read their code without cloning anything.

The constraint this is built around

Claude Code clones a fresh copy of a repo for each task and throws it away. So index identity is git object identity, never filesystem location: a chunk's Qdrant point id is uuid5(NS, "{repo_id}:{blob_sha}:{chunk_idx}"). Blob shas are identical in every clone, forever, which means:

• a fresh clone at any commit needs zero re-indexing;

• indexing runs server-side against bare mirrors, never in the agent's checkout;

• the agent only ever contributes deltas for its uncommitted edits, into a disposable scope=session:<id> overlay.

Measured on this codebase: a single commit changes 0.5–0.8% of indexable files, so an incremental pass re-embeds one or two files rather than 927.

Related MCP server: PAMPA

Layout

Running it

cp .env.example .env          # then set OPENROUTER_API_KEY
docker compose up -d qdrant postgres
docker compose build indexer

Ports are deliberately non-default — 6335/6336 for Qdrant and 5433 for Postgres — because the Qdrant on 6333 holds live person_reid and face collections and this is a tool whose own collections get dropped and rebuilt.

The vector dimension is fixed at collection creation and OpenRouter does not document it per model, so it is measured:

python3 scripts/probe_embedding.py --rerank   # prints EMBED_DIM -> put it in .env
docker compose run --rm indexer python -m codeindex.cli setup --owner acme

setup applies the schema and creates one organisation's collections; run it per org. It is safe to re-run: the base schema is created once and the migrations after it are idempotent, which is how an existing database picks up a newly added one.

Bootstrap from the checkouts already on this machine (no Bitbucket credentials needed — LOCAL_REPO_ROOT is bind-mounted read-only at /repos):

docker compose run --rm indexer python -m codeindex.cli add-repo /repos/BACKEND
docker compose run --rm indexer python -m codeindex.cli sync-manifests

Check the ignore policy against real repos before spending anything on embeddings — this needs no Docker, no Postgres and no API key:

python3 scripts/scan_report.py ~/Documents/Ikshana-V2/*/

Organisations

Everything is scoped to an organisation, and Qdrant collections are per organisation: code_chunks__<org> holds that org's code and its documents, repo_cards__<org> its routing cards. CODE_COLLECTION is a name prefix, not a collection name.

Per-org rather than per-repo, because a repo is not a unit anything is ever dropped or rebuilt at, whereas an org is -- drop_org is two calls -- and because it bounds the blast radius of a mistake. Every query still filters on owner_id inside the collection, so a bug that opened the wrong collection returns nothing rather than another customer's source.

A request names its organisation with a bearer token; the server no longer infers one from its own config:

docker compose run --rm indexer python -m codeindex.cli token --mint "acme ci" --owner acme
docker compose run --rm indexer python -m codeindex.cli token            # list
docker compose run --rm indexer python -m codeindex.cli token --revoke vmcp_AbC123

Only the sha256 is stored, so the plaintext is shown once and a leaked database row cannot be replayed. REQUIRE_AUTH=false serves an unauthenticated caller as DEFAULT_OWNER_ID and exists so a local checkout works with no token; a wrong token is refused either way.

Registering repos from a product

add-repo takes a checkout on the indexer's own disk, which is fine for a bootstrap and useless from a product. The control plane is the other way in:

It runs on 8081, separate from the MCP endpoint on 8080, and takes a different credential (CONTROL_TOKEN) because it can delete an organisation's whole index. With CONTROL_TOKEN unset it refuses every request rather than running open.

/hooks/... is the one public route, so it authenticates per repo on an HMAC of the raw body against that repo's own secret -- checked before the body is parsed, and answering 401 for an unknown repo so the endpoint cannot be used to enumerate which repos exist. The secret is returned on every registration, not just the first, so a caller that lost it can reconfigure the git host without re-registering.

Queued work is coalesced: at most one outstanding job per repo, enforced by a partial unique index. Ten pushes in a minute cause one index run against the newest commit rather than ten against ten commits.

Credentials for a private repo are stored as a reference (credential_ref), not a token. A token copied here would outlive its revocation and would have to be re-copied on rotation, and there would then be two places it could leak from.

Keeping it current

serve-indexer drains a queue. Three things fill it -- a push webhook, a manual reindex, and a stale sweep -- and because they all go to the same queue, a push and a schedule tick cannot race into two concurrent index runs of one repo.

Each job runs the whole sequence, in the order the data depends on it:

mirror -> fetch -> scan -> manifest -> index delta -> extract -> card

Every step is cheap when nothing changed, which is what makes it safe to run on every push. A repo whose head has not moved and is already indexed costs one git fetch and two queries; a repo one commit ahead re-embeds the one or two files that actually changed, because chunk identity is the blob sha and the embedding cache is keyed on content.

There is deliberately no partial pipeline. A half-synced repo -- manifest current but vectors stale -- answers searches with confident wrong line numbers, which is worse than being briefly out of date.

docker compose run --rm indexer python -m codeindex.cli sync --repo cloud-ikshana
docker compose run --rm indexer python -m codeindex.cli sync --force   # ignore the commit check
docker compose run --rm indexer python -m codeindex.cli jobs           # what is queued, running, failed

Jobs, leases and retries

Jobs are claimed with FOR UPDATE SKIP LOCKED, so more than one indexer can run without two of them embedding the same commit -- which costs money, not just time. A claim takes a lease: an indexer that dies mid-run would otherwise leave its job running forever and the repo would silently stop being indexed, so an expired lease is reclaimable by anyone.

A failed job goes back to the queue with a backoff (1 min, 5 min, 15 min). The backoff is the part that matters: without it the same drain loop re-claims the job it just failed and spends all three attempts in the same millisecond, which cannot outlast even a one-second outage -- and a one-second outage is the only thing retrying is for. Past the attempt limit the job stays error so a genuinely broken repo is visible rather than quietly burning quota.

The stale sweep queues any repo not fetched in the last hour. It is the safety net under webhooks: a delivery that never arrived, or a host we could not register a hook on, would otherwise leave a repo frozen at an old commit with nothing to notice.

Private repos

A private repo is registered with a secret:// reference, and the token is fetched back from Agent Studio at clone and fetch time (CREDENTIAL_CALLBACK_URL). Storing the token here would fail in two ordinary ways -- it outlives its own revocation, and it has to be re-sent on every rotation -- and it would mean a compromise of this database yields credentials rather than references.

The callback will only resolve a reference already attached to a registered repo in that tenant, so it cannot be used to read the project's secret store generally; a leaked callback token exposes the repo credentials the index was given and nothing else.

Using it from Claude Code

claude mcp add --transport http codeindex http://localhost:8080/mcp \
  --header "Authorization: Bearer $CODEINDEX_TOKEN"

The header is what tells the server which organisation you are; it can be omitted only against a local stack running REQUIRE_AUTH=false.

Tools, in the order an agent should reach for them:

The UI

http://localhost:8090 — three views over the same data the MCP tools return, so what you see is what the agent sees.

The Index tab also shows the structural graph — extraction totals and the cross-repo coupling table. Open session overlays appear below it, and entering a session id on the Search tab includes that session's uncommitted edits — flagged uncommitted in the results. Repo point counts are scoped to main, so an overlay never inflates a repo's committed total.

Clicking any path:line-line opens the real file at that commit, read from git objects with 25 lines of context either side and absolute line numbers — so a reported range can be checked against the actual file rather than trusted.

Views are linkable: ?q=<query> runs a search (add &sess=<id> to include a session's edits), ?t=<task> routes a task, ?tab=index|search|route switches.

This is served by the stack rather than published anywhere, because it has to reach Qdrant, Postgres and OpenRouter on localhost.

Checks

docker compose run --rm indexer python -m codeindex.cli verify-contents  # reject blobs the bytes disqualify
docker compose run --rm indexer python -m codeindex.cli index            # safe to re-run; unchanged blobs hit the cache
docker compose run --rm indexer python -m codeindex.cli extract          # structural facts (~6s, pure function of the commit)
docker compose run --rm indexer python -m codeindex.cli edges            # derive cross-repo coupling
docker compose run --rm indexer python -m codeindex.cli verify-index     # reconcile manifest -> chunks -> points
docker compose run --rm indexer python scripts/eval_routing.py           # routing regression suite (9 hand-verified cases)
docker compose run --rm indexer python scripts/eval_session.py           # session overlay lifecycle (5 assertions)
docker compose run --rm indexer python scripts/eval_tenancy.py           # org isolation + token auth (31 assertions)
docker compose run --rm indexer python scripts/eval_control.py           # registration, webhook auth, repo state, deletion (41 assertions)
docker compose run --rm indexer python scripts/eval_pipeline.py          # sync pipeline, leases, retries (44 assertions)

verify-index exits non-zero if any repo is short of points, and reconciles against repo_commit.chunk_count rather than the blob_chunk cache -- that cache is content-addressed and shared across repos and commits, so it accumulates and cannot serve as a baseline.

The structural graph

Vector search answers "what looks like this". It cannot answer "what else breaks if I change this", because that is a question about structure — and structure is deterministic, so tree-sitter gives it up for free with no model and no ambiguity. Extraction over all seven repos takes ~6 seconds:

Calls are matched by name, not resolved to a definition. Resolution needs type inference; a name match over 800 Python files is cheap, has no failure modes, and answers the two questions that matter: who calls this, and which other repo is coupled to it. Model.objects.filter(...) also records the root of the dotted chain, so referencing a shared model counts as a usage rather than being invisible behind filter.

Coupling rules

Chosen for what is genuinely invisible to a reader — an import graph misses all of these:

route_task reports coupling filtered to the task: an edge is only listed if its evidence appears in the code the search actually surfaced. Every repo here is coupled to the largest one somehow, so listing a repo's couplings wholesale says nothing about the task at hand. Counts read 2 here of 9 — matched versus total — rather than overstating.

Session overlays

The server indexes commits, which is everything except the file the agent is currently editing. So the agent sends its dirty files to sync_workspace and they land in the same collection under scope=session:<id>:

sync_workspace(session_id="fix-alerts", repo="cloud-ikshana",
               files={"path.py": "<contents>"}, deleted=["gone.py"])
search_code(query="...", session_id="fix-alerts")   # sees the edits
end_session("fix-alerts")

Shadowing happens in the Qdrant filter, not after retrieval: paths the session has touched are excluded at main scope. Post-filtering cannot do this — a deleted file has no session points to shadow with, so it would keep returning its committed version forever.

Cost is proportional to what the agent touched. The other 5,400 chunks are already embedded and stay that way, which is the entire point: nothing an agent does can corrupt or re-cost the committed index. Overlays are dropped by end_session and swept after 24 idle hours by the indexer loop, so a crashed agent leaks nothing permanent.

Routing

route_task scores each repo on four signals and returns the ones that survive a relative floor:

Reranking happens before grouping, because RRF fusion scores are rank-based and not comparable across repos. The candidate pool is capped per repo, since one repo holds 78% of all chunks and would otherwise crowd the alternatives out of the reranker entirely. A final substance factor demotes repos whose evidence is all fixtures and docs -- as a threshold, not a ratio, so a small repo whose README genuinely explains its behaviour is not punished for that.

Retrieval design

Small corpus (~26k chunks), so quality wins over every efficiency trade:

• no quantization, and exact=True while under 250k points — brute force with perfect recall instead of an HNSW approximation;

• two dense vectors per point: the code itself, and an LLM-written description of it. Task text is natural language and matches the latter far better than it matches Django internals;

• a sparse lex vector with server-side IDF, because tasks name exact identifiers that dense vectors miss;

• cross-encoder rerank of the fused top ~150 down to ~20. Largest single precision gain in the pipeline.

Status

Phases 0-4 done and verified end to end. 5,467 chunks across 7 repos, 100% with an LLM summary, verify-index clean, routing 9/9 at rank 1, session overlays 5/5, and a structural graph of 6,253 symbols / 52,911 calls / 476 routes with 90 cross-repo coupling edges. Measured: hybrid fusion ~900 ms (including the query embedding round trip), cross-encoder rerank ~1.8 s, extraction ~6 s for everything.

Multi-org is in: collections are per organisation, and every MCP tool resolves its org from the caller's token instead of server config -- 31/31 in eval_tenancy, covering collection isolation, token lifecycle, and the fact that a grant on one org's repo cannot widen what another org sees. The tool schemas are unchanged, so existing clients see the same surface.

Registration is in too: Agent Studio registers a repo over the control API, which creates the org's collections, records the credential reference, and queues a sync -- 41/41 in eval_control, and verified end to end from the Agent Studio API through to Qdrant.

And the queue now drains: serve-indexer runs the whole pipeline per job, with leases, backoff and a stale sweep -- 44/44 in eval_pipeline, run against a real git repository with the embedding API stubbed, so the incremental path is checked rather than assumed. Private repos resolve their clone token back through Agent Studio, verified end to end across both services.

Next: documents alongside code, under kind=doc in the same per-org collection, and the workflow node that turns a task into the repos that own it.

Credential material is excluded, and there is some to deal with

Indexing a secret copies it into a vector store and then into agent context on every loosely-related search, so credential material is rejected on the path where possible (serviceAccountKey.json, .env, *.pem, *.key) and on the bytes where not. Source files are handled at chunk granularity instead -- dropping all of settings.py or 25 chunks of backup_manager.py would cost real code, so the file is indexed and only the offending chunk is held back. Verified: 0 of 5,467 points contain credential markers.

What that turned up in the repos themselves is listed under Known gaps.