# Local Docs Retriever

Local documentation retrieval for coding agents, with sparse, dense, and hybrid search through MCP.

A coding agent needs documentation for the framework and version it is working with. Similar API
names across ecosystems and changing documentation make that context difficult to retrieve reliably.
This project indexes local Markdown and TypeScript JSDoc, searches an explicitly versioned corpus,
and returns fragments with source paths, sections, and commits that a caller can inspect.

It implements the retrieval layer of a retrieval-augmented generation (RAG) system. A consuming
agent supplies the LLM and answer generation. After dependencies, models, and source checkouts are
provisioned, retrieval can run offline.

**Stack:** Python · Haystack · FastEmbed · Qdrant · MCP · pytest

[Architecture](docs/ARCHITECTURE.md) · [Evaluation](docs/EVALUATION.md) · [Security](docs/SECURITY.md) ·
[Guía en español](docs/DOCUMENTACION.md)

## AI engineering demonstrated

| Area | Implementation | Engineering purpose |
| --- | --- | --- |
| Corpus preparation | Manifest-driven Markdown and TypeScript JSDoc ingestion; heading-aware chunking with overlap | Preserve documentation structure and restrict which files enter the index |
| Embeddings | Local dense embeddings with BGE and learned sparse embeddings with SPLADE through FastEmbed | Support semantic similarity and technical-term matching |
| Retrieval strategies | Sparse, dense, and hybrid retrieval with Qdrant reciprocal-rank fusion | Compare retrieval approaches through one application interface |
| Context selection | Exact framework and version filters applied before ranking | Narrow retrieval to the requested ecosystem and release |
| Evaluation | Hit@1/3/5, MRR, nDCG@10, section hit rate, and p50/p95 latency | Measure ranking against expected sources and sections |
| Agent integration | Three MCP tools backed by the same engine as the CLI | Let agents retrieve documentation using a structured tool contract |
| Corpus lifecycle | Deterministic chunk IDs, overwrite-by-ID, and atomic alias activation | Re-index revisions consistently and activate a prepared corpus explicitly |

## How it works

```mermaid
flowchart LR
    A[Local Markdown / TypeScript JSDoc] --> B[Extract and chunk documentation]
    B --> C[Local dense and sparse embeddings]
    C --> D[(Qdrant corpus)]
    E[CLI or MCP query] --> F[Embed query and apply metadata filters]
    F --> G[Sparse / dense / hybrid retrieval]
    D --> G
    G --> H[Fragments with source and revision metadata]
    H --> I[Consuming coding agent]
```

Document embeddings use context-enriched text while stored results retain the documentation body.
Haystack and Qdrant integration details stay in adapters; the engine returns application-owned
`RetrievedChunk` values to both transports. See [ARCHITECTURE.md](docs/ARCHITECTURE.md) for the boundaries.

## Design decisions and limits

- **Sparse by default:** the configured default uses learned sparse retrieval. Dense and hybrid
  remain selectable through the CLI and evaluation runner; MCP uses the configured strategy.
- **Explicit provenance:** framework, version, path, heading, and commit travel with retrieved
  fragments. Version metadata comes from the source manifest rather than automatic inference.
- **Separate retrieval from generation:** this service does not synthesize answers, train or
  fine-tune models, or use a reranker. It does not inspect the consuming agent's code repository.
- **Evaluate what is implemented:** relevance is scored by expected source paths and optional
  headings. There is no semantic answer evaluation or automatic detection of unsupported LLM claims.
- **Local operation with prerequisites:** offline mode requires cached models and prepared sources.
  The service has no application authentication and is intended for a trusted local environment.

## Evidence and evaluation status

The repository includes a ten-query [smoke dataset](data/evaluation/smoke.json), a retrieval
[evaluation runner](src/local_docs_rag/evaluation/runner.py), unit tests, and an opt-in Qdrant
integration test. Tests cover chunking, metadata filters, deterministic document identity, metrics,
source discovery, JSDoc extraction, engine behavior, and MCP tool contracts.

The smoke dataset is a starting point, not a comprehensive benchmark. This repository does not
include a published benchmark report with strategy scores and hardware/corpus details. To assess
retrieval quality, verify the expected paths against the indexed revisions and run each strategy on
the same corpus and queries. The runner includes first-use initialization in its latency readings;
see [EVALUATION.md](docs/EVALUATION.md) for interpretation and warm-up guidance.

For a small end-to-end demonstration, follow the bundled fixture commands under
[Development](#development) after provisioning Qdrant and the models. These exercise ingestion and
all three retrieval strategies without requiring a full framework corpus.

## Run locally

The Python distribution remains `local-docs-rag`, the import package is `local_docs_rag`, and the
CLI command is `rag`.

## Requirements

- Python 3.11 or newer (tested dependency set includes Python 3.14)
- Qdrant at `http://127.0.0.1:6333`
- Git checkouts of the official documentation sources
- The dense model `BAAI/bge-small-en-v1.5` (384 dimensions)
- The sparse model `prithivida/Splade_PP_en_v1`

## Install

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'
cp .env.example .env
set -a
. ./.env
set +a
```

The application does not automatically read `.env`; source it as above or export its values with your
service manager. Haystack and Hugging Face telemetry are disabled in package initialization before
Haystack is imported. The example permits initial model provisioning; enable its offline flags after
the models are cached.

## Prepare a source

Clone or otherwise prepare repositories while network access is available. In production, ingestion
operates on an existing local checkout and records its exact `HEAD` commit.

Create one YAML or JSON manifest per source. See `examples/sources/angular.yaml`. The same manifest
schema supports Angular, TypeScript, Node.js, and RxJS:

```yaml
framework: angular
framework_version: "20.0"
repository: https://github.com/angular/angular.git
ref: 20.0.0
content_roots: [adev/src/content]
file_globs: ["**/*.md"]
source_type: markdown
```

The version is explicit metadata. The service never infers or guesses it.

### TypeScript JSDoc sources

Use `source_type: typescript_jsdoc` for API documentation embedded in TypeScript source, such as RxJS
operators. Discovery remains restricted to the manifest roots and globs:

```yaml
framework: rxjs
framework_version: "8"
repository: https://github.com/ReactiveX/rxjs.git
ref: master
content_roots:
  - packages/rxjs/src/internal/operators
file_globs:
  - "**/*.ts"
source_type: typescript_jsdoc
```

See `examples/sources/rxjs-jsdoc.yaml`. This adapter extracts only `/** ... */` documentation. It
removes comment delimiters and leading `*` formatting, preserves Markdown and JSDoc tags such as
`@example`, `@param`, `@returns`, `@deprecated`, and `@see`, and associates blocks with an immediately
following exported function, class, interface, type, constant, or operator when possible. Function
bodies and other implementation code are never added to indexed content. The extracted documentation
then passes through the same heading-aware chunker and deterministic identity rules as Markdown.

## Build and activate a corpus

Use a concrete collection for a build. Multiple sources can be added to the same collection before
the alias is activated:

```bash
rag ingest examples/sources/angular.yaml \
  --checkout /srv/docs/angular \
  --collection docs_2026_09_29

rag ingest /srv/manifests/typescript.yaml \
  --checkout /srv/docs/typescript \
  --collection docs_2026_09_29 \
  --activate
```

`--activate` atomically points `RAG_COLLECTION_ALIAS` (default `docs_current`) to the concrete
collection. Ingestion never uses `recreate_index=True`; deterministic IDs and overwrite-by-ID make
re-indexing the same revision idempotent. Search and evaluation refuse to create a missing index.

For a non-Git fixture, pass an already verified full commit:

```bash
rag ingest manifest.yaml --checkout ./fixture --commit 0123456789abcdef0123456789abcdef01234567 --activate
```

## Search

```bash
rag search "How do computed signals track dependencies?" --framework angular --version 20.0
rag search "How are CommonJS modules resolved?" --framework nodejs --strategy hybrid --limit 5
rag health
rag sources
```

Filters are applied by Qdrant before ranking. Omitting a framework or version omits only that filter.

## MCP

Start streamable HTTP on `127.0.0.1:8000/mcp`:

```bash
rag serve --transport streamable-http
```

Or use stdio:

```bash
rag serve --transport stdio
```

The server exposes only `docs.search`, `docs.health`, and `docs.sources`. The public `docs.search`
schema accepts `query`, `framework`, `version`, and `limit`; it does not expose retrieval strategy.
MCP searches use the configured `RAG_RETRIEVAL_STRATEGY`, which defaults to the production `sparse`
strategy. Dense and hybrid remain available through the CLI and evaluation commands for diagnostics
and benchmarking. For a remote coding agent, keep the service bound to Ubuntu loopback and expose it
through an SSH local-forward configured by the operator. Do not bind publicly without adding
authentication and transport security.

## Evaluate an existing index

```bash
rag eval data/evaluation/smoke.json --strategies sparse,dense,hybrid
```

Evaluation never ingests, creates, recreates, or activates a collection. Update the smoke dataset's
oracle paths to match the exact official revisions represented by your manifests.

## Offline mode

After dependencies, source repositories, and both model caches are prepared, export:

```bash
export HAYSTACK_TELEMETRY_ENABLED=False
export HF_HUB_DISABLE_TELEMETRY=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export RAG_QDRANT_URL=http://127.0.0.1:6333
export RAG_COLLECTION_ALIAS=docs_current
export RAG_RETRIEVAL_STRATEGY=sparse
```

With `HF_HUB_OFFLINE=1`, FastEmbed is initialized with `local_files_only=True` and fails clearly if a
model is missing. Perform one controlled online ingestion/search during provisioning to populate the
cache, or copy a prepared `FASTEMBED_CACHE_PATH` onto the host.

## Development

```bash
ruff format --check src tests
ruff check src tests
pytest -m 'not network'
RAG_RUN_QDRANT_TESTS=1 pytest tests/integration/test_qdrant.py
```

Network-marked tests are skipped by default. The integration test creates a uniquely named temporary
collection in local Qdrant and deletes only that collection afterward.

To validate real model loading, ingestion, sparse filtering, dense retrieval, and hybrid fusion with
the tiny bundled fixture while leaving `docs_current` untouched:

```bash
export RAG_COLLECTION_ALIAS=docs_smoke_current
rag ingest tests/fixtures/source_manifest.yaml \
  --checkout tests/fixtures/source_repo \
  --commit 0123456789abcdef0123456789abcdef01234567 \
  --collection docs_smoke_fixture_v1 \
  --activate
rag search "reactive fixture documentation" --framework fixture --version 1.0 --strategy sparse
rag search "reactive fixture documentation" --framework fixture --version 1.0 --strategy dense
rag search "reactive fixture documentation" --framework fixture --version 1.0 --strategy hybrid
```

See [DOCUMENTACION.md](docs/DOCUMENTACION.md) for the Spanish project and MCP API guide. See
[SPEC.md](docs/SPEC.md), [ARCHITECTURE.md](docs/ARCHITECTURE.md), [SECURITY.md](docs/SECURITY.md), and
[EVALUATION.md](docs/EVALUATION.md) for the operating contract and design rationale.

## License

This project's code is available under the [MIT License](LICENSE), which permits commercial use,
modification, and redistribution while preserving the copyright and license notice.
Third-party documentation and models remain subject to their respective licenses.