# Citable

[![CI](https://github.com/Pranava77/citable/actions/workflows/ci.yml/badge.svg)](https://github.com/Pranava77/citable/actions/workflows/ci.yml)
![Python 3.12](https://img.shields.io/badge/python-3.12-blue)
![MCP](https://img.shields.io/badge/MCP-server-black)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Cited answers on Indian criminal law for AI assistants.** Citable is an MCP
server that lets Claude (or Cursor, or any MCP client) search the official text of
India's three new criminal laws and answer with the exact Act, section and Gazette
page for every claim, instead of answering from memory.

It is built as a measured system: a hand-labelled eval set gates search quality in
CI, and every tool call is logged for latency and cost.

## Why

On 1 July 2024 India replaced its three core criminal laws, and every section
number changed:

| New law | Replaced | Sections |
|---|---|---|
| Bharatiya Nyaya Sanhita, 2023 (**BNS**) — offences and punishments | Indian Penal Code, 1860 | 358 |
| Bharatiya Nagarik Suraksha Sanhita, 2023 (**BNSS**) — arrest, bail, FIR, trial | Code of Criminal Procedure, 1973 | 531 |
| Bharatiya Sakshya Adhiniyam, 2023 (**BSA**) — evidence | Indian Evidence Act, 1872 | 170 |

Cheating was IPC s. 420; it is now BNS s. 318. Language models learned mostly from
the old codes, so they quote repealed sections or invent provisions — and in law a
wrong section number is worse than no answer. Citable gives the assistant the
current text to quote, with a citation the reader can check.

## What it looks like

Ask Claude *"Someone on a motorbike snatched my gold chain. What offence is that?"*
and it calls Citable's `search_law` tool. The same search from the command line:

```console
$ uv run citable search "someone on a motorbike snatched my gold chain" -k 3 --rerank
1. BNS s. 304, p. 80 — Snatching.  [keyword #1, vector #3]
   304. (1) Theft is snatching if, in order to commit theft, the offender suddenly or
   quickly or forcibly seizes or secures or grabs or takes away from any person or from his
   possession any movable property. (2) Whoever ...
2. BNS s. 112, p. 36 — Petty organised crime.  [keyword #2, vector #11]
3. BNS s. 309, pp. 81-82 — Robbery.  [vector #2]

(keyword 2 ms, vector 9 ms, rerank 247 ms, total 258 ms)
```

Claude then reads the full section with `get_section` and answers citing
"BNS s. 304", with a link straight to page 80 of the Gazette PDF.

## At a glance

| | |
|---|---|
| Corpus | 1,059 sections, 1,476 passages, parsed from the Gazette of India PDFs |
| Search quality | **80% recall@5** on 85 hand-labelled questions; 100% on citations and statutory wording |
| Latency | 234 ms median, 272 ms p95 per search, on a laptop CPU |
| Cost | ≈1,300 tokens of context per search: $1.90 per 1,000 calls on Claude Sonnet 5.5 |
| Runs on | one SQLite file and two small local models: no servers, no API keys, offline after setup |

## How it works

```mermaid
flowchart TD
    subgraph build ["Build once"]
        G["Gazette of India PDFs"] -->|"parse the page layout"| S["1,059 sections"]
        S -->|"chunk + embed"| DB[("SQLite<br/>FTS5 + sqlite-vec")]
    end
    subgraph ask ["Every question"]
        A["Claude Desktop / Cursor"] <-->|"MCP"| M["search_law · get_section · list_contents"]
        M --> H["citation lookup + BM25 + vector"]
        H --> F["fusion (RRF) + rerank"]
    end
    DB --> H
    M --> T[("telemetry log")] --> D["cost & latency dashboard"]
    E["85 labelled questions"] --> CI["CI quality gate"]
```

1. **Ingestion** ([`parse.py`](src/citable/parse.py)). The Gazette prints each
   section's heading as a note in the page margin, level with the section's first
   line. The parser reads the PDF line by line with coordinates, pairs margin notes
   with sections, and tracks parts, chapters and sub-headings. Section numbers must
   follow on, which catches stray lines that look like sections.
2. **Search** ([`search.py`](src/citable/search.py)), in one SQLite file:
   - **Citation lookup**: "BNS 318" or "section 482 of BNSS" goes straight to the section.
   - **Keyword**: FTS5 BM25, with section headings weighted 4x over body text.
   - **Vector**: `BAAI/bge-small-en-v1.5` embeddings in `sqlite-vec`, run locally with fastembed.
   - **Fusion**: Reciprocal Rank Fusion of the two rankings, per section.
   - **Rerank**: an `ms-marco-MiniLM-L-6-v2` cross-encoder over the top 20.
3. **MCP server** ([`server.py`](src/citable/server.py)): three read-only tools,
   plus instructions telling the assistant to answer from the returned text and cite
   every provision.
4. **Evaluation** ([`evaluate.py`](src/citable/evaluate.py)): recall@k and MRR for
   every search configuration, checked in CI.
5. **Telemetry and dashboard** ([`telemetry.py`](src/citable/telemetry.py),
   [`dashboard.py`](src/citable/dashboard.py)): every tool call logged; one HTML
   page of latency and cost.

## Quick start

Requires [uv](https://docs.astral.sh/uv/). The parsed sections are in the repo, so
you don't need to download the PDFs.

```bash
git clone https://github.com/Pranava77/citable.git
cd citable
uv sync
uv run citable index                       # embed and build data/citable.db (~1 min; downloads ~150 MB of models once)
uv run citable search "can the police arrest me without a warrant" --rerank
uv run citable show bns 318                # print one section
```

### Connect it to Claude Desktop

Add this to `~/Library/Application Support/Claude/claude_desktop_config.json`
(macOS) and restart Claude Desktop. Use the full path to `uv` (`which uv`), because
desktop apps don't see your shell's `PATH`:

```json
{
  "mcpServers": {
    "citable": {
      "command": "/Users/you/.local/bin/uv",
      "args": ["--directory", "/path/to/citable", "run", "citable", "serve"]
    }
  }
}
```

The tools:

| Tool | Returns |
|---|---|
| `search_law(query, act?, limit?)` | the best-matching sections, each with citation, chapter, the passage that matched, and a Gazette link |
| `get_section(act, section)` | the full text of one section, with its chapter, sub-heading and the Acts it cites |
| `list_contents(act, chapter?)` | an Act's chapters, or the section headings in one chapter |

The server reranks by default; append `"--no-rerank"` to `args` to trade about 8
points of recall for about 240 ms per search.

## Evaluation

[`evals/questions.jsonl`](evals/questions.jsonl) holds 85 questions, each labelled
with the sections that answer it. Every label was checked against the parsed text.

| Category | n | Example | Answer |
|---|---|---|---|
| citation | 6 | "Section 63 of the Bharatiya Sakshya Adhiniyam" | BSA s. 63 |
| statutory | 20 | "maximum period for which an undertrial prisoner can be detained" | BNSS s. 479 |
| plain | 32 | "Someone on a motorbike grabbed my gold chain and rode away" | BNS s. 304 |
| jargon | 21 | "Punishment for mob lynching", words the Act never uses | BNS s. 103(2) |
| old_number | 6 | "Section 438 CrPC", a repealed code's numbering | BNSS s. 482 |

`uv run citable eval` (recall@k = share of questions with a correct section in the top k):

| Search | R@1 | R@3 | R@5 | R@10 | MRR | p50 / p95 |
|---|---|---|---|---|---|---|
| keyword (BM25) | 0.42 | 0.56 | 0.60 | 0.66 | 0.50 | 1 / 1 ms |
| vector | 0.54 | 0.67 | 0.71 | 0.79 | 0.62 | 6 / 7 ms |
| hybrid (RRF) | 0.48 | 0.60 | 0.71 | 0.79 | 0.57 | 7 / 8 ms |
| vector + rerank | 0.59 | 0.71 | 0.78 | 0.81 | 0.66 | 239 / 267 ms |
| **hybrid + rerank** (default) | 0.56 | 0.71 | **0.80** | **0.83** | 0.66 | 243 / 276 ms |

Recall@5 by category:

| Search | citation | statutory | plain | jargon | old_number |
|---|---|---|---|---|---|
| keyword | 1.00 | 1.00 | 0.47 | 0.33 | 0.50 |
| vector | 1.00 | 1.00 | 0.75 | 0.43 | 0.17 |
| hybrid | 1.00 | 1.00 | 0.69 | 0.43 | 0.50 |
| hybrid + rerank | 1.00 | 1.00 | 0.84 | 0.62 | 0.33 |

What the numbers say:

- **Citations and the Act's own wording are solved.** Every miss is a vocabulary gap.
- **The reranker earns its 240 ms**: +9 points recall@5, mostly on plain-English and
  jargon questions. Inside an assistant's tool call that latency is small, so it is on
  by default.
- **Hybrid fusion is no better than vector search alone** here: equal at R@5, worse
  at R@1. The difference is one or two questions, too few to call.
- **Practitioners' terms fail most.** "FIR", "anticipatory bail", "chargesheet" and
  "sedition" appear nowhere in the text. The tool description tells the assistant to
  rephrase in statutory language; how much that recovers isn't measured yet.

**The CI gate** ([`ci.yml`](.github/workflows/ci.yml)) builds the index from the
committed sections, runs the tests, then `citable eval --check`, which fails the
build if any configuration's recall@5 or MRR drops more than 0.03 below
[`evals/baseline.json`](evals/baseline.json). After an intended improvement, refresh
it with `uv run citable eval --save-baseline`.

The questions were written alongside the search code, before its results were known,
and nothing has been tuned on them. A held-out set written by someone else would be
a stronger test.

## Cost & latency dashboard

Every MCP tool call is logged to `data/telemetry.db`: the arguments, the time spent
in each search stage, errors, and the size of the text sent back into the
assistant's context. `uv run citable dashboard --open` turns the log into one
self-contained HTML page (no network, no JavaScript libraries), with source and
time-range filters, a table view for every chart, and light and dark modes.

![Citable dashboard: tool calls, search latency, context and cost per call, latency by tool, and where search time goes](docs/dashboard.jpg)

`uv run citable replay` sends the 85 eval questions through the tools the way an
assistant would (search, then read the top section). From that run, 181 calls:

| | |
|---|---|
| Search latency, median / p95 | 234 / 272 ms |
| Share of search time spent reranking | 96% (227 ms) |
| Context per call (search / get_section / list_contents) | ≈1,300 / 600 / 670 tokens |
| Cost per 1,000 calls (Claude Opus 5.5 / Sonnet 5.5 / Haiku 5.5) | $3.79 / $1.90 / $0.09 |
| First call after start (loading the models) | +375 ms |

Cost is what Citable's results add to the assistant's bill: tool results are input
tokens for whichever model the assistant runs, while Citable's own search runs locally
and is free. Tokens are estimated at about 4 characters each (exact counts need
Anthropic's token-counting API); the logged size is exactly the text the MCP SDK hands
the model, which a test checks.

Two conclusions. Reranking is nearly all of the latency, so it is the one stage worth
optimising (a smaller cross-encoder, or fewer candidates), and the eval says what each
option costs in recall. And context, not compute, is the real cost: a search returns
about 1,300 tokens, almost all of it statute text (JSON indentation is about 6%), so
`limit` and snippet length are the levers.

The log contains the questions people asked, so it stays on your machine
(`data/telemetry.db` is gitignored). Set `CITABLE_TELEMETRY=off` to stop logging.

## Limitations

- **Not legal advice.** Citable finds and quotes the statute; interpreting it is up to
  the assistant and, ultimately, a lawyer.
- **Three Acts, as enacted.** No case law, rules, state laws, or amendments after the
  2023 Gazette text.
- **Old section numbers mislead.** "Section 438 CrPC" returns BNSS ss. 438–439, unrelated
  provisions that share the number. Fixing this needs a correspondence table from the
  old codes to the new ones.
- **Jargon the statute never uses** ("FIR", "anticipatory bail") is the weakest
  category, at 62% recall@5.

## Project layout

```text
src/citable/
  sources.py     the three Acts and their Gazette URLs
  download.py    fetch the PDFs (egazette.gov.in serves an incomplete TLS chain; uses the OS trust store)
  parse.py       PDF -> sections, using the Gazette's page layout
  chunk.py       sections -> ~1,200-character passages
  embed.py       local embedding and reranking models (fastembed, ONNX)
  index.py       build the SQLite index (FTS5 + sqlite-vec)
  search.py      citation lookup, BM25, vector, RRF, rerank
  server.py      the MCP server and its three tools
  evaluate.py    recall@k, MRR, the CI gate
  telemetry.py   per-call logging
  dashboard.py   the HTML dashboard
  cli.py         `citable ...` commands
evals/           labelled questions and the CI baseline
data/processed/  parsed sections (committed); everything else in data/ is generated
tests/           63 tests: parser, search, server, eval, telemetry
```

## Development

```bash
uv run pytest                    # all tests (a few need the built index or the PDFs, and skip without them)
uv run ruff check src tests
uv run citable eval --misses     # scores, plus every question answered badly
uv run citable download          # re-fetch the Gazette PDFs into data/raw/
uv run citable parse             # re-parse them into data/processed/
```

Built with Python 3.12, PyMuPDF, SQLite FTS5, sqlite-vec, fastembed, Pydantic, Typer
and the MCP Python SDK.

## License

The code is released under the [MIT License](LICENSE). The statute text in
`data/processed/` comes from the Gazette of India (egazette.gov.in), the official
publication of the Acts, and is not covered by this license.