Skip to main content
Glama
README.md
# ai-act-rag

Ask questions about the EU AI Act (Regulation (EU) 2024/1689) in Spanish or English and get answers grounded in cited articles — with hybrid retrieval, an evaluation suite and an MCP server so Claude can use it as a tool.

![Web page answering "Is emotion recognition in the workplace prohibited?" with Article 5(f), Recital 44 and Recital 18 as sources (demo mode, no API key)](docs/screenshot.png)

*The screenshot shows demo mode: no API keys were available on the machine where this was built, so the answer is extractive (the best sentence of each retrieved passage, clearly labelled). With `ANTHROPIC_API_KEY` set, Claude writes the answer from the same passages.*

## Features

- **Official corpus, parsed structurally.** The EUR-Lex HTML (ES + EN) becomes 113 articles, 180 recitals and 13 annexes per language, each with a stable id (`art-5`, `rec-27`, `anx-III`) and a localized label (`Art. 5` / `Article 5`, `Considerando 27` / `Recital 27`).
- **BM25 retrieval in SQLite FTS5**, with Spanish and English Snowball stemming done in Python (FTS5's built-in stemmer is English-only). No server, no cost, fully offline.
- **Optional hybrid retrieval**: with `VOYAGE_API_KEY`, dense vectors from Voyage embeddings are fused with BM25 using Reciprocal Rank Fusion.
- **Grounded answers with Claude** (`claude-sonnet-5-5` by default, `ANTHROPIC_MODEL` to override) via structured outputs: `found`, `answer`, `cited_ids`. Cited ids are checked against the retrieved passages, and unsupported questions get *"Not found in the AI Act."*
- **Demo mode** without a key: an extractive answer from the top passages, labelled as such.
- **FastAPI** (`/ask`, `/search`, `/articles/{id}`) and a small black-and-white web page.
- **MCP server** (`search_ai_act`, `get_article`) so Claude Desktop or Claude Code can query the Regulation directly.
- **Evals**: 47 gold questions (22 ES, 25 EN), each with an evidence quote that a test checks against the parsed official text. Recall@1/@5 and MRR, per language, with a CI regression gate.
- **Pluggable store**: the same `Store` interface backed by SQLite (default) or Postgres + pgvector (`docker-compose.yml`).

## Quickstart

Requires Python 3.12.

```bash
python -m venv .venv
source .venv/bin/activate                 # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

python -m ai_act_rag.ingest --download    # fetches ~2.6 MB of HTML into data/raw, builds data/ai_act.sqlite
uvicorn ai_act_rag.api:app --reload       # web page at http://127.0.0.1:8000, API docs at /docs
```

The app reads its configuration from environment variables; all are optional (see [`.env.example`](.env.example)).

| Variable | Effect |
|---|---|
| `ANTHROPIC_API_KEY` | Claude answers instead of the extractive demo mode |
| `ANTHROPIC_MODEL` | Model id (default `claude-sonnet-5-5`) |
| `VOYAGE_API_KEY` | Dense embeddings + RRF; re-run `ingest` after setting it |
| `AI_ACT_STORE`, `DATABASE_URL` | `postgres` to use the pgvector backend |

```bash
curl -s localhost:8000/ask -H 'content-type: application/json' \
  -d '{"question": "Is emotion recognition in the workplace prohibited?", "k": 5}'
```

```json
{
  "answer": "Demo mode (no ANTHROPIC_API_KEY): the most relevant passages, not a generated answer.\n\n• [Article 5] (f) the placing on the market, the putting into service for this specific purpose, or the use of AI systems to infer emotions of a natural person in the areas of workplace and education institutions, except where ... for medical or safety reasons;\n• [Recital 44] ...",
  "citations": [
    {"id": "Article 5", "title": "Prohibited AI practices", "snippet": "(f) the placing on the market, ...", "score": 17.0316}
  ],
  "mode": "extractive",
  "retrieval": "bm25",
  "lang": "en"
}
```

Other endpoints: `GET /search?q=multas&k=5`, `GET /articles/Art.%205?lang=es` (ids such as `5`, `art-5`, `Article 5`, `Considerando 27` or `Annex III` are all accepted), `GET /health`.

## MCP server

`python -m ai_act_rag.mcp_server` speaks MCP over stdio and exposes two read-only tools:

- `search_ai_act(query, k)`: the top k articles, recitals or annexes with their passage text and ids.
- `get_article(id, lang?)`: the full text of one unit; the language follows the id (`Considerando 27` returns Spanish).

The server only retrieves; the client model does the reasoning, so it needs no Anthropic key. Use the absolute path of the virtualenv's Python so the client finds the package and the index wherever it starts the process.

**Claude Desktop**: add this to `claude_desktop_config.json` (Windows: `%APPDATA%\Claude\`, macOS: `~/Library/Application Support/Claude/`) and restart the app:

```json
{
  "mcpServers": {
    "ai-act": {
      "command": "/absolute/path/to/ai-act-rag/.venv/bin/python",
      "args": ["-m", "ai_act_rag.mcp_server"]
    }
  }
}
```

On Windows the command is `C:\\path\\to\\ai-act-rag\\.venv\\Scripts\\python.exe`.

**Claude Code**:

```bash
claude mcp add ai-act -- /absolute/path/to/ai-act-rag/.venv/bin/python -m ai_act_rag.mcp_server
```

Then ask, for example: *"Using the ai-act tools, which obligations does Article 50 put on deployers of deep fakes?"*

## Architecture

```mermaid
flowchart LR
    subgraph ingest["Ingest (offline, once)"]
        html["EUR-Lex HTML, ES + EN"] --> parser["parser.py: articles, recitals, annexes"]
        parser --> chunker["chunker.py: max 300 words, paragraph overlap"]
        chunker --> store[("SQLite: FTS5 BM25 + optional vectors")]
        chunker -. "VOYAGE_API_KEY" .-> voyage["Voyage embeddings"]
        voyage -.-> store
    end
    subgraph query["Query"]
        q["Question"] --> lang["detect ES / EN"]
        lang --> bm25["BM25 top 50"]
        lang -. "if vectors" .-> dense["dense top 50"]
        bm25 --> rrf["RRF k=60, best chunk per unit"]
        dense -.-> rrf
        rrf --> claude["Claude: structured JSON, citations checked"]
        rrf --> extractive["Extractive demo answer"]
    end
    store --- bm25
    store --- dense
    claude --> api["FastAPI + web page"]
    extractive --> api
    store --- mcp["MCP server: search_ai_act, get_article"]
```

| Module | Responsibility |
|---|---|
| `corpus.py` | Downloads both languages from EUR-Lex, retries, rejects error pages served with HTTP 200 |
| `parser.py` | HTML to `Unit`s (article, recital, annex); enumerations flattened, footnote markers dropped, `10^25` kept as an exponent |
| `chunker.py` | One chunk per unit when it fits; otherwise paragraph packing with overlap |
| `text.py` | Tokenising, stopwords, Snowball stemming, ES/EN detection, snippets |
| `stores/` | `Store` protocol; `sqlite.py` (FTS5 + NumPy cosine) and `postgres.py` (tsvector + pgvector HNSW) |
| `retrieval.py` | BM25 or BM25 + dense with RRF; deduplication to one chunk per unit |
| `answer.py` | Claude structured-output answerer, extractive answerer, citation validation |
| `api.py`, `mcp_server.py` | HTTP and MCP front ends over the same `RagService` |
| `eval.py` | Gold-set metrics and the optional LLM judge |

## Evaluation

`python -m ai_act_rag.eval` runs the 47 questions in [`evals/gold.jsonl`](evals/gold.jsonl). Each item lists the ids that answer it (an article, sometimes with an annex or recital that also answers it) and an evidence quote; `tests/test_gold.py` checks that every quote occurs verbatim in the parsed text of the first expected unit.

- **recall@k**: share of questions with at least one gold id among the top k retrieved units.
- **MRR@10**: mean of 1/rank of the first gold id within the top 10 (0 if absent).

Offline BM25 results (SQLite FTS5, measured on this repository's index):

| retrieval | questions | n | recall@1 | recall@5 | MRR@10 |
|---|---|---|---|---|---|
| BM25 | all | 47 | 0.553 | 0.830 | 0.682 |
| BM25 | ES | 22 | 0.636 | 0.818 | 0.722 |
| BM25 | EN | 25 | 0.480 | 0.840 | 0.647 |

**Hybrid (BM25 + Voyage, RRF) has not been measured**: no Voyage key was available when these numbers were produced. With a key, `VOYAGE_API_KEY=... python -m ai_act_rag.ingest && python -m ai_act_rag.eval` adds the hybrid rows. Likewise, `--judge` (Claude grading each answer for faithfulness against the exact passages it was given) exists but has not been run.

Retrieval latency: median 17 ms, p95 38 ms per query (235 in-process BM25 queries over the gold questions, on a Windows 11 PC).

### Error analysis

21 of 47 questions do not have a gold id at rank 1, and 8 miss the top 5.

- **Recitals outrank the article (10 of the 13 questions answered at ranks 2–4).** Recitals explain the articles in plainer, longer prose that overlaps more with natural questions. For "How must high-risk AI systems enable humans to oversee them?", Recital 73 ranks above Article 14. The recital is relevant, but the binding text is the article, and binary relevance counts this as a miss at rank 1.
- **Vocabulary mismatch (5 of the 8 top-5 misses).** "banned outright" vs "prohibited" (Art. 5), "build the model into their own AI systems" vs "integrate" (Art. 53), "chatbot" / "talking to a machine" / "hablando" vs "interacting with an AI system" / "interactuando" (Art. 50, twice), "what date" vs "2 August 2026" (Art. 113). This is the gap dense retrieval is meant to close.
- **Shared vocabulary (3 of the 8).** "sistema de IA" occurs in 116 of the 306 Spanish units, so Article 3's definition does not stand out, and Article 3 is split into 13 chunks of definitions that compete individually. For systemic risk and training data, Recitals 111–112 and Article 52, or Annex VII and Recitals 67–68, use the question's wording and outrank Articles 51 and 10.
- **Neighbouring articles.** Article 100 (fines for Union institutions, which also mentions the Article 5 prohibitions) ranks above Article 99 for both fine questions.

With 47 questions the 95% confidence interval on recall@5 is roughly ±0.11, so the ES/EN difference above is within noise.

## Technical decisions and trade-offs

- **BM25 first.** Legal questions lean on defined terms ("deployer", "provider", "high-risk", "systemic risk") where exact lexical matching is strong, and BM25 is deterministic, explainable, free and testable offline in CI. Dense retrieval is an add-on enabled by a key, not a dependency; no local embedding model or GPU is needed.
- **Stemming in Python, not FTS5.** FTS5's `porter` tokenizer only stems English. Text is analysed with Snowball (Spanish or English) plus stopword lists before it reaches FTS5, and queries go through the same function, so "prohibidas" matches "prohibido".
- **Chunk by legal unit.** An article, recital or annex is both the semantic unit and the citation unit. Units of up to 300 words stay whole (most recitals, about half the articles); longer ones are packed paragraph by paragraph with up to 60 words of overlap, so a "paragraph 1" reference keeps its context (Article 3, the definitions, becomes 11 chunks in English and 13 in Spanish). Results are deduplicated to the best chunk per unit, so k results are k distinct citable units. The 300/60 values were set before running the eval and have not been tuned.
- **Reciprocal Rank Fusion.** BM25 scores are unbounded and cosine similarities lie in [-1, 1]; RRF (`1/(60 + rank)`) uses only ranks, so the two never need calibrating and there is a single, standard constant. The cost: score magnitudes are discarded, so a very confident match in one list counts the same as a marginal one.
- **Structured outputs instead of parsing prose.** Claude returns `{found, answer, cited_ids}` through the SDK's `messages.parse` with a Pydantic model. "Not found" is a boolean, not a string match, and every cited id must map to a passage that was actually retrieved, or it is dropped. Excerpts are XML-escaped and the system prompt marks them as reference material, not instructions.
- **Model settings.** `claude-sonnet-5-5` with effort `low`: short answers over a handful of passages, where latency matters more than deep reasoning. Requests opt into server-side fallback (`fallbacks: "default"`), so a safety decline is retried on Anthropic's recommended model. A refusal, truncation, invalid JSON or API error falls back to the extractive answer and is logged.
- **SQLite by default, Postgres optional.** A single file with no server is enough for about 1,000 chunks: dense search is a brute-force NumPy dot product. The Postgres backend implements the same `Store` protocol with `tsvector` + `ts_rank_cd` (not BM25, so its scores are not comparable) and a pgvector HNSW index, for multi-instance deployments.
- **MCP exposes retrieval, not generation.** The MCP client is already an LLM, so the server returns passages and full texts and lets Claude reason and cite. Both tools are marked read-only, and logs go to stderr because stdout carries the protocol.
- **Untrusted inputs.** The downloaded HTML is parsed as data; scripts are dropped and never run. FTS5 query terms are quoted, so user input cannot inject `NEAR`, `*` or column filters. The page renders answers with `textContent`, never `innerHTML`.

## Tests and quality

```bash
ruff format --check . && ruff check .    # formatting + lint
mypy                                      # strict, src and tests
pytest                                    # tests/test_postgres.py is skipped without a database
python -m ai_act_rag.eval --fail-under 0.80
```

The tests cover the parser (on verbatim EUR-Lex excerpts in `tests/fixtures/`), chunking, id parsing, stemming and language detection, RRF, BM25 and hybrid retrieval (with a deterministic fake embedder), the Voyage client (mock HTTP transport), the answerer (including one test that sends a request through the real Anthropic SDK to a mock transport and checks the wire body), the API through `TestClient`, the MCP tools through an in-process MCP client, the downloader's retry on error pages, and the gold set itself. CI (`.github/workflows/ci.yml`) runs lint, type checks, tests and the eval gate on pushes to `main` and on pull requests.

## Deployment

`Dockerfile` builds a self-contained image (corpus and index baked in at build time, non-root user, health check) and `docker-compose.yml` adds Postgres + pgvector. Neither has been built here: Docker was not available on the development machine. Notes for Azure Container Apps and AWS (App Runner for existing accounts, ECS Express Mode for new ones) are in [docs/deploy.md](docs/deploy.md).

## Limitations and next steps

- The gold set is small and was written by the author of the system, who knows the text, so questions may echo its vocabulary more than real users would. Next: questions from real users, and graded relevance (article = 2, recital = 1) with nDCG.
- Hybrid retrieval, the LLM judge and the Claude answer path have not been run against the live APIs; the Claude path is tested with mocks only.
- Rank-1 accuracy is limited by recitals outranking articles. Options to measure: an article prior, a cross-encoder reranker over the top 20, or query expansion.
- The language filter means a Spanish question only searches the Spanish text. Multilingual embeddings could search both and still cite in the question's language.
- The index is the original Official Journal text (CELEX 32024R1689), not a consolidated version with later amendments or corrigenda.
- Before a public deployment: authentication and rate limiting on `/ask` (it spends API credit), response caching, and streaming answers.

## License and data

Code: MIT, see [LICENSE](LICENSE). Corpus: Regulation (EU) 2024/1689 from [EUR-Lex](https://eur-lex.europa.eu/eli/reg/2024/1689/oj), © European Union, 1998-2026, reused under the EUR-Lex legal notice (Commission Decision 2011/833/EU). Changes made by this project: the HTML is parsed into articles, recitals and annexes, reformatted (enumeration labels joined to their text, footnote markers removed, whitespace normalised) and split into passages. Only EU documents published in the Official Journal of the European Union are deemed authentic. Nothing here is legal advice.