Skip to main content
Glama
docintel MCP Server

docintel MCP Server

by Shyam2119
README.md
# Document Intelligence Assistant

Ask questions about government welfare-scheme documents, in Telugu or English, and get answers that cite the exact page. Scanned PDFs are read with OCR. A second mode checks whether a person qualifies for each scheme and explains why, rule by rule.

![Asking a question](docs/img/ask.png)

## What it does

- **Reads messy documents.** It reads text PDFs, scanned PDFs and photos (Tesseract OCR, English and Telugu) and plain text. Repeated headers and stamps are removed before indexing.
- **Finds the right page.** Hybrid retrieval combines BM25 for exact words, amounts and scheme names with vector search for meaning, merged with Reciprocal Rank Fusion.
- **Answers with citations or says "not found".** The model may only use the retrieved passages. Every fact carries a `[n]` citation that points at a file and page. Citations that point nowhere are removed. If retrieval is too weak, the system refuses before calling the model.
- **Checks eligibility with an agent** (stage 2). A LangGraph workflow reads a free-text description of a person and checks each scheme's rules. It answers *eligible*, *not eligible* or *need info*, and asks follow-up questions. The rules are JSON that cites its source page. Verdicts come from code, not from the model, so they are repeatable.
- **Measures itself.** It includes an evaluation set, a threshold calibration command, 100 agent test cases, and quality gates in CI that fail the build if accuracy drops.
- **Compares models** (stage 3). One script compares an extractive baseline, a QLoRA fine-tuned 1.5B–3B open model and hosted API models on the same questions.
- **Ships as a service.** It has a FastAPI backend, a web UI, SQL query logging with feedback and stats, a Dockerfile, docker-compose with Postgres, GitHub Actions CI, and an MCP server so other AI agents can use its tools.

![Eligibility check](docs/img/eligibility.png)

## Results so far

The numbers below are on the **fictional demo corpus** in `data/demo`. That corpus has four made-up schemes: two text PDFs, one scanned PDF and one Telugu document. There are 39 questions, including 8 that the documents cannot answer. The defaults were used: TF-IDF embeddings and the extractive answerer, with no API key. Reproduce them with `python -m docintel eval`.

| Metric | Extractive baseline |
|---|---|
| Retrieval hit@5 (gold page in top 5) | 93.5% (100% for same-language questions) |
| Answer accuracy (all required facts present) | 67.7% |
| Citation accuracy (answer cites the gold page) | 86.7% |
| Correct refusals on unanswerable questions | 50.0% |
| Wrong refusals on answerable questions | 3.2% |
| Eligibility agent, 100 cases | 100% verdicts, 100% field extraction |

The baseline is weakest at refusing near-miss questions, such as "interest rate on a Kisan Sahaya crop loan". An LLM that may answer `NOT_FOUND` is expected to fix that, and stage 3 measures it. The two English-question/Telugu-document questions fail with TF-IDF by design. They need the multilingual embedder (`st:intfloat/multilingual-e5-small`).

The 100 agent cases were written together with the parser, so they guard against regressions. They are not a measure of how well it generalises.

**Still to do:** run the evaluation on real scheme documents (see [docs/REAL_DATA.md](docs/REAL_DATA.md)), and run the stage 3 fine-tune on a GPU and fill in the comparison table. No numbers are claimed for either yet.

## Quick start

```bash
python -m venv .venv
.venv\Scripts\activate            # Windows  (macOS/Linux: source .venv/bin/activate)
pip install -r requirements-dev.txt
python scripts/make_demo_corpus.py   # only if data/demo/docs is empty
python -m docintel ingest data/demo/docs
python -m docintel ask "What is the hostel allowance for degree students?"
python -m docintel serve             # open http://localhost:8000  (API docs at /docs)
pytest -q
```

OCR needs Tesseract:
- **Windows:** install the UB-Mannheim build. Tick **Telugu** under "Additional language data", then add `C:\Program Files\Tesseract-OCR` to PATH.
- **Linux:** run `sudo apt install tesseract-ocr tesseract-ocr-tel poppler-utils`.

Scanned PDFs on Windows also need Poppler on PATH.

### Better models

Copy `.env.example` to `.env` and choose:

| Goal | Setting |
|---|---|
| Telugu and English questions over both languages | `DOCINTEL_EMBEDDER=st:intfloat/multilingual-e5-small` (needs `pip install -r requirements-gpu.txt`) |
| Free local LLM on your GPU | Install [Ollama](https://ollama.com), run `ollama pull qwen2.5:3b`, then set `DOCINTEL_LLM_PROVIDER=ollama` |
| Hosted LLM | `DOCINTEL_LLM_PROVIDER=gemini` plus `GEMINI_API_KEY`, or `openai` with any OpenAI-compatible endpoint |

After changing the embedder, re-run `ingest`, then `python -m docintel calibrate` to pick the "not found" thresholds.

## Commands

| Command | What it does |
|---|---|
| `python -m docintel ingest <folder>` | Index every PDF, image and text file in a folder |
| `python -m docintel ask "<question>"` | Answer one question with citations |
| `python -m docintel eval [file]` | Retrieval and answer metrics; writes `eval/results/<model>.md` |
| `python -m docintel calibrate` | Choose the refusal thresholds from the evaluation set |
| `python -m docintel agent --text "..."` | Run the eligibility agent |
| `python -m docintel agent-eval` | Score the agent on 100 test cases |
| `python -m docintel serve` | API and web UI on port 8000 |
| `python -m docintel mcp` | MCP server (stdio) exposing `search_documents`, `ask_documents`, `check_eligibility`, `list_schemes` |

## API

| Method | Path | Purpose |
|---|---|---|
| GET | `/health` | Status, number of passages, models in use |
| GET / POST | `/documents` | List documents, or upload one (≤ 25 MB) |
| POST | `/ask` | `{"question": "..."}` returns an answer, citations, latency and retrieval details |
| POST | `/feedback` | `{"query_id": 1, "helpful": true}` |
| GET | `/stats` | Per-model question count, latency, refusal rate and feedback (SQL `GROUP BY`) |
| POST | `/agent/eligibility` | `{"text": "...", "profile": {...}}` returns per-scheme verdicts, reasons and follow-up questions |

## How it works

```
 file ─▶ loader ─(no text layer?)─▶ OCR ─▶ boilerplate removal ─▶ page-aware chunks
                                                                       │
                                              BM25 index ◀─────────────┼────────▶ vector index
                                                   └──────── RRF fusion ┘
 question ─▶ hybrid search ─▶ "enough evidence?" ─no─▶ "not found"
                                   │ yes
                                   ▼
                     numbered passages ─▶ LLM ─▶ keep valid [n] citations ─▶ answer + pages
```

Eligibility agent (LangGraph):

```
parse_profile ─▶ select_schemes ─▶ check_eligibility ─┬─▶ explain        (some verdicts known)
                                                       └─▶ ask_follow_up  (every scheme lacks facts)
```

[docs/HOW_IT_WORKS.md](docs/HOW_IT_WORKS.md) explains each design decision and the trade-offs.

## Stage 3: fine-tuning on an 8 GB laptop GPU

```bash
pip install torch --index-url https://download.pytorch.org/whl/cu128   # RTX 50-series needs CUDA 12.8 builds
pip install -r finetune/requirements.txt
# 1. a teacher model writes Q&A pairs for each passage (questions close to eval questions are dropped)
set DOCINTEL_LLM_PROVIDER=gemini & set GEMINI_API_KEY=...
python finetune/build_dataset.py --per-chunk 4
# 2. QLoRA: 4-bit base model, LoRA r=16 on all projections, loss on answer tokens only
python finetune/train_qlora.py --model Qwen/Qwen2.5-1.5B-Instruct --epochs 2
# 3. same questions, same retrieval, different generators
set DOCINTEL_HF_ADAPTER=finetune/out/adapter
python finetune/compare.py --providers extractive hf_local ollama gemini --price gemini=0.35
```

## Project layout

```
docintel/
  ingest/     loader (PDF, OCR, text), sentence splitting, chunking, boilerplate removal
  index/      BM25 + embeddings + RRF hybrid index
  generate/   prompt, LLM providers, citation checking, refusal gate
  agent/      rules engine, free-text profile parser, tools, LangGraph graph
  eval/       answer evaluation, threshold calibration, agent evaluation
  api.py      FastAPI app      db.py  SQLAlchemy models and stats
  mcp_server.py                static/index.html  web UI
finetune/     dataset builder, QLoRA trainer, model comparison
data/demo/    fictional documents and their rules
eval/         question sets and agent cases
tests/        42 tests, including evaluation quality gates
```

## Limits

- The demo documents are fictional. Real government PDFs have tables, two columns and stamps that this pipeline will handle less well. That is why the real-data evaluation matters.
- Eligibility rules are written by hand from the documents, one JSON file per scheme. Extracting them with an LLM is possible, but every rule would still need a human check.
- The agent's verdict is guidance only. The department makes the final decision.

## License

MIT