Skip to main content
Glama
docintel MCP Server

docintel MCP Server

by Shyam2119

Document Intelligence Assistant

Ask questions about government welfare-scheme documents, in Telugu or English, and get answers that cite the exact page. Scanned PDFs are read with OCR. A second mode checks whether a person qualifies for each scheme and explains why, rule by rule.

Asking a question

What it does

  • Reads messy documents. It reads text PDFs, scanned PDFs and photos (Tesseract OCR, English and Telugu) and plain text. Repeated headers and stamps are removed before indexing.

  • Finds the right page. Hybrid retrieval combines BM25 for exact words, amounts and scheme names with vector search for meaning, merged with Reciprocal Rank Fusion.

  • Answers with citations or says "not found". The model may only use the retrieved passages. Every fact carries a [n] citation that points at a file and page. Citations that point nowhere are removed. If retrieval is too weak, the system refuses before calling the model.

  • Checks eligibility with an agent (stage 2). A LangGraph workflow reads a free-text description of a person and checks each scheme's rules. It answers eligible, not eligible or need info, and asks follow-up questions. The rules are JSON that cites its source page. Verdicts come from code, not from the model, so they are repeatable.

  • Measures itself. It includes an evaluation set, a threshold calibration command, 100 agent test cases, and quality gates in CI that fail the build if accuracy drops.

  • Compares models (stage 3). One script compares an extractive baseline, a QLoRA fine-tuned 1.5B–3B open model and hosted API models on the same questions.

  • Ships as a service. It has a FastAPI backend, a web UI, SQL query logging with feedback and stats, a Dockerfile, docker-compose with Postgres, GitHub Actions CI, and an MCP server so other AI agents can use its tools.

Eligibility check

Related MCP server: Kela MCP Server

Results so far

The numbers below are on the fictional demo corpus in data/demo. That corpus has four made-up schemes: two text PDFs, one scanned PDF and one Telugu document. There are 39 questions, including 8 that the documents cannot answer. The defaults were used: TF-IDF embeddings and the extractive answerer, with no API key. Reproduce them with python -m docintel eval.

Metric

Extractive baseline

Retrieval hit@5 (gold page in top 5)

93.5% (100% for same-language questions)

Answer accuracy (all required facts present)

67.7%

Citation accuracy (answer cites the gold page)

86.7%

Correct refusals on unanswerable questions

50.0%

Wrong refusals on answerable questions

3.2%

Eligibility agent, 100 cases

100% verdicts, 100% field extraction

The baseline is weakest at refusing near-miss questions, such as "interest rate on a Kisan Sahaya crop loan". An LLM that may answer NOT_FOUND is expected to fix that, and stage 3 measures it. The two English-question/Telugu-document questions fail with TF-IDF by design. They need the multilingual embedder (st:intfloat/multilingual-e5-small).

The 100 agent cases were written together with the parser, so they guard against regressions. They are not a measure of how well it generalises.

Still to do: run the evaluation on real scheme documents (see docs/REAL_DATA.md), and run the stage 3 fine-tune on a GPU and fill in the comparison table. No numbers are claimed for either yet.

Quick start

python -m venv .venv
.venv\Scripts\activate            # Windows  (macOS/Linux: source .venv/bin/activate)
pip install -r requirements-dev.txt
python scripts/make_demo_corpus.py   # only if data/demo/docs is empty
python -m docintel ingest data/demo/docs
python -m docintel ask "What is the hostel allowance for degree students?"
python -m docintel serve             # open http://localhost:8000  (API docs at /docs)
pytest -q

OCR needs Tesseract:

  • Windows: install the UB-Mannheim build. Tick Telugu under "Additional language data", then add C:\Program Files\Tesseract-OCR to PATH.

  • Linux: run sudo apt install tesseract-ocr tesseract-ocr-tel poppler-utils.

Scanned PDFs on Windows also need Poppler on PATH.

Better models

Copy .env.example to .env and choose:

Goal

Setting

Telugu and English questions over both languages

DOCINTEL_EMBEDDER=st:intfloat/multilingual-e5-small (needs pip install -r requirements-gpu.txt)

Free local LLM on your GPU

Install Ollama, run ollama pull qwen2.5:3b, then set DOCINTEL_LLM_PROVIDER=ollama

Hosted LLM

DOCINTEL_LLM_PROVIDER=gemini plus GEMINI_API_KEY, or openai with any OpenAI-compatible endpoint

After changing the embedder, re-run ingest, then python -m docintel calibrate to pick the "not found" thresholds.

Commands

Command

What it does

python -m docintel ingest <folder>

Index every PDF, image and text file in a folder

python -m docintel ask "<question>"

Answer one question with citations

python -m docintel eval [file]

Retrieval and answer metrics; writes eval/results/<model>.md

python -m docintel calibrate

Choose the refusal thresholds from the evaluation set

python -m docintel agent --text "..."

Run the eligibility agent

python -m docintel agent-eval

Score the agent on 100 test cases

python -m docintel serve

API and web UI on port 8000

python -m docintel mcp

MCP server (stdio) exposing search_documents, ask_documents, check_eligibility, list_schemes

API

Method

Path

Purpose

GET

/health

Status, number of passages, models in use

GET / POST

/documents

List documents, or upload one (≤ 25 MB)

POST

/ask

{"question": "..."} returns an answer, citations, latency and retrieval details

POST

/feedback

{"query_id": 1, "helpful": true}

GET

/stats

Per-model question count, latency, refusal rate and feedback (SQL GROUP BY)

POST

/agent/eligibility

{"text": "...", "profile": {...}} returns per-scheme verdicts, reasons and follow-up questions

How it works

 file ─▶ loader ─(no text layer?)─▶ OCR ─▶ boilerplate removal ─▶ page-aware chunks
                                                                       │
                                              BM25 index ◀─────────────┼────────▶ vector index
                                                   └──────── RRF fusion ┘
 question ─▶ hybrid search ─▶ "enough evidence?" ─no─▶ "not found"
                                   │ yes
                                   ▼
                     numbered passages ─▶ LLM ─▶ keep valid [n] citations ─▶ answer + pages

Eligibility agent (LangGraph):

parse_profile ─▶ select_schemes ─▶ check_eligibility ─┬─▶ explain        (some verdicts known)
                                                       └─▶ ask_follow_up  (every scheme lacks facts)

docs/HOW_IT_WORKS.md explains each design decision and the trade-offs.

Stage 3: fine-tuning on an 8 GB laptop GPU

pip install torch --index-url https://download.pytorch.org/whl/cu128   # RTX 50-series needs CUDA 12.8 builds
pip install -r finetune/requirements.txt
# 1. a teacher model writes Q&A pairs for each passage (questions close to eval questions are dropped)
set DOCINTEL_LLM_PROVIDER=gemini & set GEMINI_API_KEY=...
python finetune/build_dataset.py --per-chunk 4
# 2. QLoRA: 4-bit base model, LoRA r=16 on all projections, loss on answer tokens only
python finetune/train_qlora.py --model Qwen/Qwen2.5-1.5B-Instruct --epochs 2
# 3. same questions, same retrieval, different generators
set DOCINTEL_HF_ADAPTER=finetune/out/adapter
python finetune/compare.py --providers extractive hf_local ollama gemini --price gemini=0.35

Project layout

docintel/
  ingest/     loader (PDF, OCR, text), sentence splitting, chunking, boilerplate removal
  index/      BM25 + embeddings + RRF hybrid index
  generate/   prompt, LLM providers, citation checking, refusal gate
  agent/      rules engine, free-text profile parser, tools, LangGraph graph
  eval/       answer evaluation, threshold calibration, agent evaluation
  api.py      FastAPI app      db.py  SQLAlchemy models and stats
  mcp_server.py                static/index.html  web UI
finetune/     dataset builder, QLoRA trainer, model comparison
data/demo/    fictional documents and their rules
eval/         question sets and agent cases
tests/        42 tests, including evaluation quality gates

Limits

  • The demo documents are fictional. Real government PDFs have tables, two columns and stamps that this pipeline will handle less well. That is why the real-data evaluation matters.

  • Eligibility rules are written by hand from the documents, one JSON file per scheme. Extracting them with an LLM is possible, but every rule would still need a human check.

  • The agent's verdict is guidance only. The department makes the final decision.

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to answer questions about Finnish social benefits (Kela) by providing tools for searching, checking eligibility, and getting application steps.
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables searching live myScheme.gov.in schemes by eligibility, keyword, and detail lookup through MCP tools.
      -
    • F
      license
      Not graded
      quality
      C
      maintenance
      Enables natural-language question answering over Australian government policy documents with grounded, source-cited responses and a tool to list indexed documents.
      -