Skip to main content
Glama
README.md
# gkb

A lightweight contract knowledge base. It reads scanned (image-only) Mongolian purchase agreements, extracts the key terms and priced items with OpenAI, and indexes them in Qdrant for hybrid search (dense + sparse) through a CLI and an MCP server, so people and AI agents can find suppliers and contracts for one or more items.

> **Status:** working end to end, tuned on four sample contracts. Known weak spots: see [Known limitations](#known-limitations).

## How it works

1. **Render:** each PDF page becomes a PNG (PyMuPDF), cached by the PDF's sha256 in `data/derived/`.
2. **Pass 1, per page:** a vision model labels the page (`key_terms`, `item_list`, `product_sheet`, ...) and transcribes the useful ones as markdown.
3. **Pass 2:** a text model extracts the contract essentials (supplier, validity, ordering, delivery, payment, warranty), then the items page by page, carrying the table header over between pages.
4. **Result:** `data/kb/<doc_id>.json` is the source of truth and can be edited by hand.
5. **Index:** items and contracts are embedded (dense) and hashed into word + character-3-gram sparse vectors, then stored in Qdrant. Queries run both and fuse them with RRF.

Reading precision is deliberately "good enough": each page is read once with a cheap model, with no retries or review stages. Every item records its source page, so misreads are easy to check and fix by hand.

## Setup

Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).

```
uv sync
cp .env.example .env     # then set OPENAI_API_KEY
```

| Variable | Purpose |
|---|---|
| `OPENAI_API_KEY` | OpenAI credentials |
| `GKB_VISION_MODEL` | Model for page reading (pass 1) |
| `GKB_TEXT_MODEL` | Model for extraction (pass 2) |
| `GKB_EMBED_MODEL` | Embedding model |
| `QDRANT_URL` | Qdrant server; unset uses embedded storage in `data/qdrant` |
| `GKB_RENDER_DPI`, `GKB_WORKERS` | Optional: render resolution (150) and thread count (4) |

## Usage

Put PDFs in `data/contracts/`, then:

```
uv run python -m gkb.cli ingest                  # read + index all PDFs in data/contracts/
uv run python -m gkb.cli ingest path/to.pdf      # one file (--force ignores caches)
uv run python -m gkb.cli reindex [doc_id ...]    # rebuild the index after editing data/kb/*.json

uv run python -m gkb.cli search safety boots [--category C] [--supplier S] [--max-price N] [--all] [--json]
uv run python -m gkb.cli suggest "ажлын гутал" "кран операторын сургалт" "ширээ"
uv run python -m gkb.cli contracts training --active-on 2027-01-01
uv run python -m gkb.cli suppliers [name-fragment]
uv run python -m gkb.cli show sample-contract-a [--items]
```

A full ingest of four contracts (66 pages) takes roughly 15 minutes; later runs reuse the caches. Every result carries `doc_id`, `page` and the path of the scanned `page_image`, so a price can be checked against the source.

Embedded Qdrant (`data/qdrant`) allows one process at a time: stop the MCP server before using the CLI, or run a Qdrant server (`docker compose up -d`, then set `QDRANT_URL=http://localhost:6333`). Payload indexes only take effect on a server.

## MCP server

`.mcp.json` registers the server for Claude Code (`uv run python -m gkb.mcp_server`, stdio). Tools: `search_items`, `suggest_suppliers`, `search_contracts`, `get_contract`, `list_suppliers`. To try it by hand: `npx @modelcontextprotocol/inspector uv run python -m gkb.mcp_server`.

## Adding a contract

Copy the PDF into `data/contracts/` and run `ingest`. Fix any misread value in `data/kb/<doc_id>.json`, then `reindex <doc_id>`.

## Layout

```
gkb/config.py     settings from env / .env
gkb/schemas.py    JSON schemas (OpenAI strict mode) and category list
gkb/render.py     PDF -> PNG
gkb/llm.py        OpenAI calls (swap this file to change provider)
gkb/prompts/      prompt texts
gkb/ingest.py     pass 1 + pass 2 pipeline
gkb/embed.py      OpenAI embeddings
gkb/sparse.py     hashed word / char-3-gram sparse vectors
gkb/index.py      Qdrant collections, indexing, reindex
gkb/search.py     search_items, suggest_suppliers, search_contracts, get_contract, list_suppliers
gkb/cli.py        command line
gkb/mcp_server.py MCP server
data/             contracts, caches and the knowledge base (git-ignored)
```

## Known limitations

- Reading is single-pass and cheap, so some values are misread; check against the page and fix the JSON.
- Mongolian queries rely mostly on the sparse side: dense embeddings are weak on Mongolian, and equal-weight fusion can dilute good sparse hits. Mixing Mongolian and English synonyms in a query helps.
- `suggest` has no absolute relevance cut-off, so a supplier can be listed as covering an item with a weak match. Read the matched item names.
- Short queries can match through character 3-grams (e.g. `ширээ` also surfaces `ширэм...` pipe items).

## Confidential data

Contracts and everything derived from them (`data/`) and `.env` are git-ignored on purpose. Never commit them.

## License

GPL-3.0, see [LICENSE](LICENSE).