Skip to main content
Glama
HGB Basel MCP Server

HGB Basel MCP Server

by thodel
README.md
# Königsfelden — MCP Server

An [MCP](https://modelcontextprotocol.io) server that exposes the Königsfelden corpus —
the records of the Cistercian convent and the Hofmeisterei Königsfelden (1300–1658),
edited by *Die Urkunden und Akten des Klosters und der Hofmeisterei Königsfelden* —
to Claude and other MCP-compatible clients.

## Architecture

```
data/docs/*.xml       (TEI, one file per register entry)  ─┐
data/registers/                                            ├─► build_db.py ──► kf.db
  people.xml  places.xml  organizations.xml               ─┘                  (SQLite + FTS5)
                                                                                   │
                                                                              server.py
                                                              (mcp 2.0 MCPServer,
                                                               streamable HTTP)
                                                                                   │
                                                                    http://<host>:8001/mcp
```

The TEI sources are parsed once into a SQLite database with two FTS5 indexes. The server
then runs stateless read-only queries against it (`PRAGMA query_only`).

The server targets **mcp 2.0**, which renamed the high-level server class
(`FastMCP` → `MCPServer`), removed `mcp.server.fastmcp`, and moved the bind address from
the constructor into `run()`; `requirements.txt` pins the major version accordingly.

**The transport is streamable HTTP** (`/mcp`), not the legacy HTTP+SSE (`/sse`) this
server used previously. SSE is deprecated, and — more practically — its handshake hands
the client an absolute `/messages/` path computed from the app's own mount point, which
a client cannot reach when the server sits under a reverse-proxy sub-path. Streamable
HTTP has one endpoint and no such handshake. **Existing clients pointed at `/sse` must
be repointed at the new endpoint;** it is a deliberate cutover, not a compatible change.

Entity identifiers come from the TEI `xml:id` attributes — persons `perXXXXXX`, places
`locXXXXXX`, organisations `orgXXXX`. Place records additionally carry an HLS id, a GND
id and coordinates where the edition supplies them (30.09.2026 build: 237 of 1,333
places have an HLS id, 170 a GND id, 384 coordinates). Person records have an `hls_id`
column too, but it is always null: the TEI person register carries no `idno[@type="HLS"]`
(0 of 5,260 persons), so the column exists for shape symmetry only.

## Setup

### 1. Install dependencies

```bash
pip install -r requirements.txt
```

### 2. Build the database

```bash
python build_db.py --docs ../data/docs --registers ../data/registers --db kf.db
```

`--docs` is a directory of per-entry TEI files; the entry id is the **filename** without
its extension. `--registers` must contain `people.xml`, `places.xml`, and
`organizations.xml`. Both default to `../data/docs` and `../data/registers`; `--batch`
(default 200) controls the commit batch size.

Run it once, and again whenever the TEI changes. **Rebuilding is destructive**: the five
tables the script owns (`entries`, `spans`, `persons`, `places`, `orgs`) and both FTS
indexes are cleared and repopulated, so a rebuild always mirrors the current sources
rather than accumulating duplicates. It prints `Existing database: clearing N entries`
when it does this. Nothing else in the file is touched.

Malformed records are skipped individually and reported on stderr as
`WARNING: N record(s) skipped` — check for that line, since the build otherwise
completes normally.

### 3. Start the server

```bash
python server.py --db kf.db --host 0.0.0.0 --port 8001
```

Each flag also has an environment variable — `KF_DB`, `KF_HOST`, `KF_PORT`,
`KF_HTTP_PATH` — which the flags override. Importing `server.py` never reads `sys.argv`,
so it is safe to import from tests or an ASGI loader.

`--http-path` (default `/mcp`) is the path the MCP endpoint is served at. **Behind a
reverse proxy, set it to the public path** — see [Reverse proxy](#reverse-proxy-nginx).

### 4. Connect a client

**Claude Code** — the name and URL are positional; there is no `--url` flag:

```bash
claude mcp add --transport http kf http://<server-ip>:8001/mcp -s user
```

`-s user` makes the server available in every project; `-s project` writes it to
`.mcp.json` to share with a repository; the default `local` scope is just you, in the
current project. `claude mcp list` then reports the connection status.

**Claude Desktop, Cowork, claude.ai** — Customize → Connectors → **+** → *Add custom
connector*, and paste the same URL. These clients connect from Anthropic's cloud rather
than from your machine, so the server has to be reachable over the public internet;
`claude_desktop_config.json` only configures local stdio servers, not remote URLs.

**Project-scoped `.mcp.json`:**

```json
{
  "mcpServers": {
    "kf": {
      "type": "http",
      "url": "http://<server-ip>:8001/mcp"
    }
  }
}
```

`type` is required, and `streamable-http` is accepted as an alias for `http`. An entry
with a `url` but no `type` is read as a stdio server and skipped with an error.

---

## Docker deployment

### Build image

```bash
docker compose build
```

### First-time: build the database

Copy the TEI sources onto the server (the compose file mounts `/home/dh/kf_data` as
`/data`), then:

```bash
docker run --rm -v /home/dh/kf_data:/data kf-mcp python build_db.py --docs /data/kf_raw/docs --registers /data/kf_raw/registers --db /data/kf.db
```

### Run

```bash
docker compose up -d
```

The container serves on port 8001 and expects `kf.db` at `/data/kf.db`. Adjust the volume
path in `docker-compose.yml` if your data lives elsewhere.

### Reverse proxy (nginx)

<a id="reverse-proxy-nginx"></a>

Serving under a sub-path (`https://tei.example.ch/mcp/kf/mcp`) has exactly one rule:
**the app's `--http-path` and the nginx `location` must be the same string.** The
endpoint is one path that answers `POST` (requests), `GET` (the server→client stream),
and `DELETE` (session teardown); it builds no URLs of its own, so all nginx has to do is
forward the path unchanged.

```nginx
server {
    listen 443 ssl;
    server_name tei.example.ch;

    # KF_HTTP_PATH=/mcp/kf/mcp — same string, no trailing slash on proxy_pass,
    # so the path reaches the app unrewritten.
    location /mcp/kf/mcp {
        proxy_pass         http://127.0.0.1:8001;
        proxy_http_version 1.1;
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        # The GET stream must not be buffered or timed out mid-session.
        proxy_set_header   Connection '';
        proxy_buffering    off;
        proxy_cache        off;
        proxy_read_timeout 3600s;
        chunked_transfer_encoding on;
    }
}
```

Two failure modes worth knowing, both of which return a bare `Not Found` or `405`:

- **A trailing slash on `proxy_pass`** (`http://127.0.0.1:8001/`) strips the location
  prefix, so the app sees `/` and no route matches.
- **`location` and `--http-path` disagree** — the app 404s every request. Check the
  startup line, which prints the exact path being served:
  `Starting KF MCP server on 0.0.0.0:8001/mcp/kf/mcp`.

Verify from outside before wiring up a client:

```bash
curl -sS -o /dev/null -w '%{http_code}\n' -X POST https://tei.example.ch/mcp/kf/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'
```

`200` means the endpoint is live. `404` is a path mismatch, `405` means nginx is not
passing `POST` to the app (a static `location` or a `limit_except` is shadowing it).

> **Note:** the server has no authentication. By default `docker-compose.yml` publishes
> port 8001 on all interfaces; if a proxy fronts it, bind it to loopback instead so the
> corpus is not reachable directly:
>
> ```bash
> KF_BIND=127.0.0.1 docker compose up -d
> ```
>
> Otherwise restrict access at the firewall.

---

## Available tools

| Tool | Description |
|------|-------------|
| `corpus_stats()` | Entry/span/person/place/org counts and year range |
| `list_entries(limit=50, offset=0)` | Paginated list of register entries, ordered by year |
| `get_entry(entry_id)` | Full entry: title, year, source, pages, transcription, all spans |
| `search_persons(query, limit=50)` | Person authority file by name (substring match) |
| `get_person(pid)` | Person record with occupation, life dates, and mentions (`hls_id` always null) |
| `search_places(query, limit=50)` | Place authority file by name (German or French) |
| `get_place(pid)` | Place record with mentions, and geo, HLS id, GND id where present |
| `search_orgs(query, limit=50)` | Organisation authority file by name or description |
| `search_fulltext(query, limit=20)` | Full-text search over transcriptions, with snippets |
| `get_entries_for_person(pid, limit=50)` | Entries mentioning a person, by authority id |
| `get_entries_for_place(pid, limit=50)` | Entries mentioning a place, by authority id |
| `get_entries_by_year(year_from, year_to, limit=100)` | Entries in a year range (max span 300 years) |

## Available resources

| URI | Description |
|-----|-------------|
| `kf://stats` | Corpus statistics (JSON) |
| `kf://persons` | Person index — `{total, returned, truncated, persons: [...]}`, capped at 1000 rows and flagged when truncated |
| `kf://entry/{entry_id}` | Single entry (JSON) |

## Query behaviour

**Limits.** Every `limit` is clamped to at most 500; a negative, zero, or non-numeric
value falls back to that tool's own default rather than returning the whole table. Use
`list_entries(limit, offset)` to page through the full corpus.

**Result size.** Claude.ai and Claude Desktop truncate a tool or resource result at
roughly 150,000 characters. `kf://persons` is capped at 1000 rows (about 100 KB) for
that reason and reports its own truncation; the 500-row tool ceiling stays comfortably
under the limit too.

**Full-text search.** `search_fulltext` passes the query to FTS5, so operators work —
`Brugg OR Königsfelden`, `Heinr*`, `NEAR(...)`. If the query isn't valid FTS5 syntax
(a stray quote, a dangling `AND`), it silently falls back to a literal word search
instead of erroring. Only a query with no usable words returns `{"error": ...}`.

**Name search.** `search_persons`, `search_places`, and `search_orgs` do a plain
case-insensitive substring match. SQL wildcards in the query are escaped, so searching
for `100%` finds a literal "100%" rather than matching every record.

**Missing records.** `get_entry`, `get_person`, and `get_place` return
`{"error": "... not found."}` rather than raising.

**Spans.** `get_entry` returns every span in the entry — `persName`, `placeName`,
`orgName`, `date`, `measure`. The `ref` field holds the authority id and is empty for
unlinked mentions and for dates/measures; `norm` holds the normalised `@when` or
`@quantity` value.

**How entry years are assigned.** The year comes from the first `<date when="...">` in the
document `<body>`. If the body has no date, the `<sourceDesc>` in the header is used as a
fallback. Dates in `publicationStmt` or `revisionDesc` are never used — they describe the
edition, not the charter. Years outside 1000–1800 are ignored, and entries with no usable
date have `year = NULL`.

## Database schema

| Table | Contents |
|-------|----------|
| `entries` | id, title, short_id, year, source, pages, text_raw |
| `spans` | entry_id, span_id, class, ref, text, norm |
| `persons` | id, forename, surname, full_name, main_name, occupation, birth, death, org_ref, hls_id, note |
| `places` | id, name_de, name_fr, country, region, geo, hls_id, gnd_id, place_type |
| `orgs` | id, name, desc_de, desc_fr |
| `fts_entries`, `fts_spans` | FTS5 indexes (external content, populated by AFTER INSERT triggers at build time — there are no update/delete triggers, which is why a rebuild clears and repopulates) |
| `chunks`, `embeddings`, `embedding_runs` | the semantic index (`vectorstore.SCHEMA_SQL`, shared): passages keyed `parent_id` = entries.id, one vector per passage, one row per `embed_db.py` run |

## Deployment

This server runs on `tei.dh.unibe.ch` at
**`https://tei.dh.unibe.ch/mcp/kf/mcp`**, alongside four sibling MCP servers:
[SSRQ](https://github.com/thodel/ssrq_mcp), [HLS](https://github.com/thodel/hls_mcp), [HBLS](https://github.com/thodel/hbls_mcp), [EOS / HGB Basel](https://github.com/thodel/eos_mcp).

What they share — the nginx routing, the landing pages, and the deploy sequence —
lives in **[tei_mcp_ops](https://github.com/thodel/tei_mcp_ops)**. Start there for
anything that spans the fleet; in particular, the app's `--http-path` and the nginx
`location` have to be the same string, which is the rule a sub-path deployment turns
on.

### Where the process actually runs (since 30.09.2026)

The public URL is tei's, the process is not. nginx on tei forwards `/mcp/kf/mcp` to
`127.0.0.1:8001`, which is an SSH tunnel (`mcp-tunnel-nginx@8001.service` on tei) to
**idhefix** (`srv`), where the server runs without Docker as a systemd **user** unit of
`tobias`:

| | on idhefix |
|---|---|
| unit | `mcp-kf.service` (`systemctl --user …`) |
| process | `~/mcp/venv/bin/python server.py --db ~/mcp/data/kf.db --host 172.17.0.1 --port 8001 --http-path /mcp/kf/mcp` |
| checkout | `~/mcp/kf_mcp` (this repository, `main`) |
| venv | `~/mcp/venv`, shared with the sibling servers |
| database | `~/mcp/data/kf.db` |
| TEI sources | `~/mcp/data/koenigsfelden-data`, a clone of [Staatsarchiv-Aargau/koenigsfelden-data](https://github.com/Staatsarchiv-Aargau/koenigsfelden-data) at `32f6d17` |

The Docker path above is the generic one; it is not what serves tei. In particular,
`/home/dh/kf_data/kf.db` on tei is a leftover from before the move and is read by
nothing — a rebuild there changes nothing anyone sees (learned 16.11.2026, when the
first rebuild for #15 went into that file).

**Rebuilding the database after a change to `build_db.py` or to the TEI**, on
idhefix as `tobias` (about 15 s; the server reads the file per request, so stopping it
is a courtesy, not a requirement):

```bash
cd ~/mcp/kf_mcp && git pull --ff-only
systemctl --user stop mcp-kf.service
cp ~/mcp/data/kf.db ~/mcp/data/kf.db.bak-$(date +%F)
~/mcp/venv/bin/python build_db.py \
  --docs ~/mcp/data/koenigsfelden-data/data/docs \
  --registers ~/mcp/data/koenigsfelden-data/data/registers \
  --db ~/mcp/data/kf.db
systemctl --user start mcp-kf.service && systemctl --user is-active mcp-kf.service
```

Expect `Existing database: clearing 1,557 entries and rebuilding from source` and no
`WARNING: … skipped`. Check from any MCP client: `get_entry("U-17_0828_02")` starts
`Staatsarchiv AARGAU Unnsz₎ Unnszer andaͣchtig …` (two whole layers, #15), and
`search_fulltext("hofmeister", limit=500)` returns well over 100 entries. To move to a
newer edition commit, `git -C ~/mcp/data/koenigsfelden-data checkout <sha>` first and
note the sha in the table above.

`build_db.py` leaves the `chunks` and `embeddings` tables alone, so semantic search
keeps the passages of the previous build until `embed_db.py` runs again (see
[Semantic search](#semantic-search)); `reuse_vectors` keeps the vectors whose chunk
text did not change.

## Tests

```bash
pip install -r requirements-dev.txt
```

```bash
pytest test_kf_mcp.py
```

Unit tests (TEI parsing, authority registers, rebuild idempotency) run with no setup.
The DB and server tests skip unless you point them at a built database and a running
server:

```bash
KF_DB=/home/dh/kf_data/kf.db KF_SERVER=http://localhost:8001 pytest test_kf_mcp.py
```

The suite also runs standalone, with grouped output and a non-zero exit on failure:

```bash
python test_kf_mcp.py --unit --db /home/dh/kf_data/kf.db --server http://localhost:8001
```

Note that the DB tests assert corpus-size floors (≥1550 entries, ≥5000 persons, ≥1300
places, ≥2000 orgs) — they will fail against a small sample database.

## Semantic search

`search_fulltext` finds entries containing the words you typed. `search_semantic`
finds passages that *mean* what you asked — which matters more here than in a modern
corpus. The transcriptions are 14th–17th century Alemannic and Latin, so a question
like *"wer hat den Hof Lind gepachtet?"* shares almost no surface forms with
*"wellchen der hof Lind soll hingelichen werden"*. Keyword search reaches this
material only if you already know how it was spelled.

### The expanded reading

The edition transcribes diplomatically: segment boundaries are marked `✳` (82,062 of
them), and every abbreviated word appears as the raw manuscript form immediately
followed by the editor's expansion.

| transcription | reading |
|---|---|
| `un̄ und` | und — the raw form is a whole word |
| `Hein r₎ rich` | Heinrich — the raw form continues a fragment |
| `stif tˀin terin` | stifterin |
| `Diz ist dˀ der` | Diz ist der |

Embedded as written, this is close to unusable: names arrive split in half, every
abbreviated word is doubled, and the segment markers punctuate the text at random.
**Passages therefore hold the expanded reading**, not the raw transcription.

Whether an expansion joins the token before it depends on whether that token is
already a word — "ist" is, "Hein" and "stif" are not. Counting occurrences does not
separate them, because "Hein" recurs constantly (the name is abbreviated the same way
every time), and neither does a rare/common threshold, because "Hein" is also a name.
What does separate them is the *ratio* of fragment to standalone occurrences, which
the corpus supplies itself — better than a word list for a language with no settled
orthography.

`search_fulltext` still searches the raw transcription, so an exact historical string
remains findable.

### Building the index

```bash
GPUSTACK_API_KEY=... python embed_db.py             # whole corpus
GPUSTACK_API_KEY=... python embed_db.py --limit 100 # trial run on a sample
GPUSTACK_API_KEY=... python embed_db.py --recompute # after changing the model
python embed_db.py --status                         # coverage, nothing embedded
```

`embed_db.py`, `vectorstore.py` and `embeddings.py` are the fleet's shared files,
vendored byte-identical from [eperiodica_mcp](https://github.com/thodel/eperiodica_mcp)
(ch-h-bot `scripts/check_vendored_drift.py` checks the checksums). Everything that
is Königsfelden's lives in `corpus.py`: the expanded reading (`clean`), the head line
on every passage (`head_line`), and `iter_documents`, which builds the expansion
vocabulary over the whole corpus before the first entry is yielded.

Entries are windowed into ~1000-character passages (150 overlap), each with its
title and year on the first line — a charter passage otherwise carries no trace of
which document or century it belongs to, and "the farm at Lind" reads the same in
1360 as in 1644. The head is for the vector: a `search_semantic` hit carries title
and year as fields and its `snippet` is the passage without that line.
Embedding uses `qwen3-embedding-4b` on GPUStack (2560 dimensions; 0.6b until
24.10.2026, ch-h-bot#570); vectors are
stored L2-normalised as float32 BLOBs, so search is one exact matrix multiply.

Runs are resumable — chunks already embedded with the same model are skipped, each
batch is committed — and recorded in `embedding_runs` with model, dimensions and
window settings.

### An index built before #11

Until thodel/kf_mcp#11 this repo had its own `embed_db.py`, and its index names the
parent column `chunks.entry_id` and the run counter `embedding_runs.n_articles`; the
shared files say `parent_id` and `n_parents`. The passages themselves are the same
(all 6,621 measured row-identical, ch-h-bot#555), so the move is a rename, not a
re-embed. With the server stopped and a backup made:

```bash
cp kf.db kf.db.pre-parent-id
python migrate_parent_id.py --db kf.db       # one transaction, then integrity_check
python embed_db.py --db kf.db --status       # n_chunks unchanged, nothing pending
```

The script refuses a database that is already migrated. Code and rename have to be
deployed together: the old `db.py` asks for `entry_id` and answers every
`search_semantic` on a renamed database with «no such column».

### Query-time requirements

The server embeds the incoming query, so it needs `GPUSTACK_API_KEY` at runtime even
though passage vectors are already in the database. GPUStack is reachable only from
inside the UniBE network; from outside it returns **403 before checking the key**, so
a 403 means the wrong network, not a bad credential. Without a key the other tools
work normally and `search_semantic` returns an explanatory error.