# bbaw-dse-mcp

An [MCP server](https://modelcontextprotocol.io/) for dialogical, exploratory access to digital scholarly editions.

> **DHd2026 Poster**: *Agenten im Dienst der Edition*

## Overview

Instead of complex search forms and navigation structures, this MCP server enables **natural-language conversation** with digital scholarly editions. An AI agent translates questions like *"Who was important for Schleiermacher in 1810?"* into structured database queries — and maintains conversational context across follow-up questions.

The current server composes edition backends and reusable external research
services into a single MCP endpoint using [FastMCP](https://gofastmcp.com/):

| Service | Prefix | Backend | Content |
|---|---|---|---|
| [schleiermacher digital](https://schleiermacher-digital.de) | `sd_` | Public TEI + native search/register pages | Letters, diaries, lectures, chronology, glossary |
| [Praktiken der Monarchie](https://actaborussica.bbaw.de) | `mop_` | Public TEI/HTML + GeoJSON/Zotero | Documents on Prussian monarchy and governance |
| [Jean Paul Briefwechsel](https://www.jeanpaul-edition.de) | `jp_` | HTTPS connector | Letters, Umfeldbriefe, topics, correspondence networks |
| [edition humboldt digital](https://edition-humboldt.de) | `ehd_` | Public TEI API v2.3 + native web modules | Letters, travel journals, thematic documents, research, chronology, registers |
| [correspSearch](https://correspsearch.net) | `cs_` | REST API | Cross-edition correspondence network search |
| Authority services | `authority_` | Lobid GND, GeoNames, Wikidata APIs | Reusable identifier and entity lookup |

## Available Tools

The composed server exposes these shared and edition-native tool surfaces:

| Surface | Description |
|---|---|
| Shared | Cross-edition access, failure-isolated search, correspondence statistics, capabilities, and identity comparison |
| Schleiermacher Digital (`sd_*`) | Bounded advanced search, glossary, passages, diaries, chronology, thesaurus, and Bible references |
| Praktiken der Monarchie (`mop_*`) | Passages, research literature, biogrammes, adjutant journals, residential topography, court calendar, organigrams, thematic dossiers |
| Jean Paul Briefwechsel (`jp_*`) | Native full-text and letter-metadata filters, controlled topics, and curated correspondence networks |
| edition humboldt digital (`ehd_*`) | TEI catalogs, native search, correspondent/year filters, thematic documents, complete passages, chronology and place-event links |
| correspSearch (`cs_*`) | Cross-edition correspondence metadata, normalized relations, typed networks, and edition discovery |
| Authority services (`authority_*`) | GND, GeoNames, and Wikidata lookup |

The full signatures and usage boundaries are documented in
[docs/TOOLS.md](docs/TOOLS.md); the source-code-derived edition-module
inventory is in [docs/EDITION_MODULES.md](docs/EDITION_MODULES.md).

Every successful `sd_*`, `mop_*`, `jp_*`, `ehd_*`, or `cs_*` response, including
shared calls that resolve edition data, carries MCP `_meta.provenance` with
consulted sources, configured backends, datasets, and UTC retrieval time.
Humboldt preserves the actual release asserted in returned TEI/canonical
resource URLs (for example `v12`); HTML searches without release evidence report
`release_status="unknown"`. The other connectors report `data_release=null`
where no coherent source release exists. API/software versions are never
substituted for data releases.

The capability tool, `get_capabilities(check_health=False)`, reports SD, MoP,
JP, Humboldt, correspSearch, Lobid GND, Wikidata, and GeoNames without exposing
credential values. It declares datasets, modules, filters, releases/interface
versions, licenses, and explicit source limitations. Its default is network-free;
`check_health=True` runs isolated read-only probes for configured providers. A missing GeoNames
username is visible as `configured=false`, `available=false`, and
`status="missing_credentials"`.

`search_documents(...)` and `search_entities(...)` return typed pages with
`results`, offset continuation, per-source counts, failures and filter handling,
rather than bare lists. Both accept `strict_filters=False`; strict mode rejects
an affected source rather than silently ignoring unsupported filters. Document
search defaults to `include_commentary=None` (no explicit commentary filter).
Even when every selected backend fails, the structured failure envelope is returned.
`total_count` is exact only when `total_known=true`; otherwise it is an
explicit lower bound. `results_omitted`, `extraction_complete`, and text
`TextExtent.truncated` describe different limits; previews carry executable
complete-content routes. SD uses stable source order for repeatable pages.
This is a clean response-contract cutover, not a compatibility shim; see
[shared contracts and signatures](docs/TOOLS.md#shared-page-text-and-xml-contracts).

`get_entity("sd:…" | "mop:…" | "jp:…" | "ehd:…", include_mentions=True)`
now reads the editions' public register pages without a database account.
Edition-specific parsers preserve native metadata, digital text/commentary
mentions, correspondence/authorship, print references and relationships where
exposed. `mentions.references` is one stable, globally paginated row stream;
`mentions.sections` retains the full section inventory on every page.
`total_count` counts rendered rows, not distinct documents or corpus occurrences.
Absent sections and unresolved native destinations remain explicit.
Published inventories are authoritative within their declared source scope:
`corpus_complete=true` means complete extraction, even when the returned window
has more rows. Unavailable surfaces and actual extraction failures remain
incomplete; published-reference counts are not historical corpus totals.
Shared search, normalized relations and SD/MoP specialist tools now also use
public sources, without a database account or hidden database fallback.

Public retrieval covers every register published in the four editions:
SD's four classic registers plus glossary, Bible and aesthetics thesaurus;
MoP's six classic registers plus bibliography; JP's person/place/two work
registers plus topics and correspondence circles; Humboldt's person/place/
institution/plant/glossary/bibliography/sigle registers. Its drawings gallery
has no published entity IDs and is paged by `ehd_list_drawings`.
Native types and edition namespaces remain intact. Use public specialist
discovery for SD's extra registers; the manifest separates searchable
`entity_types` from `entity_retrieval_types`.

`get_resource_relations(ref=...)` exposes one common, evidence-backed relation
model for Schleiermacher, MoP, Jean Paul, and Humboldt document resources,
including the explicit Jean-Paul module refs `jp:topic:<id>` and `jp:network:<id>`. Its resource kinds
and predicates are closed vocabularies, and every predicate validates the
allowed subject and object kinds. Relations retain provider-scoped references
and external identifiers; the server does not silently merge resources into a
cross-edition `sameAs` identity. The older `get_relations(...)` remains
available for its established edition-specific aggregate response shapes.

`compare_resource_identities(...)` performs that reconciliation explicitly.
It normalizes GND, GeoNames, Wikidata, VIAF, and source URLs, retaining
field-level evidence, conflicts, and partial failures. Only an exact authority
identifier or explicitly equal source URI confirms `same_as`; equal labels
produce an unconfirmed `candidate` and never merge records.

Schleiermacher correspondence relations are atomic sender-to-recipient edges
backed by individual letters. MoP relation extraction reads the complete
register-mention set and distinguishes explicit Hofkalendarium participants
from mere mentions. The query-oriented correspSearch equivalent,
`cs_search_correspondence_relations(...)`, retains every sent/received action
with all participants, places, and date assertions from each fetched record.

**Breaking correspSearch response migration:** `cs_search_correspondences`
records now expose correspondence data only through `letters[].actions[]`.
Select actions by `type` (`sent` or `received`) and read every `participants`,
`places` and `dates` entry. The singular `sender`, `receiver`, `send_place`,
`receive_place` and letter-level `date*` fields have been removed, without
aliases or fallback synthesis. `title` is display text, not structured evidence.
Tool names and inputs are unchanged; see the [migration mapping](docs/TOOLS.md#correspsearch-response-migration).

correspSearch metadata searches may start from any supported substantive
filter; the API defaults to ten records but accepts bounded page sizes up to
1,000. Normalized relation responses deliberately process one upstream page
and expose further pages through `upstream_continuations`.

Jean Paul's native `jp_search_documents_advanced` and `jp_filter_letters`
surfaces expose all three letter corpora with exact totals, stable offset
continuation, concrete source collections, and original citation routes. The
first adds full-text, lemma, sender/receiver, year, and counted-facet access;
the second combines correspondence metadata including co-senders,
co-recipients, topics, joint letters, and year bounds. Both fail explicitly on
unexpected source markup. Shared JP work lookup also covers the separate
`werke-jp` register and retains that collection in search and detail results.

Humboldt uses the conservative `ehd` namespace (edition humboldt digital).
`search_documents(..., editions=["ehd"])`, `get_document("ehd:H1055854")`,
`search_entities(..., editions=["ehd"])`, and `get_entity("ehd:H0012069")`
use the same shared adapter/identity surface as the other editions. Native
tools expose exact local TEI-catalog pages and the site's search/filter pages;
`ehd_get_document_passages` reads complete blocks rather than truncating text.
Use `ehd_get_chronology_entry("H0005699")` for first-party events and
`ehd_get_place_chronology("H0000531")` for source-supplied geographic links.
The API's `2.3` interface is distinct from edition `v12` (8 June 2026) and
GitHub/Zenodo dataset release identifiers. Entity retrieval primarily uses
public HTML, supplemented by the unversioned ID API where needed because
versioned register XML is defective upstream. Measures such as `reaumur`
remain public-TEI-retrievable, with references explicitly `not_exposed`;
production HTML publication has not been established.
Plants use publisher-local selectors (`ehd:plant:Abies`), bibliography uses
group-qualified Zotero keys (`ehd:zotero-667230-R7AXEBAP`), and
`ehd_list_drawings` preserves image crops and source locators without inventing
drawing IDs. Shared date-range/commentary search filters, aggregate networks,
and inferred coordinate or topic graphs are not advertised. See [docs/TOOLS.md](docs/TOOLS.md) for
endpoint research, implementation choices, and exact native signatures.

MoP's public full-text search covers `Druck`, `Adjutantenjournale` and
`Hofkalendarium`; semantic `letter` also includes the published letters in
`Druck`. Private `Korrespondenz`/`Instruktionen` selectors and separate
instruction classification are not established by the public search and are
reported as unsupported, not guessed from titles or returned as false zeros.
`mop_get_document_passages` provides division/page-filtered text blocks;
shared `get_document` now returns complete public text or TEI. The current source-gap and acceptance records are
in [docs/DATA_SOURCE_GAPS.md](docs/DATA_SOURCE_GAPS.md) and
[docs/MOP_ACCEPTANCE.md](docs/MOP_ACCEPTANCE.md).

## Installation

```bash
# Clone the repository
git clone https://github.com/telota/bbaw-dse-mcp.git
cd bbaw-dse-mcp

# Install with uv (recommended)
uv sync
```

Requires Python 3.11+.

## Public document retrieval

`get_document` uses the editions' public TEI exports for `sd:`, `mop:`, `jp:`,
and `ehd:` references, without database credentials. The internal HTTP receiver
shares bounded downloads and connection handling; edition adapters retain
their own identifiers, catalogs, parsing, and namespaces.

Authority normalization and TEI correspondence-action parsing are shared
primitives, not parallel edition implementations. Original URI/XML evidence,
multiple participants and uncertain dates are retained; edition adapters still
choose their own metadata precedence and resolve their own local IDs. Distinct
authority IDs are not silently collapsed, and VIAF/GeoNames are not substituted
for GND.

MoP enables optional normal TELOTA Gatekeeper verification: wait for the issued
challenge, submit it to `/verify-request`, retain the cookie, and retry once.
No browser, forged token, database fallback, or alternate snapshot is used.
Downloads exceeding 16 MiB fail explicitly; XML and document text are not
silently truncated. Facsimile-only documents report availability rather than
automatically generating OCR.

This public-only cutover also covers searches, registers, relations and
specialists. Biograms retain family, offices, education and awards; journals
retain dated sessions and explicit author/place identities; organigrams retain
hierarchies and dated holders. Calendar search uses counted native pages plus
local exact-day/text filtering. Catalog-declared calendar UUIDs use public HTML;
their unavailable TEI view is an explicit publisher gap.

SD person networks use person-scoped native discovery and letter TEI.
Native whole-corpus correspondent statistics require the publisher's bulk CMIF,
currently unavailable (HTTP 504); this is an error, not an empty network.
`get_correspondence_statistics(editions=["sd", "jp", "ehd"])` provides an explicit
correspSearch-backed alternative: live indexed-record counts, or sent/received
counts and bounded partner rankings with `ref="sd:S0003676"` or `person_gnd`.
Exact counts remain separate from incomplete partner analysis and corpus coverage.
JP includes print-only An-Briefe metadata. MoP returns `unsupported` with null
counts because no source association has been verified; native tools stay intact.
AB topography uses the five publisher-linked GeoJSON datasets; literature uses
the live public Zotero group, not a fabricated dated snapshot. Journal text
queries hydrate only their native result window; person/place-only searches
without text/date/monarch narrowing can require a bounded, cold-cache scan.

The obsolete private eXist clients, database/query tools, SD legacy parsers and
exclusive database settings have been removed. Existing `.env` files must remove
the old SD/AB database URLs, collection paths, usernames/passwords and `SD_LOCAL`
keys. Use `.env.example` for the supported public configuration; the GeoNames
account remains an external-service setting. No database compatibility aliases
or hidden fallback are retained.

## Configuration

Configuration uses environment variables (prefix `EDITIONS_`). Create a `.env` file:

```env
# Schleiermacher Digital public research interfaces; no database account
EDITIONS_SD_PUBLIC_URL=https://schleiermacher-digital.de

# Praktiken der Monarchie public research interfaces; no database account
EDITIONS_AB_PUBLIC_URL=https://actaborussica.bbaw.de
EDITIONS_AB_GATEKEEPER_ENABLED=true

# correspSearch API (no auth required)
EDITIONS_CS_API_URL=https://correspsearch.net/api/v2.0
EDITIONS_CS_SPARQL_URL=https://lod.academy/correspsearch/data/main/sparql

# Required for the GeoNames lookup tools; capability discovery works without it
EDITIONS_GEONAMES_USERNAME=

```

## Usage

### Running the server

```bash
uv run bbaw-dse-mcp
```

The default transport remains **stdio**, including for Claude Desktop. The
server now uses FastMCP 4.0.11+ and the official MCP Python SDK 2.3.0+, which
natively implement the published
[MCP 2026-07-28 specification](https://modelcontextprotocol.io/specification/2026-07-28).
Legacy clients still use their `initialize` handshake on the same server;
modern clients do not initialize or create protocol sessions.

#### Streamable HTTP deployment

```bash
uv run bbaw-dse-mcp --transport http --host 127.0.0.1 --port 8000
# Equivalent transport spelling: --transport streamable-http
```

The endpoint is `/mcp` (`--path` overrides it). Host and Origin protection is
enabled even when binding to a non-loopback address. Behind a TLS reverse
proxy, explicitly allow the externally visible hostname:

```bash
uv run bbaw-dse-mcp --transport http --host 127.0.0.1 --port 8000 \
  --allowed-host editions.example.org
```

Repeat `--allowed-host` for additional trusted hostnames. `--allowed-origin`
allows an explicit browser origin; it does not configure CORS. Preserve the
original `Host` and `Origin` headers through the proxy, disable buffering for
SSE responses, and configure authentication/access restrictions at your
deployment boundary before exposing the endpoint publicly. The repository
does not configure an OAuth authorization server or issue access tokens.
Modern requests need no worker affinity; legacy stateful HTTP clients still
need session affinity when using multiple workers.

The deprecated HTTP+SSE transport remains available for legacy integrations
with `--transport sse` (default path `/sse`). New integrations should use
Streamable HTTP; the modern transport has no standalone GET stream, session
DELETE, or resumable SSE event IDs.

For ASGI embedding, preserve the FastMCP lifespan and the body-primary
validation guard:

```python
from bbaw_dse_mcp.main import app
from bbaw_dse_mcp.protocol import modern_http_middleware

http_app = app.http_app(
    path="/mcp",
    middleware=[modern_http_middleware("/mcp")],
    host_origin_protection=True,
    allowed_hosts=["editions.example.org"],
)
# Mount http_app with its lifespan, not just its routes.
```

Before release, run `uv run python scripts/verify_all_tools.py` against the local
composition, or add `--url https://your-host/mcp` to verify a deployed endpoint.
The [release verification guide](tests/README.md#release-verification) documents
coverage, semantic checks, expected limitations, JSON reports and failure exit
status. Scoped `--tool` runs are not full release approval.

#### Modern protocol and raw smoke requests

Every modern request includes `params._meta` with
`io.modelcontextprotocol/protocolVersion` and
`io.modelcontextprotocol/clientCapabilities` (an empty object is valid).
`io.modelcontextprotocol/clientInfo` is recommended but optional. Requests
are independently negotiated and can call tools before discovery.
`server/discover` advertises protocol versions, capabilities, instructions,
and server identity; ordinary results include `resultType: "complete"`.
Cacheable discovery/list/resource results include `ttlMs` and `cacheScope`.
Tool diagnostics use standard Python module logging (stderr on stdio), not
the deprecated MCP logging capability. They do not require client logging
opt-in or emit unsolicited protocol log notifications.

On HTTP, include `Accept: application/json, text/event-stream`,
`MCP-Protocol-Version`, and `Mcp-Method` on every request. `tools/call`,
`prompts/get`, and `resources/read` additionally require `Mcp-Name` matching
the body name or URI. Missing/mismatched routing headers return HTTP 400
with `HeaderMismatch` (`-32020`); missing required metadata returns
`-32602`; unsupported versions return `-32022` with `supported` and
`requested`. Unknown or removed modern methods (including `ping`) return
HTTP 404 and `-32601`. Header-safe values and Base64-sentinel names are
validated by the SDK.

With the HTTP server running, these network-free calls exercise discovery,
tool enumeration, and a real capability-tool execution:

```bash
META='{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientCapabilities":{}}'
for METHOD in server/discover tools/list; do
  curl -sS http://127.0.0.1:8000/mcp \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json, text/event-stream' \
    -H 'MCP-Protocol-Version: 2026-07-28' -H "Mcp-Method: $METHOD" \
    --data "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"$METHOD\",\"params\":{\"_meta\":$META}}"
done
curl -sS http://127.0.0.1:8000/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'MCP-Protocol-Version: 2026-07-28' \
  -H 'Mcp-Method: tools/call' -H 'Mcp-Name: get_capabilities' \
  --data "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"_meta\":$META,\"name\":\"get_capabilities\",\"arguments\":{\"check_health\":false}}}"
```

The default HTTP response is request-scoped SSE, so JSON-RPC messages appear
on `data:` lines. To check rejection, repeat the last request with
`Mcp-Name: wrong_name`; expect HTTP 400 and code `-32020`. To check
negotiation, set **both** the body version and version header to
`2099-01-01`; expect HTTP 400 and `-32022`, not a legacy fallback.

On stdio, write JSON bodies one per line without HTTP headers and keep stdin
open until responses arrive (closing it is the graceful shutdown signal):

```bash
uv run python - <<'PY'
import asyncio
import json
import sys

async def smoke():
    proc = await asyncio.create_subprocess_exec(
        sys.executable, "-m", "bbaw_dse_mcp.main",
        stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE,
        limit=8 * 1024 * 1024,
    )
    meta = {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientCapabilities": {},
    }
    try:
        calls = [
            ("server/discover", {}),
            ("tools/list", {}),
            ("tools/call", {"name": "get_capabilities", "arguments": {}}),
        ]
        for ident, (method, args) in enumerate(calls, 1):
            body = {"jsonrpc": "2.0", "id": ident, "method": method,
                    "params": {"_meta": meta, **args}}
            proc.stdin.write((json.dumps(body) + "\n").encode())
            await proc.stdin.drain()
            reply = json.loads(await asyncio.wait_for(proc.stdout.readline(), 30))
            assert reply["id"] == ident and reply["result"]["resultType"] == "complete"
            print(json.dumps(reply))
        proc.stdin.close()
        await asyncio.wait_for(proc.wait(), 10)
        assert proc.returncode == 0
    finally:
        if proc.returncode is None:
            proc.kill()
            await proc.wait()

asyncio.run(smoke())
PY
```

`tests/contract/test_protocol.py` covers raw HTTP and subprocess stdio,
per-request errors, cache metadata, streaming-body replay, DNS-rebinding
protection, and actual FastMCP modern/legacy clients. Run with
`uv run --extra dev pytest tests/contract/test_protocol.py --no-cov`.

Upstream boundaries: SDK v2 supports both protocol eras, MRTR and
subscriptions, but core experimental tasks have moved to an optional
extension; this server does not register that extension or require client
sampling/roots/elicitation. Its edition tools return complete results.
A narrow ASGI guard compensates for SDK 2.3.0's header-only era router so
modern bodies cannot evade header validation by omitting the version header
or supplying a legacy one. It buffers/replays only MCP POST bodies, bounded
to 8 MiB and 4,096 chunks (HTTP 413 above those limits); Host/Origin checks
remain in FastMCP's outer middleware. See the
[official SDK v2 changes](https://py.sdk.modelcontextprotocol.io/whats-new/),
[FastMCP 4 migration guide](https://gofastmcp.com/getting-started/upgrading/from-fastmcp-3),
and [HTTP transport requirements](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/streamable-http).

### Prompt templates

Clients with MCP prompt support can start common, source-grounded workflows with:

- `research_correspondence` — investigate a person's correspondence network.
- `analyze_edition_document` — retrieve and analyze a known SD or MoP document.
- `deep_research` — plan and execute a multi-step cross-edition investigation.

These prompts orchestrate the existing MCP tools in the host model. The server does
not require a separate model-provider API key.

### Claude Desktop

Add to your `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "editions": {
      "command": "uv",
      "args": ["run", "bbaw-dse-mcp"],
      "cwd": "/path/to/bbaw-dse-mcp"
    }
  }
}
```

### Example Conversation

```
User: Search for letters mentioning Humboldt.

Agent: → search_documents(query="Humboldt", editions=["sd"], doc_types=["letter"])
       Inspect the returned native search hits and their source metadata.

User: Who was important for Schleiermacher in 1810?

Agent: → get_correspondence_statistics(editions=["sd"], ref="sd:S0003676",
             date_from="1810", date_to="1810", max_letters_to_analyze=500)
       Report indexed correspondence counts and the bounded partner ranking;
       distinguish frequency from historical importance and index coverage.

User: Can we trace Humboldt's correspondence network beyond this edition?

Agent: → authority_search_gnd(name_query="Humboldt, Wilhelm von")
       → cs_search_correspondent_network(person_gnd="118554727")
       Report the analyzed records, authority identities and completeness limits.
```

## Tech Stack

- **[FastMCP](https://gofastmcp.com/) 4.x / MCP Python SDK 2.x** — Native modern and legacy MCP protocol support
- **[httpx](https://www.python-httpx.org/)** — Async public HTTP APIs and pages
- **[lxml](https://lxml.de/)** — TEI-XML parsing
- **[Pydantic](https://docs.pydantic.dev/) v2** — Data models and settings

## Documentation

- [AGENT.md](./AGENT.md) — Project overview and design decisions
- [docs/TOOLS.md](./docs/TOOLS.md) — Detailed tool specifications
- [docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md) — Technical architecture
- [docs/CODING_GUIDELINES.md](./docs/CODING_GUIDELINES.md) — Extension and testing rules

## License

MIT

## Author

Tim Westphal, [BBAW TELOTA](https://www.bbaw.de/bbaw-digital/telota)