OKF MCP Server

Turn a folder of enterprise documents into cited, trust-aware answers for any MCP-capable AI.

OKF MCP Server converts PDFs, Word, Excel, PowerPoint, HTML and Markdown into Google's Open Knowledge Format (OKF). OKF stores knowledge as plain Markdown files with YAML metadata. The server exposes that knowledge through the Model Context Protocol, so an assistant such as Claude can browse it, search it and quote it, pointing back to the exact page, sheet or section it came from.

Ask "What is the hotel limit in London?" and the assistant finds the current, human-reviewed travel policy: 260 USD per night (travel-policy.md, section "Hotels", lines 9–12). The superseded 2023 policy says 150 USD; it is still findable, but flagged deprecated and ranked lower.

Contents

• Why

• How it works

• Quick start

• Connect an AI client

• Ingest your own documents

• Access control

• Connectors and scheduled refresh

• MCP tools

• Configuration

• Deploy on Kubernetes or OpenShift

• Security model

• Project layout

• Development

• Troubleshooting

• Roadmap

• Acknowledgements

Related MCP server: mcp-rag-bridge

Why

Enterprise knowledge is spread across formats, much of it is out of date, and a model that "sort of remembers" a policy is worse than one that says it does not know. This project keeps three things visible in every answer:

OKF itself is only files: you can cat it, diff it in git, open it in Obsidian, or throw the server away and keep your knowledge.

How it works

flowchart LR
    A["Source folder<br/>PDF · DOCX · XLSX · PPTX<br/>HTML · MD · TXT"] -->|okf-ingest build| B["Extract<br/>Docling / native"]
    B --> C["OKF bundle<br/>+ extraction metadata"]
    C --> D{"Validate<br/>OKF v0.2 §11"}
    D -->|pass| E["Keyword + vector index<br/>→ publish snapshot"]
    D -->|fail| X["Discard;<br/>previous snapshot<br/>keeps serving"]
    E --> F["MCP server<br/>browse · search · get"]
    F <-->|streamable HTTP| G["AI client<br/>Claude Code, …"]

1. Extract. Markdown and text are read natively. Everything else goes through Docling, running locally. Page and sheet provenance is kept.

2. Write OKF. One concept per source file, with frontmatter filled deterministically from file metadata and an optional _okf.yaml. No generative model is involved in ingestion.

3. Validate and publish. The new snapshot must pass conformance checks before an atomic switch makes it live. A failed build never replaces a good snapshot.

4. Serve. Three read-only MCP tools. Search is hybrid: keyword (BM25) and semantic (local embeddings) rankings are fused, so "doctor's note" finds a policy that says "fit note". The AI client composes the answer; the server supplies evidence.

Quick start

Requires Python 3.12+, uv and make.

git clone git@github.com:bsahane/okf-mcp-server.git
cd okf-mcp-server
make install          # venv, dev dependencies, pre-commit hooks; opens a subshell (type `exit` to leave)
make ingest-install   # adds Docling for PDF/Office formats (large download)
okf-ingest fetch-model  # optional: local embedding model for semantic search (~493 MB)
make sample           # builds a snapshot from samples/finance into ./data
make local            # serves http://localhost:5001/mcp

Check it is up:

curl http://localhost:5001/health

The sample corpus is synthetic ("Acme" finance policies). It deliberately includes a deprecated policy that conflicts with the current one, a stale procedure, a Word document with a SKU table and a two-sheet Excel workbook.

Connect an AI client

The server speaks MCP over streamable HTTP at http://localhost:5001/mcp. It only accepts local connections (see Security model), so use a client running on the same machine.

Claude Code

claude mcp add --transport http okf http://localhost:5001/mcp

MCP Inspector, useful to try the tools by hand:

npx @modelcontextprotocol/inspector

Choose the Streamable HTTP transport and enter http://localhost:5001/mcp.

Python (FastMCP client): see examples/fastmcp_client.py.

from fastmcp import Client

async with Client("http://localhost:5001/mcp") as client:
    result = await client.call_tool("search_knowledge", {"query": "hotel limit London"})

Ingest your own documents

okf-ingest build /path/to/documents     # build and publish a new snapshot (-v for Docling/OCR logs)
okf-ingest validate                     # re-check the published bundle
okf-ingest eval questions.yaml          # measure retrieval quality (Hit@5)
okf-ingest suggest-config /path/to/documents > _okf.yaml   # draft metadata to review

Supported formats: .pdf .docx .pptx .xlsx .html .htm .csv (via Docling) and .md .markdown .txt (native). Hidden files are skipped. Unsupported types (archives, diagrams, …) are listed as SKIPPED and recorded in manifest.json, but do not block publishing. Empty files, and scans where OCR finds no text, are failures.

What a build does

• Mirrors your folders. policies/Travel Policy.pdf becomes concept policies/travel-policy. Machine-generated file names (UUIDs, long hex, bare numbers) take their title from the document's first heading instead. Same-name files get their extension appended (budget-pdf, budget-xlsx), and the reserved names index/log get a -doc suffix.

• Reuses work. Files whose SHA-256 is unchanged reuse the previous extraction, so Docling does not run again. Renamed files are detected by content. Deleted files are gone from the new snapshot, and log.md records every creation, update, rename and deletion.

• Keeps citations durable. Locations are stored in extraction/<source-id>.json next to the bundle (including Docling's own JSON), so rebuilding the index never guesses locations from Markdown.

• Bounded disk use. Each build is a full snapshot, so only the newest 3 are kept (--keep N, minimum 2 so the previous one survives). Failed builds delete nothing.

• One build at a time. A second build on the same data directory exits with an error instead of racing the first; leftovers from an interrupted build are cleaned up by the next one.

• Publishes safely. If any file fails to extract, or the bundle fails validation, nothing is published. --allow-failures publishes the rest and writes failures.json.

• Stays inside the source folder. Keep OKF_DATA_DIR outside the source folder; a data directory inside it is rejected so builds cannot ingest their own snapshots. Symlinked files that resolve outside the source folder, and partial Docling conversions, are reported as failures.

Metadata with _okf.yaml

Put an optional _okf.yaml in the source root to set OKF types, lifecycle and review state. Full example: samples/finance/_okf.yaml.

Start from okf-ingest suggest-config <folder>. It prints a draft with a comment on every guess: types from folder and file names, older dated versions (… 2024 next to … 2026) and identical copies marked deprecated, and files named draft kept as drafts. It never reads document content and never marks anything verified; that is for document owners. Builds also warn about identical files until one of them is marked deprecated.

bundle_title: Acme Finance knowledge
default_type: Reference
types:                              # longest matching path prefix wins
  policies/: Policy
  sops/: SOP
documents:
  policies/travel-policy.md:
    status: stable                  # draft (default) | stable | deprecated
    description: Current rules for booking and reimbursing business travel.
    tags: [finance, travel]
    stale_after: 2027-03-31T00:00:00Z
    verified:                       # a human: actor makes it "human-reviewed"
      - { by: "human:finance-owner", at: 2026-09-01T09:00:00Z }
  policies/travel-policy-2023.md:
    status: deprecated

The resulting concept looks like this (abridged):

---
type: Policy
title: Acme Travel Policy (2026)
description: Current rules for booking and reimbursing business travel.
resource: file:///…/policies/travel-policy.md
tags: [finance, travel]
sources:
- id: src-d9ab7f29f6825d9a
  resource: file:///…/policies/travel-policy.md
  title: travel-policy.md
  last_modified: '2026-09-24T02:04:27Z'
generated: { by: okf-ingest/0.1.0, at: '2026-09-24T02:04:35Z' }
verified:
- { by: human:finance-owner, at: '2026-09-01T09:00:00Z' }
status: stable
stale_after: '2027-03-31T00:00:00Z'
source_revision: sha256:a9a99336…
---

## Hotels

The nightly hotel limit is 180 USD in most cities. In London, New York and Tokyo the limit is 260 USD per night.

Measuring retrieval

Write the questions your users actually ask, together with the concepts that answer them. Include questions with no answer.

questions:
  - question: What is the hotel limit per night in London?
    expected: [policies/travel-policy]
  - question: Who approves an expense claim over 1,000 USD?
    expected: [sops/expense-claims, reference/cost-centres]   # multi-source
  - question: What is the parental leave policy?
    expected: []                                               # no answer in the corpus

okf-ingest eval reports Hit@5, notes which multi-source questions were missing evidence, and flags deprecated or stale results. Expected entries can also be source file paths as users know them (HR/Leave Policy 2024.docx). See samples/finance-questions.yaml and the pilot template samples/pilot-template/questions.yaml.

Scanned PDFs and speed

OCR runs only for PDFs where some page has no embedded text layer, which is a real scan. On born-digital PDFs it added no text in the pilot but made conversion 4.5× slower (3.5 s vs 0.8 s per page). The trade-off: text inside figures and diagrams of born-digital PDFs is not OCR'd. Unchanged files are never converted twice: a rebuild of four PDFs (171 pages) took 0.4 s instead of 12 minutes.

Very large embedded images (above about 179 megapixels) are refused by Pillow's decompression-bomb protection and the file fails; around 90 megapixels Pillow only warns.

A reproducible synthetic benchmark lives in samples/northwind.

Semantic search

Search combines two rankings with reciprocal-rank fusion: BM25 keywords, which are exact for codes and names, and a local multilingual embedding model (intfloat/multilingual-e5-small, MIT, 94 languages, pinned revision), which matches meaning ("sign off a 25,000 purchase" finds the approval matrix). Each result reports matched_by: keyword, semantic or both.

• Setup: okf-ingest fetch-model downloads the model once (~493 MB) into the Hugging Face cache. Nothing is downloaded at build or query time, and no text leaves the machine.

• Builds embed every passage (69 passages/s on an Apple GPU, 17/s on CPU) and reuse vectors for unchanged passages, so a rebuild with no changes takes under a second and does not load the model. --no-embeddings builds keyword-only.

• Fallback: without the model or the semantic extra (torch, transformers), builds warn and search is keyword-only. OKF_SEMANTIC_SEARCH=false forces keyword-only. The server image installs no torch, so it is keyword-only unless built with .[semantic] and given the model cache.

• Measured: on the pilot papers, Hit@5 went from 0.81 to 0.94 and all-evidence from 0.69 to 0.81; on Northwind, Hit@1 went from 0.81 to 0.97; nothing got worse. Vector search always returns nearest neighbours, even for questions the corpus cannot answer, so the assistant must still judge relevance.

• Scale: vectors are compared by brute force in memory (about 10 ms for 600 passages). That is fine for tens of thousands of passages; move to pgvector or sqlite-vec beyond that.

Offline and air-gapped use

Docling downloads layout models on its first PDF conversion. To run without network access, pre-download them and set DOCLING_ARTIFACTS_PATH, or pass okf-ingest build --artifacts-path <dir>.

Access control

With ENABLE_AUTH=True every tool call needs a bearer token from your OIDC provider (Keycloak / Red Hat build of Keycloak, Entra ID, Okta, …). The server validates it by introspection and passes the caller's identity to the tools; tool arguments never carry identity.

• Tokens: the audience (aud) must include OKF_REQUIRED_AUDIENCE (403 otherwise); OKF_REQUIRED_SCOPE optionally requires a scope. Groups come from the OKF_GROUPS_CLAIM claim, by full path (Keycloak's /finance/payroll matches group:finance/payroll, not group:payroll). user: rules match the subject, the email only when email_verified is true, and the username unless it looks like someone else's email.

• Documents: each concept carries an access list of principals: group:<name>, user:<email, username or subject>, or * (any signed-in user). It comes from, in order, documents.<file>.access in _okf.yaml, the source system's permissions recorded by a connector, and path-prefix rules:

  access:                     # in _okf.yaml; longest matching prefix wins
    finance/: [group:finance]
    hr/: {groups: [hr], users: [ceo@example.com]}
    public/: ["*"]

  Documents with no list follow OKF_DEFAULT_ACCESS: deny (default) or authenticated.

• Filtering: search filters keyword and semantic candidates before ranking, so hidden documents never take a result slot or leak a snippet; browse hides documents and folders with nothing visible; a denied fetch answers "not found", exactly like a missing one.

• Changes: group membership comes from the token, so removing someone from a group applies to their next token. Document permissions change with the next refresh, so the refresh interval is the maximum permission-sync delay. Revoked sessions fail introspection immediately (401).

• Audit: every tool call is logged as an okf.audit event with the caller, tool, query and returned documents, and appended as JSON lines to OKF_AUDIT_LOG if set. Ingestion adds ingest.sync and ingest.build events.

tests/e2e/auth_e2e.py checks all of this against a real Keycloak in Docker (12 checks: 401/403, per-group visibility, not-found on denial, revocation, group removal, audit).

Connectors and scheduled refresh

Connectors mirror source systems, content and permissions, into a local folder that okf-ingest build reads. Secrets are only read from environment variables.

# connectors.yaml
mirror: /var/lib/okf/mirror
okf_config: /etc/okf/_okf.yaml        # optional build rules, copied into the mirror
sources:
  - name: finance-share               # file share: local, NFS or SMB mount
    kind: fileshare
    path: /mnt/finance
    permissions: posix                # owner/group/mode bits -> principals; or none
  - name: hr-sharepoint               # SharePoint / OneDrive via Microsoft Graph (app-only)
    kind: sharepoint
    tenant_id: <tenant-id>
    client_id: <app-id>
    client_secret_env: OKF_SHAREPOINT_SECRET
    site: contoso.sharepoint.com:/sites/HR
    drive: Documents
    folder: Policies
  - name: ops-drive                   # Google Drive v3, service account
    kind: gdrive
    service_account_key_env: OKF_GDRIVE_KEY_FILE
    folder_id: <folder-id>

okf-ingest sync connectors.yaml       # mirror only
okf-ingest refresh connectors.yaml    # sync, then build and publish (schedule this)

• Incremental and exact: unchanged items are not downloaded, deletions are removed, a failed download keeps the last good copy, and a source that fails to list keeps its previous mirror. refresh does not build after a failed sync unless --allow-failures.

• Permissions: file shares map owner, group and mode bits; SharePoint maps users, Entra and SharePoint groups and organization/anonymous links (group_names: id if your tokens carry group IDs); Drive maps users, groups, domain and anyone. Each source can set a default access for items without source permissions.

• Formats: Google Docs, Sheets and Slides are exported as DOCX, XLSX and PPTX.

• Scheduling: on Kubernetes/OpenShift the okf-refresh CronJob runs refresh nightly (concurrencyPolicy: Forbid; builds also lock the data directory). On a server, use cron: 0 2 * * * /opt/okf/.venv/bin/okf-ingest --data-dir /var/lib/okf/data refresh /etc/okf/connectors.yaml.

SharePoint and Google Drive are tested against mock servers that follow the Graph and Drive APIs (paging, throttling, redirects, exports, permissions); they have not yet been run against a live tenant. tests/e2e/connectors_live.py does that read-only with your connectors.yaml, see docs/live-checks.md. File shares are tested on real files and in the Kubernetes end-to-end run.

MCP tools

Example search_knowledge result (trimmed):

{
  "concept_id": "policies/travel-policy",
  "title": "Acme Travel Policy (2026)",
  "type": "Policy",
  "section": "hotels",
  "excerpt": "The nightly hotel limit is 180 USD in most cities. In London, New York and Tokyo the limit is 260 USD per night. …",
  "status": "stable",
  "trust_tier": "human-reviewed",
  "stale": false,
  "source": {
    "uri": "file:///…/finance/policies/travel-policy.md",
    "location": { "lines": [9, 12], "section": "Hotels" },
    "location_text": "section “Hotels”, lines 9–12"
  }
}

Behaviour worth knowing:

• Errors are real MCP errors. Invalid input returns isError: true, and so does a path outside the bundle. An empty search is a successful result with no evidence, so the assistant can say it does not know.

• Search is hybrid and returns one result per section. Exact codes such as SKU-4471 match through keywords, paraphrases through embeddings (see Semantic search), and Unicode math letters and ligatures from PDFs match plain words. search_mode says whether a request ran hybrid or keyword. Measure changes with okf-ingest eval, which accepts concept#section expectations and reports Hit@1, Hit@5, MRR and all-evidence.

• Pagination is snapshot-bound. Once a new snapshot is published, older cursors ask the client to restart from the first page, so pages from two revisions are never mixed.

• Excerpts are capped. Search excerpts are at most 1,500 characters. When excerpt_truncated is true (for example, one oversized table row), read the section with get_knowledge and follow its cursor.

Configuration

Settings come from environment variables or .env. make local creates .env from .env.example if it is missing.

The template's OAuth, SSL and PostgreSQL settings are documented in docs/authentication.md.

Snapshot layout:

data/
├── current -> snapshots/20260924T021553Z-66aefd
└── snapshots/<id>/
    ├── bundle/          # the OKF bundle: concepts, index.md per folder, log.md
    ├── extraction/      # per-source extraction metadata and locations
    ├── index.db         # SQLite FTS5 passage index
    └── manifest.json    # source ID → concept, revision, extractor

Deploy on Kubernetes or OpenShift

docker build -t <registry>/okf-mcp-server:<tag> -f Containerfile .                  # server
docker build -t <registry>/okf-mcp-server-ingest:<tag> --build-arg EXTRAS=ingest -f Containerfile .  # PDF/Office ingestion
kubectl apply -k deployment/kubernetes      # plain Kubernetes (set the image in its kustomization)
make deploy openshift NAMESPACE=<project>   # OpenShift: builds both images in the cluster, ImageStream + Route

The server image is 1.25 GB. The ingest image (3.1 GB) adds Docling with CPU-only PyTorch and headless OpenCV, and bakes Docling's layout, table and OCR models into /app/models, so refresh jobs need no internet and no writable paths beyond /tmp; point the CronJob at it for PDF and Office sources (the OpenShift overlay does, and builds it with a second BuildConfig). Verified: the full Northwind corpus ingests in it air-gapped (--network none), read-only, as an arbitrary UID, with the same 59 concepts as a local build.

deployment/base holds the Deployment, Service, ConfigMap, Secret, PVC and the refresh CronJob with its connectors ConfigMap; the overlays add the OpenShift-only BuildConfig, ImageStream and Route, or a local image for plain Kubernetes. Pods match OpenShift's restricted-v2 SCC: any non-root UID, no privilege escalation, all capabilities dropped, RuntimeDefault seccomp and a read-only root filesystem. Authentication is on by default; fill in the ConfigMap's SSO_* URLs and the Secret, and provision PostgreSQL for the template's OAuth state. The CronJob writes snapshots to the PVC and the server reads them read-only, so a refresh is served without a restart. The PVC can be ReadWriteOnce: the Deployment uses the Recreate strategy and the CronJob has pod affinity to the server's node; with a ReadWriteMany storage class the affinity can be removed.

Verified: tests/e2e/k8s_e2e.py deploys the Kubernetes overlay with Keycloak and PostgreSQL into a throwaway namespace (10 checks: arbitrary UID, read-only root, refresh Job from the CronJob, 401, per-group visibility, refresh without restart, audit); with --ingest-image the CronJob runs the ingest image and converts a Word document with Docling in the cluster, and tests/e2e/validate_manifests.sh validates both overlays strictly, the OpenShift kinds against OpenShift 4.18 schemas (also run in CI). k8s_e2e.py --platform openshift --build runs the same checks on OpenShift (in-cluster builds, restricted-v2 UID range, Route); it is ready but has not yet been run on a live OpenShift cluster, see docs/live-checks.md.

Security model

• Two modes. With ENABLE_AUTH=False (local pilot) the operator is trusted and sees everything, and the server rejects any request whose Host or Origin is not loopback (HTTP 421/403) to stop DNS-rebinding attacks from web pages; FastMCP 2.14.2 does not enable the MCP SDK's own check. With ENABLE_AUTH=True (shared) every call needs a valid token with the right audience, and results are filtered per user; see Access control. Without a validated identity the tools fail closed.

• Confined reads. Tools resolve paths, including symlinks, inside the bundle only, and reject reserved files. Connectors sanitize remote names so nothing escapes the mirror, and file shares follow links only when they stay inside the share. Output is bounded and paginated.

• Evidence, not instructions. Retrieved text and source links are returned as data. The server never fetches URLs or executes anything from the corpus.

• Ingestion is operator-only. AI clients cannot trigger builds, syncs or file changes.

• Known template gap. The template's own /auth/token endpoint issues placeholder tokens. Clients should get tokens from your identity provider; this server only validates them.

Project layout

okf_mcp_server/src/
├── api.py, main.py, mcp.py, settings.py   # template app, plus the Host/Origin guard and tool registration
├── tools/          # browse_knowledge, search_knowledge, get_knowledge (one file each)
├── knowledge/      # bundle reader, FTS5 + vector index, embeddings, snapshots, cursors, validation
├── ingest/         # okf-ingest CLI, pipeline, native extractors, Docling adapter, eval
├── connectors/     # fileshare, SharePoint (Graph), Google Drive; mirror sync; connectors.yaml
└── oauth/, storage/                        # template OAuth: token validation feeds the caller identity
samples/finance/     # synthetic pilot corpus and _okf.yaml
samples/northwind/   # synthetic messy-company benchmark (generator + questions)
samples/gitlab-handbook/  # Phase 1 pilot on a real public HR corpus (fetch script + questions)
tests/               # unit, ingestion, tool, access, connector and protocol tests
tests/e2e/           # Keycloak auth, Kubernetes deployment and manifest validation
deployment/          # base + kubernetes and openshift overlays

Development

make lint             # ruff + mypy
make test             # pytest
make coverage         # 80% minimum, as enforced in CI
pytest -m docling     # Docling adapter tests (needs `make ingest-install`)
make pre-commit       # ruff, ruff-format, mypy, pydocstyle, bandit, file checks
pytest tests/test_http.py  # starts the real server and checks MCP over HTTP
python tests/e2e/auth_e2e.py          # real Keycloak + PostgreSQL in Docker
python tests/e2e/k8s_e2e.py           # deployment on a Kubernetes cluster (default context: orbstack)
python tests/e2e/k8s_e2e.py --ingest-image okf-mcp-server-ingest:dev   # refresh with Docling, incl. a DOCX
tests/e2e/validate_manifests.sh       # kubeconform, strict, incl. OpenShift schemas

• CI runs the main suite on Python 3.12 and 3.13 without Docling. The adapter is excluded from coverage and tested by a separate ingest job with the extra installed.

• New tools follow the template's docstring metadata format and raise fastmcp.exceptions.ToolError on failure. A returned {"status": "error"} payload would reach clients as a success.

• Containers: make container (Podman or Docker Compose; OKF_HOST_PORT sets the host port). Compose publishes the MCP port on 127.0.0.1 only, mounts ./data read-only, and starts PostgreSQL only with --profile auth.

Troubleshooting

Roadmap

Tracked in detail in PLAN.md.

• Ingestion to OKF v0.2 with provenance, validation and atomic snapshots

• Browse, search and fetch MCP tools with trust and lifecycle signals

• Hybrid keyword + semantic search with a local embedding model

• Pilot on a real department corpus (GitLab Handbook HR, 94 pages, 30 questions): Hit@5 0.93

• Per-user authentication and permission filtering, synchronized from source systems; audit log

• sqlite-vec backend chosen automatically when the corpus outgrows in-memory search

• Connectors (file shares, SharePoint, Google Drive) and scheduled refresh

• Container image and Kubernetes deployment verified end to end; OpenShift manifests schema-validated

• Pilot on your own department's documents with questions from its users (runbook and question template ready: docs/live-checks.md)

• SharePoint and Google Drive against a live tenant; OpenShift on a live cluster (one-command checks ready, need your access)

Acknowledgements

• Open Knowledge Format by Google Cloud Platform.

• Built on the Red Hat Data & AI template MCP server (commit 286b9e7); its guides live in docs/.

• Docling for document conversion and FastMCP for the MCP server.

License

Apache 2.0, the same license as the template this project is based on.