netherbymultimodal (Project Netherby)

Air-gapped multimodal document processing for Harrow Risk. It fuses document layout parsing with text extraction: given a page's extracted text spans and the layout regions detected on that page, it produces ordered, labelled reading blocks and reconstructed text.

The defining constraint is that it runs fully air-gapped. Nothing reaches the network at runtime. That is enforced in code (netherbymultimodal/airgap.py) and proven by the test suite, not just asserted in prose.

Layout

netherbymultimodal/
  __init__.py        public API: fuse_page, fuse_document, domain types
  types.py           BBox, TextSpan, LayoutRegion, Page, FusedBlock, FusedPage
  airgap.py          no_network(): blocks sockets/DNS for a code section
  fusion.py          the core algorithm (assignment, ordering, reconstruction)
  server.py          MCP server over stdio (JSON-RPC 2.0)
  service.py         validate payload -> fuse -> plain data; timing + logging
  config.py          Config with env overrides and validation
  logging.py         structured JSON logging to stderr
  errors.py          error taxonomy (InvalidInputError, ResourceLimitError)
  providers/
    base.py          LayoutProvider interface + ProviderUnavailable
    stub.py          deterministic, dependency-free provider (tests + offline)
    real.py          local-ONNX provider (loads under the air-gap guard)
tests/

Related MCP server: Xberg MCP Server

The algorithm

fuse_page(page, regions) in fusion.py:

1. Assignment — each text span is attached to the region that contains the largest fraction of the span's area, above a threshold (min_containment). Containment, not IoU: a small span inside a large region should score 1.0.

2. Orphans — spans no region claims are grouped by proximity into their own blocks. Extracted text is never silently dropped (an audit requirement).

3. Intra-block order — spans are grouped into visual lines (by vertical centre) and ordered top-to-bottom, then left-to-right within a line.

4. Inter-block order — blocks are grouped into columns by left-edge clustering and read column-by-column, so two-column pages read correctly.

fuse_document(pages, provider) sources regions from a LayoutProvider and runs the whole thing inside no_network().

Providers

All model behaviour goes through LayoutProvider so the core never depends on a model and the suite runs offline with no API key.

• StubLayoutProvider — derives regions from span geometry. Deterministic, no dependencies. Default for offline runs and the basis of the tests.

• OnnxLayoutProvider — loads a local ONNX detector. The model must already be on disk; there is no download path. Loading happens inside no_network(), so any library that tries to fetch weights or send telemetry fails loudly. The tensor inference is a marked seam (real.py TODO) pending Harrow Risk's model choice; a missing or unloadable model raises ProviderUnavailable rather than returning an empty layout.

Threat model (air-gap)

no_network() patches the standard socket API (socket, create_connection, getaddrinfo, gethostbyname[_ex]). It stops accidental egress from our code and from libraries that use standard sockets. It is defence in depth, not a sandbox: it does not contain code that bypasses socket or calls the OS directly. Production deployment still isolates the process at the network layer.

Quick start

make venv
make install
make test

make uses PY ?= .venv/bin/python; override it, e.g. make test PY=python3.12.

MCP server

The entry point is an MCP server speaking JSON-RPC 2.0 over stdio, one JSON object per line. Logs go to stderr so they never corrupt the protocol stream on stdout. No socket is opened.

make run
# or: python -m netherbymultimodal.server

Methods: initialize, tools/list, and tools/call for the one tool, fuse_document. Example exchange (request in, response out):

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"fuse_document","arguments":{"pages":[{"width":600,"height":800,"spans":[{"text":"Certificate of Insurance","bbox":[50,20,300,40]}]}]}}}

The result's content[0].text is a JSON string with provider, elapsed_ms, and per-page blocks.

Configuration

Read from the environment (NETHERBY_ prefix); all values are validated at startup and a bad value exits non-zero with a message.

Failure handling and degradation

• Malformed payloads (wrong types, non-finite numbers, inverted boxes, missing fields) return JSON-RPC -32602 with a message pointing at the offending path; they never crash the loop.

• Payloads over a configured limit return -32602 as a ResourceLimitError.

• Unparseable input lines return -32700 and the server keeps serving.

• If the onnx provider cannot load or run (missing/unstaged model — the expected air-gapped case), the service logs a degradation record and falls back to the deterministic stub, unless ALLOW_PROVIDER_FALLBACK is false.

Using it as a library

from netherbymultimodal import fuse_document
from netherbymultimodal.providers.stub import StubLayoutProvider
from netherbymultimodal.types import BBox, Page, TextSpan

page = Page(number=1, width=600, height=800, spans=[
    TextSpan("Certificate of Insurance", BBox(50, 20, 300, 40)),
    TextSpan("This policy covers the named insured.", BBox(50, 100, 320, 115)),
])
fused = fuse_document([page], StubLayoutProvider())
for block in fused[0].blocks:
    print(block.reading_order, block.label, repr(block.text))

Development notes

• No third-party runtime dependencies. pytest is the only dev dependency; onnxruntime is an optional extra (pip install -e '.[real]') for the real provider.

• Known limit: full-width elements (banners, page-spanning tables) are bucketed into the leftmost column during column detection. See the TODO in fusion.py (_detect_columns).

• The onnx provider's tensor inference is not wired up (see the TODO in providers/real.py); loading and air-gap enforcement are done, inference awaits a model choice. Use the stub provider until then.

Design decisions

Architecture decision records for the contested calls live in docs/adr/: in-process air-gap enforcement, containment vs. IoU for assignment, hand-rolled JSON-RPC, and provider degradation.

Harrow Risk is an illustrative client; this repository is a self-directed reference implementation built to work end to end.