# netherbymultimodal (Project Netherby)

Air-gapped multimodal document processing for Harrow Risk. It fuses **document
layout parsing** with **text extraction**: given a page's extracted text spans
and the layout regions detected on that page, it produces ordered, labelled
reading blocks and reconstructed text.

The defining constraint is that it runs **fully air-gapped**. Nothing reaches the
network at runtime. That is enforced in code (`netherbymultimodal/airgap.py`) and
proven by the test suite, not just asserted in prose.

## Layout

```
netherbymultimodal/
  __init__.py        public API: fuse_page, fuse_document, domain types
  types.py           BBox, TextSpan, LayoutRegion, Page, FusedBlock, FusedPage
  airgap.py          no_network(): blocks sockets/DNS for a code section
  fusion.py          the core algorithm (assignment, ordering, reconstruction)
  server.py          MCP server over stdio (JSON-RPC 2.0)
  service.py         validate payload -> fuse -> plain data; timing + logging
  config.py          Config with env overrides and validation
  logging.py         structured JSON logging to stderr
  errors.py          error taxonomy (InvalidInputError, ResourceLimitError)
  providers/
    base.py          LayoutProvider interface + ProviderUnavailable
    stub.py          deterministic, dependency-free provider (tests + offline)
    real.py          local-ONNX provider (loads under the air-gap guard)
tests/
```

## The algorithm

`fuse_page(page, regions)` in `fusion.py`:

1. **Assignment** — each text span is attached to the region that contains the
   largest fraction of the span's area, above a threshold (`min_containment`).
   Containment, not IoU: a small span inside a large region should score 1.0.
2. **Orphans** — spans no region claims are grouped by proximity into their own
   blocks. Extracted text is never silently dropped (an audit requirement).
3. **Intra-block order** — spans are grouped into visual lines (by vertical
   centre) and ordered top-to-bottom, then left-to-right within a line.
4. **Inter-block order** — blocks are grouped into columns by left-edge
   clustering and read column-by-column, so two-column pages read correctly.

`fuse_document(pages, provider)` sources regions from a `LayoutProvider` and runs
the whole thing inside `no_network()`.

## Providers

All model behaviour goes through `LayoutProvider` so the core never depends on a
model and the suite runs offline with no API key.

- `StubLayoutProvider` — derives regions from span geometry. Deterministic,
  no dependencies. Default for offline runs and the basis of the tests.
- `OnnxLayoutProvider` — loads a local ONNX detector. The model must already be
  on disk; there is no download path. Loading happens inside `no_network()`, so
  any library that tries to fetch weights or send telemetry fails loudly. The
  tensor inference is a marked seam (`real.py` TODO) pending Harrow Risk's model
  choice; a missing or unloadable model raises `ProviderUnavailable` rather than
  returning an empty layout.

## Threat model (air-gap)

`no_network()` patches the standard `socket` API (`socket`, `create_connection`,
`getaddrinfo`, `gethostbyname[_ex]`). It stops accidental egress from our code
and from libraries that use standard sockets. It is defence in depth, not a
sandbox: it does not contain code that bypasses `socket` or calls the OS
directly. Production deployment still isolates the process at the network layer.

## Quick start

```
make venv
make install
make test
```

`make` uses `PY ?= .venv/bin/python`; override it, e.g. `make test PY=python3.12`.

## MCP server

The entry point is an MCP server speaking JSON-RPC 2.0 over stdio, one JSON
object per line. Logs go to stderr so they never corrupt the protocol stream on
stdout. No socket is opened.

```
make run
# or: python -m netherbymultimodal.server
```

Methods: `initialize`, `tools/list`, and `tools/call` for the one tool,
`fuse_document`. Example exchange (request in, response out):

```json
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"fuse_document","arguments":{"pages":[{"width":600,"height":800,"spans":[{"text":"Certificate of Insurance","bbox":[50,20,300,40]}]}]}}}
```

The result's `content[0].text` is a JSON string with `provider`, `elapsed_ms`,
and per-page `blocks`.

### Configuration

Read from the environment (`NETHERBY_` prefix); all values are validated at
startup and a bad value exits non-zero with a message.

| Variable | Default | Meaning |
| --- | --- | --- |
| `NETHERBY_PROVIDER` | `stub` | `stub` or `onnx` |
| `NETHERBY_MODEL_PATH` | — | required when provider is `onnx` |
| `NETHERBY_ALLOW_PROVIDER_FALLBACK` | `true` | degrade to stub if the model won't load |
| `NETHERBY_MAX_PAGES` | `500` | reject payloads with more pages |
| `NETHERBY_MAX_SPANS_PER_PAGE` | `50000` | reject payloads with more spans per page |
| `NETHERBY_MAX_TEXT_LEN` | `100000` | reject spans with longer text |
| `NETHERBY_MIN_CONTAINMENT` | `0.5` | span->region assignment threshold |
| `NETHERBY_LOG_LEVEL` | `info` | `debug`/`info`/`warning`/`error` |

### Failure handling and degradation

- Malformed payloads (wrong types, non-finite numbers, inverted boxes, missing
  fields) return JSON-RPC `-32602` with a message pointing at the offending
  path; they never crash the loop.
- Payloads over a configured limit return `-32602` as a `ResourceLimitError`.
- Unparseable input lines return `-32700` and the server keeps serving.
- If the `onnx` provider cannot load or run (missing/unstaged model — the
  expected air-gapped case), the service logs a degradation record and falls
  back to the deterministic stub, unless `ALLOW_PROVIDER_FALLBACK` is false.

### Using it as a library

```python
from netherbymultimodal import fuse_document
from netherbymultimodal.providers.stub import StubLayoutProvider
from netherbymultimodal.types import BBox, Page, TextSpan

page = Page(number=1, width=600, height=800, spans=[
    TextSpan("Certificate of Insurance", BBox(50, 20, 300, 40)),
    TextSpan("This policy covers the named insured.", BBox(50, 100, 320, 115)),
])
fused = fuse_document([page], StubLayoutProvider())
for block in fused[0].blocks:
    print(block.reading_order, block.label, repr(block.text))
```

## Development notes

- No third-party runtime dependencies. `pytest` is the only dev dependency;
  `onnxruntime` is an optional extra (`pip install -e '.[real]'`) for the real
  provider.
- Known limit: full-width elements (banners, page-spanning tables) are bucketed
  into the leftmost column during column detection. See the TODO in
  `fusion.py` (`_detect_columns`).
- The `onnx` provider's tensor inference is not wired up (see the TODO in
  `providers/real.py`); loading and air-gap enforcement are done, inference
  awaits a model choice. Use the stub provider until then.

## Design decisions

Architecture decision records for the contested calls live in
[`docs/adr/`](docs/adr/README.md): in-process air-gap enforcement, containment
vs. IoU for assignment, hand-rolled JSON-RPC, and provider degradation.

---

*Harrow Risk is an illustrative client; this repository is a self-directed reference implementation built to work end to end.*