Semantic layer gap closure

Follow-up to Jagadish-semantic-engineering-layer (commit e290442). That repo shows why an agent needs governed meanings before it acts. This repo closes the gaps it listed or that probing found, then measures the result on a held-out set the engine never saw.

Recruiter summary. The original layer scored 1.0 on 30 cases its author labelled. On 318 new held-out cases it scored 0.346 accuracy, with 102 unsafe proceeds and 12 crashes. The labels came from a facts-only oracle, and the split was frozen by hash before any engine work. This engine adds negation, hedging, typo tolerance, Spanish, French and German, units, timestamps, text-vs-source cross-checks, domain routing, declared ambiguity, governance, a persisted review queue, drift monitoring and a stdio MCP server. On the same split it scores 0.965, with 2 unsafe proceeds and 0 crashes, and it still scores 1.0 on the original 30. All data is synthetic, every company is fictional, and everything runs offline with no API keys.

Purpose and who it is for

• Data and AI engineers putting an agent in front of domain text, such as ops chat or case notes, where "positive", "clear" or "30 hours" can trigger an irreversible action.

• Reviewers who want to see what it takes to turn "1.0 on my own cases" into a claim that holds up.

• The owner (and interviewers), as a worked example of negation, held-out evaluation, ablation and glossary governance without an LLM.

This repo does not repeat the original's motivation, glossary design or baseline comparison. Read the original first.

Related MCP server: DeadBolt

The gaps and how each was closed

The "before" numbers come from the original code, vendored unchanged at reference/original_e290442/. The "after" numbers come from this engine on the frozen held-out test split. The per-gap table in the results block below is regenerated by CI.

Results

Synthetic data, measured by python scripts/render_readme.py. Tests: 185 collected, 185 passed. Held-out test split: 318 cases labelled by a facts-only oracle, frozen by SHA-256 before engine work and run once.

Original 30 author-labelled cases (regression check): original layer accuracy 1.000, gap closure 1.000, unsafe proceeds 0.

Held-out test split, overall:

Per gap, original (commit e290442) vs this repo:

Drift demo (synthetic windows): stable window alerts 0; drifted window alerts 2 (PSI 0.2848); top draft candidate 'vxm' (count 33).

The full tables, ablation and every miss are in results/heldout_eval.md.

Ablation: turning off any single feature raises unsafe proceeds on the test split. The biggest drops come from turning off text quantities (24 unsafe), fuzzy matching (16) and inflection (12). tests/test_ablation_and_eval.py enforces this on dev for every flag.

Architecture

flowchart LR
  A[agent proposes action] --> T[tool.call_tool<br/>strict schema]
  T --> N[normalize<br/>fold, sentences, clause breaks]
  N --> F[fuzzy + lemma + compounds]
  F --> M[matcher<br/>phrases, frames, polarity,<br/>ambiguity, foreign terms, quantities]
  G[(glossary 2026.11.0<br/>owners, approvers,<br/>translations)] --> M
  M --> E[engine rules<br/>strictest wins]
  C[(field contracts<br/>units, limits, bands)] --> E
  E -->|proceed| ACT[agent acts]
  E -->|escalate / block| Q[(review queue<br/>SQLite + hash chain)]
  E --> D[drift windows<br/>PSI, rates, medians]
  D --> P[draft term proposals]
  P -.owner + second approver.-> G
  S[MCP stdio server] --> T

How it works

1. Normalize. Fold accents and case, split into sentences, and keep commas and dashes as clause breaks so cues cannot leak across clauses.

2. Repair words. Lemmas and compound joins come first. Fuzzy correction applies only to tokens in the domain vocabulary, never to protected words, and is refused on ties between signals.

3. Match. Exact phrases are matched longest first, including approved translations. Slot frames ("crossmatch ... positive") come next. Then dangling checks: a polarity word with no term, or a critical topic with no outcome, escalates.

4. Polarity. Negation and hedge cues are read in the window around each hit.

5. Routing and ambiguity. Critical phrases from other domains, and declared ambiguous abbreviations, escalate.

6. Readings. Structured readings are checked against contracts. Quantities in text are cross-checked against them.

7. Decide. The strictest outcome wins. Proceed also needs a clearance: an exact, factual clearance term or a valid reading.

8. Record. Every decision carries the glossary version and hash. Anything that is not a proceed goes to the review queue.

More detail: docs/how-it-works.md.

Real code and output

# src/semgap/engine.py (excerpt)
if t.signal == "critical_block":
    if h.negated or h.hedged:
        kind = "negated" if h.negated else "hedged"
        bump("escalate", f"{t.id} is {kind} ('{h.cue}'): a person must confirm, never auto-proceed")
    else:
        bump("block", f"{t.id}: {t.definition}" + (" (matched after spelling correction)" if h.fuzzy else ""))
elif t.signal == "clear":
    if h.negated or h.hedged:
        bump("escalate", f"{t.id} is {'negated' if h.negated else 'hedged'} ('{h.cue}'): not a clearance")
    elif h.fuzzy:
        bump("escalate", f"{t.id} only matched after spelling correction: clearance must be exact")

PYTHONPATH=src python examples/demo.py (real output, reasons trimmed):

'File is not clear to close'             original=proceed           new=escalate  MTG.CTC is negated ('not'): not a clearance
'Book courier once crossmatch negative'  original=proceed           new=escalate  MED.XM_NEGATIVE is hedged ('once'): not a clearance
'xm neg, cold ischemia 30 hours'         original=proceed           new=block     cold_ischemia_elapsed in text is outside its limit (30.0h elapsed of 24h)
'crossmatch negative' + NaN reading      original=proceed           new=escalate  cold_ischemia_elapsed: value must be finite
'clear to close, crossmatch positive'    original=proceed           new=escalate  text names MED.XM_POSITIVE from medical_logistics: domain routing may be wrong
'Positive cross-match'                   original=proceed           new=block     MED.XM_POSITIVE: Recipient antibodies react against donor cells...
'positive crossmtch'                     original=proceed           new=block     MED.XM_POSITIVE ...
'prueba cruzada positiva'                original=proceed           new=block     MED.XM_POSITIVE ...
'crossmatch negative' + mixed-tz reading original=crash (TypeError) new=escalate  one timestamp has a timezone and the other does not

How to run (offline, standard library only)

python -m venv .venv && . .venv/bin/activate && pip install -r requirements-dev.txt
python scripts/count_tests.py                 # prints TOTAL_TESTS=...
PYTHONPATH=src python -m semgap.evaluate      # writes results/heldout_eval.{json,md}
PYTHONPATH=src python examples/demo.py
python scripts/render_readme.py --check       # README numbers match a fresh run
sha256sum -c heldout/FROZEN.sha256            # test split unchanged
PYTHONPATH=src python -m semgap.mcp_server    # JSON-RPC over stdio; queue at var/review_queue.sqlite
python labeling/kappa.py export --n 60        # blind sheet for real expert labelling

There is no cloud part and nothing is billable.

Guardrails

• Fail safe: unknown, malformed, negated, hedged, misspelled-clearance, ambiguous, foreign-domain, text-only and draft inputs all escalate, and the strictest outcome wins.

• Proceed needs positive evidence (a clearance), not just the absence of a problem.

• Draft terms and draft translations can never clear or block on their own.

• Two-person rule: the approver must differ from the owner, translations need a named reviewer, and a weakened signal needs a second approver.

• The review queue is append-only and hash-chained, can only be resolved by a named human, and stores a text hash rather than raw text.

• The tool schema is strict: enum domain, 4000-character text cap, no extra keys. The MCP server caps line size.

• The held-out test split is checked against its SHA-256 in CI.

See docs/guardrails.md and docs/threat-model.md.

Limitations (kept as tests in tests/test_known_limits.py)

• Novel paraphrases ("deferred interest added to principal") are not understood. With a valid reading this is an unsafe proceed (1 on test).

• Negation that splits a non-English multi-word term ("prueba cruzada no positiva") hides the term.

• A typo on a non-term word in a clearance sentence still proceeds. The oracle says escalate (1 on test).

• Decimal commas ("25,5 h") are misread. A fuzzy tie can pick the wrong language form (fails safe). A trailing modal ("neg am may apply") blocks where the oracle escalates.

• Labels are synthetic and come from one author's phrase bank. Independence comes from fact-derived labels and the freeze, not from separate people. A kappa kit is included but has not been run with experts.

• Limits are lab values, not clinical, actuarial or lending policy.

Full list: docs/limitations.md.

Built vs planned

Docs

how it works · guardrails · limitations · implementation guide · interview points · threat model · ADRs · training: labs and capstone · label policy · held-out protocol

Related repos

• Jagadish-semantic-engineering-layer: the original. Motivation, base glossary, baseline comparison and the semantic engineering checklist live there.

MIT licence. Synthetic data only. Organisation names (Northwind, Contoso, Fabrikam, Kestrel, Larkspur, Brambleton, Tidewell, Quillmark) are fictional.