mardale-trading-search-ranking-mcp

Project Fathom: relevance-drift monitoring for search ranking, built for Mardale Trading's aviation-maintenance document search.

The system answers one operational question: is our ranking getting worse? It ranks candidate documents for a query, scores that ranking against editorial relevance judgements, and compares the result to an earlier baseline to detect and classify drift. Every decision -- each ranking and each drift verdict -- is written to a tamper-evident audit trail, because the engagement requires that every decision carry a reviewable record.

Why the audit trail is central

The hard constraint on this build is auditability. It is not a logging layer added at the end; it drives the design:

• fathomsearch/audit.py maintains an append-only, hash-chained ledger. Each record hashes its own content together with the previous record's hash, so any edit, deletion, or reordering is detectable by AuditLog.verify().

• rank_documents and monitor_drift accept an AuditLog and emit a record for every decision, including the rationale and the thresholds applied.

• Domain types are frozen dataclasses: once a decision is hashed, the data behind it cannot be mutated out from under the record.

Related MCP server: findocs-mcp

Layout

fathomsearch/
  types.py            domain types (Document, Query, Judgment, Ranking, DriftReport, ...)
  audit.py            hash-chained audit ledger
  metrics.py          nDCG / precision / recall against editorial judgements
  ranking.py          provider scores -> ordered ranking (+ audit)
  drift.py            central algorithm: relevance-drift monitoring (+ audit)
  providers/
    base.py           RelevanceProvider interface
    stub.py           deterministic, offline lexical provider (used by tests)
    real.py           HTTP-backed provider (production; optional httpx)
tests/

Install and test

make venv
make install
make test

make targets run through $(PY) (default .venv/bin/python); override the interpreter with make test PY=/path/to/python.

The test suite runs fully offline with no API key: all model-like behaviour is behind RelevanceProvider, and the tests use DeterministicStubProvider. tests/test_offline.py disables socket creation and runs the whole rank -> evaluate -> drift pipeline to prove nothing reaches the network.

Quickstart

Sample data lives in examples/. From the repository root, score a baseline, score a degraded corpus (one relevant document removed), then compare them:

python -m fathomsearch evaluate \
  --documents examples/documents.json --queries examples/queries.json \
  --judgments examples/judgments.json --label v1 \
  --out v1.eval.json --audit-out v1.audit.jsonl

python -m fathomsearch evaluate \
  --documents examples/documents_degraded.json --queries examples/queries.json \
  --judgments examples/judgments.json --label v2 --out v2.eval.json

python -m fathomsearch drift --baseline v1.eval.json --current v2.eval.json \
  --out drift.json

The drift command exits 1 because removing d1 from the corpus regresses query q1; drift.json shows the per-query deltas and v1.audit.jsonl holds the verifiable ledger for the baseline run.

Using the core

from fathomsearch.types import Query, Document, Judgment, JudgmentSet, Evaluation
from fathomsearch.providers.stub import DeterministicStubProvider
from fathomsearch.ranking import rank_documents
from fathomsearch.metrics import evaluate_ranking
from fathomsearch.drift import monitor_drift
from fathomsearch.audit import AuditLog

audit = AuditLog()
provider = DeterministicStubProvider()

query = Query("q1", "hydraulic leak inspection")
docs = [
    Document("d1", "hydraulic leak inspection checklist"),
    Document("d2", "galley equipment manual"),
]
judgments = JudgmentSet([Judgment("q1", "d1", 3), Judgment("q1", "d2", 0)])

ranking = rank_documents(provider, query, docs, audit=audit)
evaluation = evaluate_ranking(ranking, judgments, k=10)

# Compare two snapshots to detect drift:
baseline = Evaluation("v1", {"q1": evaluation})
current = Evaluation("v2", {"q1": evaluation})
report = monitor_drift(baseline, current, metric="ndcg", audit=audit)

assert audit.verify()          # chain intact
audit.write_jsonl("run.audit.jsonl")

Drift severity bands

monitor_drift classifies each query's metric change (current minus baseline) using DriftThresholds:

• major_regression: delta <= major (default -0.15)

• minor_regression: major < delta <= minor (default -0.05)

• stable: within +/- stable_band (default 0.02)

• improved: delta >= stable_band

A query that appears in only one snapshot is a drift signal that cannot be measured as a delta; it is excluded from the deltas and recorded separately.

The real provider

fathomsearch/providers/real.py posts query/document text to an HTTP scoring backend. It requires httpx (pip install ".[real]") and an API key via FATHOM_API_KEY or the constructor. It never falls back to a heuristic on failure: a silent fallback would poison drift comparisons. It is not imported by the test suite.

Command-line interface

Installed as fathomsearch (or run python -m fathomsearch). Input files are JSON:

• documents: [{"id": "...", "text": "...", "metadata": {}}, ...]

• queries: [{"id": "...", "text": "..."}, ...]

• judgments: [{"query_id": "...", "doc_id": "...", "grade": 0}, ...]

Score a run into an evaluation file, capturing the audit ledger:

python -m fathomsearch evaluate \
  --documents docs.json --queries queries.json --judgments qrels.json \
  --label v1 --out v1.eval.json --audit-out v1.audit.jsonl

Compare two evaluations and classify drift:

python -m fathomsearch drift --baseline v1.eval.json --current v2.eval.json \
  --out drift.json --audit-out drift.audit.jsonl

Exit codes: 0 no regressions, 1 one or more regressions found (usable as a CI gate), 2 an expected error (bad input, missing file, bad config).

Start the MCP server (needs the mcp package installed):

python -m fathomsearch serve

Configuration

Configuration is read from the environment and validated at startup:

Error handling and observability

• fathomsearch/errors.py defines the failure hierarchy; the CLI turns any FathomError into exit code 2 with a message, and lets real bugs crash.

• fathomsearch/loader.py reports missing files, bad JSON, and wrong-shaped records with the file and record index named.

• fathomsearch/service.py enforces input validation and resource limits, times each operation, and logs structured events.

• fathomsearch/logging_config.py emits one JSON object per log line and a timed() context manager that records duration even when the block raises.

• A failed audit write is surfaced (persist_audit returns False and logs an error) rather than swallowed, because auditability is the binding constraint.

Architecture

Layered so the audit trail and validation are enforced once, regardless of entry point:

CLI (cli.py)  /  MCP server (server.py)
        \             /
       RankingService (service.py)      validation, limits, timing, audit
        /      |          \
 ranking.py  metrics.py  drift.py       core algorithm
        \      |          /
          providers/ (base, stub, real) + audit.py + types.py

config.py builds the Config and the provider; loader.py handles file I/O. Both entry points call RankingService, so an MCP caller and a CLI caller get identical validation, resource limits, and audit records.

Design decisions with a real alternative are recorded in docs/adr/.

Known limitations

• The stub provider is lexical (term overlap), not a learned relevance model. It exists for offline runs and tests; production relevance needs the real provider against a scoring backend.

• The audit chain proves internal consistency, not authorship: a party who rewrites the whole ledger produces a valid chain. Detecting that requires signing or an external anchor, which is out of scope here (see ADR 0001).

• nDCG treats a query with no relevant judgement as a perfect 1.0, which can flatter the aggregate; read nDCG alongside precision (see ADR 0002).

• The MCP server (serve) requires the mcp package, which is not installed by the offline test environment. build_server is imported lazily and is not exercised by the suite.

• Metrics assume graded judgements are complete enough to trust; unjudged documents count as non-relevant (the standard pooled-qrels assumption), which can understate quality on shallow pools.

Status

Delivered: the core algorithm, providers, configuration, the CLI and MCP entry points, input validation, error handling, structured logging, ADRs, and the test suite. The MCP server transport imports mcp lazily, so everything else runs and tests without it installed.

Mardale Trading is an illustrative client; this repository is a self-directed reference implementation built to work end to end.