# mardale-trading-search-ranking-mcp

Project Fathom: relevance-drift monitoring for search ranking, built for
Mardale Trading's aviation-maintenance document search.

The system answers one operational question: *is our ranking getting worse?*
It ranks candidate documents for a query, scores that ranking against editorial
relevance judgements, and compares the result to an earlier baseline to detect
and classify drift. Every decision -- each ranking and each drift verdict --
is written to a tamper-evident audit trail, because the engagement requires
that every decision carry a reviewable record.

## Why the audit trail is central

The hard constraint on this build is auditability. It is not a logging layer
added at the end; it drives the design:

- `fathomsearch/audit.py` maintains an append-only, hash-chained ledger. Each
  record hashes its own content together with the previous record's hash, so
  any edit, deletion, or reordering is detectable by `AuditLog.verify()`.
- `rank_documents` and `monitor_drift` accept an `AuditLog` and emit a record
  for every decision, including the rationale and the thresholds applied.
- Domain types are frozen dataclasses: once a decision is hashed, the data
  behind it cannot be mutated out from under the record.

## Layout

```
fathomsearch/
  types.py            domain types (Document, Query, Judgment, Ranking, DriftReport, ...)
  audit.py            hash-chained audit ledger
  metrics.py          nDCG / precision / recall against editorial judgements
  ranking.py          provider scores -> ordered ranking (+ audit)
  drift.py            central algorithm: relevance-drift monitoring (+ audit)
  providers/
    base.py           RelevanceProvider interface
    stub.py           deterministic, offline lexical provider (used by tests)
    real.py           HTTP-backed provider (production; optional httpx)
tests/
```

## Install and test

```
make venv
make install
make test
```

`make` targets run through `$(PY)` (default `.venv/bin/python`); override the
interpreter with `make test PY=/path/to/python`.

The test suite runs fully offline with no API key: all model-like behaviour is
behind `RelevanceProvider`, and the tests use `DeterministicStubProvider`.
`tests/test_offline.py` disables socket creation and runs the whole
rank -> evaluate -> drift pipeline to prove nothing reaches the network.

## Quickstart

Sample data lives in `examples/`. From the repository root, score a baseline,
score a degraded corpus (one relevant document removed), then compare them:

```
python -m fathomsearch evaluate \
  --documents examples/documents.json --queries examples/queries.json \
  --judgments examples/judgments.json --label v1 \
  --out v1.eval.json --audit-out v1.audit.jsonl

python -m fathomsearch evaluate \
  --documents examples/documents_degraded.json --queries examples/queries.json \
  --judgments examples/judgments.json --label v2 --out v2.eval.json

python -m fathomsearch drift --baseline v1.eval.json --current v2.eval.json \
  --out drift.json
```

The drift command exits 1 because removing `d1` from the corpus regresses query
`q1`; `drift.json` shows the per-query deltas and `v1.audit.jsonl` holds the
verifiable ledger for the baseline run.

## Using the core

```python
from fathomsearch.types import Query, Document, Judgment, JudgmentSet, Evaluation
from fathomsearch.providers.stub import DeterministicStubProvider
from fathomsearch.ranking import rank_documents
from fathomsearch.metrics import evaluate_ranking
from fathomsearch.drift import monitor_drift
from fathomsearch.audit import AuditLog

audit = AuditLog()
provider = DeterministicStubProvider()

query = Query("q1", "hydraulic leak inspection")
docs = [
    Document("d1", "hydraulic leak inspection checklist"),
    Document("d2", "galley equipment manual"),
]
judgments = JudgmentSet([Judgment("q1", "d1", 3), Judgment("q1", "d2", 0)])

ranking = rank_documents(provider, query, docs, audit=audit)
evaluation = evaluate_ranking(ranking, judgments, k=10)

# Compare two snapshots to detect drift:
baseline = Evaluation("v1", {"q1": evaluation})
current = Evaluation("v2", {"q1": evaluation})
report = monitor_drift(baseline, current, metric="ndcg", audit=audit)

assert audit.verify()          # chain intact
audit.write_jsonl("run.audit.jsonl")
```

## Drift severity bands

`monitor_drift` classifies each query's metric change (current minus baseline)
using `DriftThresholds`:

- `major_regression`: delta <= `major` (default -0.15)
- `minor_regression`: `major` < delta <= `minor` (default -0.05)
- `stable`: within +/- `stable_band` (default 0.02)
- `improved`: delta >= `stable_band`

A query that appears in only one snapshot is a drift signal that cannot be
measured as a delta; it is excluded from the deltas and recorded separately.

## The real provider

`fathomsearch/providers/real.py` posts query/document text to an HTTP scoring
backend. It requires `httpx` (`pip install ".[real]"`) and an API key via
`FATHOM_API_KEY` or the constructor. It never falls back to a heuristic on
failure: a silent fallback would poison drift comparisons. It is not imported
by the test suite.

## Command-line interface

Installed as `fathomsearch` (or run `python -m fathomsearch`). Input files are
JSON:

- documents: `[{"id": "...", "text": "...", "metadata": {}}, ...]`
- queries: `[{"id": "...", "text": "..."}, ...]`
- judgments: `[{"query_id": "...", "doc_id": "...", "grade": 0}, ...]`

Score a run into an evaluation file, capturing the audit ledger:

```
python -m fathomsearch evaluate \
  --documents docs.json --queries queries.json --judgments qrels.json \
  --label v1 --out v1.eval.json --audit-out v1.audit.jsonl
```

Compare two evaluations and classify drift:

```
python -m fathomsearch drift --baseline v1.eval.json --current v2.eval.json \
  --out drift.json --audit-out drift.audit.jsonl
```

Exit codes: `0` no regressions, `1` one or more regressions found (usable as a
CI gate), `2` an expected error (bad input, missing file, bad config).

Start the MCP server (needs the `mcp` package installed):

```
python -m fathomsearch serve
```

## Configuration

Configuration is read from the environment and validated at startup:

| Variable | Default | Meaning |
| --- | --- | --- |
| `FATHOM_PROVIDER` | `stub` | `stub` or `real` |
| `FATHOM_METRIC` | `ndcg` | `ndcg`, `precision`, or `recall` for drift |
| `FATHOM_K` | `10` | cutoff for metrics |
| `FATHOM_MAX_DOCUMENTS` | `10000` | per-query candidate limit (resource guard) |
| `FATHOM_MAX_QUERY_LENGTH` | `2048` | max query characters |
| `FATHOM_LOG_LEVEL` | `INFO` | logging level |
| `FATHOM_LOG_JSON` | `1` | `0` for human-readable logs |
| `FATHOM_AUDIT_PATH` | unset | default audit ledger path |
| `FATHOM_REAL_ENDPOINT` / `FATHOM_REAL_MODEL` / `FATHOM_API_KEY` | unset | real provider settings |

## Error handling and observability

- `fathomsearch/errors.py` defines the failure hierarchy; the CLI turns any
  `FathomError` into exit code 2 with a message, and lets real bugs crash.
- `fathomsearch/loader.py` reports missing files, bad JSON, and wrong-shaped
  records with the file and record index named.
- `fathomsearch/service.py` enforces input validation and resource limits,
  times each operation, and logs structured events.
- `fathomsearch/logging_config.py` emits one JSON object per log line and a
  `timed()` context manager that records duration even when the block raises.
- A failed audit write is surfaced (`persist_audit` returns `False` and logs an
  error) rather than swallowed, because auditability is the binding constraint.

## Architecture

Layered so the audit trail and validation are enforced once, regardless of
entry point:

```
CLI (cli.py)  /  MCP server (server.py)
        \             /
       RankingService (service.py)      validation, limits, timing, audit
        /      |          \
 ranking.py  metrics.py  drift.py       core algorithm
        \      |          /
          providers/ (base, stub, real) + audit.py + types.py
```

`config.py` builds the `Config` and the provider; `loader.py` handles file I/O.
Both entry points call `RankingService`, so an MCP caller and a CLI caller get
identical validation, resource limits, and audit records.

Design decisions with a real alternative are recorded in
[`docs/adr/`](docs/adr/README.md).

## Known limitations

- The stub provider is lexical (term overlap), not a learned relevance model.
  It exists for offline runs and tests; production relevance needs the real
  provider against a scoring backend.
- The audit chain proves internal consistency, not authorship: a party who
  rewrites the whole ledger produces a valid chain. Detecting that requires
  signing or an external anchor, which is out of scope here (see ADR 0001).
- nDCG treats a query with no relevant judgement as a perfect 1.0, which can
  flatter the aggregate; read nDCG alongside precision (see ADR 0002).
- The MCP server (`serve`) requires the `mcp` package, which is not installed by
  the offline test environment. `build_server` is imported lazily and is not
  exercised by the suite.
- Metrics assume graded judgements are complete enough to trust; unjudged
  documents count as non-relevant (the standard pooled-qrels assumption), which
  can understate quality on shallow pools.

## Status

Delivered: the core algorithm, providers, configuration, the CLI and MCP
entry points, input validation, error handling, structured logging, ADRs, and
the test suite. The MCP server transport imports `mcp` lazily, so everything
else runs and tests without it installed.

---

*Mardale Trading is an illustrative client; this repository is a self-directed reference implementation built to work end to end.*