# tidewaterdata (Project Tidewater)

Record-level lineage with replayable transformations, delivered as an MCP
server, for data-quality work at Tallowfield Technologies.

## The problem

Freight and logistics records arrive dirty: whitespace from CSV exports,
inconsistent casing on port and carrier codes, dates in half a dozen formats,
missing fields. Cleaning them is easy. Proving *what* was changed, *why*, and
that the cleaned record genuinely follows from the raw one is the hard part —
and it is what an auditor or a downstream dispute actually needs.

tidewaterdata makes every change go through a named, parameterised,
deterministic transformation and records each application with content hashes on
both sides. Given the original record and the recorded lineage you can **replay**
the whole history and prove the stored result follows from the stored steps — or
find the exact step where something was altered.

## How the CPU-only constraint shapes the design

The target environment is **CPU-only, with no GPU**, and in practice keeps a
small, often-isolated runtime footprint. That drives concrete choices:

- Correctness rests on canonical JSON + stdlib `blake2b` hashing and pure-Python
  transforms. Nothing wants an accelerator. (ADR 0002)
- The runtime core has **no third-party dependencies**, including the MCP server
  itself, which implements the JSON-RPC stdio protocol directly. (ADR 0001)
- Any model/LLM behaviour goes through a provider interface. The default
  `StubProvider` is deterministic and offline, so the whole test suite runs with
  no API key and no network. If the networked `RealProvider` is selected but
  unconfigured — the normal case on an isolated box — the service degrades to the
  stub with a logged warning instead of failing. (ADR 0004)

`tests/test_offline_cpu.py` holds this line: it runs the full flow with the
socket layer stubbed out and asserts no GPU/model libraries were imported.

## Architecture

```
                MCP client (stdio JSON-RPC)          shell
                          |                             |
                   tidewaterdata/server.py       tidewaterdata/cli.py
                          \_____________  _____________/
                                        \/
                            tidewaterdata/service.py
              validate -> run core -> time -> structured log (stderr)
                                        |
        validation.py   config.py   lineage.py (Pipeline / replay / verify_chain)
                                        |
              transforms.py (registry)   hashing.py (canonical JSON + blake2b)
                                        |
                        providers/ base | stub (offline) | real (networked)
```

The core (`lineage`, `transforms`, `hashing`, `types`) knows nothing about
transport. `service.py` is the one place that validates input, times work, and
logs. `server.py` and `cli.py` are thin adapters over it. Transformations are
identified by `(name, params)` and resolved through a registry, so a lineage
serialises to pure JSON and replays anywhere without shipping code. (ADR 0003)

Decision records for the contested calls are in [`docs/adr/`](docs/adr/).

## Install

```
make install          # creates .venv and installs the package with dev extras
```

`make` uses `PY ?= .venv/bin/python`; override with `make test PY=/path/to/python`.
The core imports only the standard library, so you can also run it straight from
a checkout without installing anything:

```
python3.12 -m tidewaterdata list-transforms
```

## Quickstart

Every command below runs against a checkout with `python3.12 -m tidewaterdata`;
after `make install`, the console script `tidewater` is equivalent.

List the available transforms:

```
python3.12 -m tidewaterdata list-transforms
```

Clean a record and get back the result plus a replayable lineage:

```
echo '{"record": {"id": "1", "name": "  acme   freight "},
       "pipeline": [{"name": "trim_whitespace", "params": {"field": "name"}},
                    {"name": "collapse_spaces", "params": {"field": "name"}}]}' \
  | python3.12 -m tidewaterdata clean -i -
```

The `result.name` is `"acme freight"`, and `lineage` carries the per-step hashes
and diffs. Save the whole `{record, lineage}` and verify it later:

```
echo '{"record": {"id": "1", "name": "  acme   freight "},
       "lineage": { ...the lineage printed above... }}' \
  | python3.12 -m tidewaterdata replay -i -
```

`replay` prints `{"result": ..., "verified": true}` on success, or exits `2`
with a `replay mismatch at step N` message if the record or lineage was altered.

Run the MCP server (reads newline-delimited JSON-RPC on stdin):

```
python3.12 -m tidewaterdata serve
```

Library use:

```python
from tidewaterdata import Pipeline, Transformation, replay

rec = {"id": "shp-1", "name": "  acme   freight ", "port": "lax"}
pipe = Pipeline([
    Transformation("trim_whitespace", {"field": "name"}),
    Transformation("collapse_spaces", {"field": "name"}),
    Transformation("to_upper", {"field": "port"}),
])
lineage, cleaned = pipe.run(rec)
assert replay(rec, lineage) == cleaned   # raises ReplayMismatch if tampered
```

## MCP tools

The server exposes four tools via `tools/list` / `tools/call`:

| Tool | Arguments | Returns |
|---|---|---|
| `list_transforms` | none | available transform names |
| `clean_record` | `record`, `pipeline` | `{result, lineage}` |
| `replay_lineage` | `record`, `lineage` | `{result, verified}` |
| `suggest_transforms` | `record`, `required_fields?` | `{suggestions}` |

Validation failures and replay mismatches come back as a normal `tools/call`
result with `isError: true` and a message; malformed JSON or an unknown method
comes back as a JSON-RPC error.

## Transforms

`trim_whitespace`, `collapse_spaces`, `to_upper`, `to_lower`, `default_missing`,
`drop_field`, `rename_field`, `coerce_int`, `standardize_date`. Register your own
with `@tidewaterdata.register("name")` — a transform is a pure `dict -> dict`
function that must not mutate its input.

## Configuration

All optional, read from the environment at startup and validated there (a bad
value fails immediately with a `configuration error`):

| Variable | Default | Meaning |
|---|---|---|
| `TIDEWATER_PROVIDER` | `stub` | `stub` (offline) or `real` (networked) |
| `TIDEWATER_MAX_RECORD_BYTES` | `1000000` | reject records larger than this |
| `TIDEWATER_MAX_STEPS` | `100` | reject pipelines longer than this |
| `TIDEWATER_MAX_BATCH` | `1000` | batch-size cap (reserved; see limitations) |
| `TIDEWATER_LOG_LEVEL` | `INFO` | `DEBUG`..`CRITICAL` |

`Config.from_file(path)` overlays these defaults with keys from a JSON object.

Logs are one JSON object per line on **stderr** (stdout is the protocol/result
stream). The size and length limits are the defence against resource
exhaustion: oversized input is rejected before any work is done.

The `real` provider additionally reads `TIDEWATER_PROVIDER_ENDPOINT` and
`TIDEWATER_PROVIDER_API_KEY`, and needs the `real` extra
(`pip install -e .[real]`, which pulls in `httpx`).

## Test

```
make test             # or: python3.12 -m pytest
```

CLI exit codes: `0` success, `2` bad input / validation / missing file / replay
mismatch, `1` unexpected.

## Known limitations

- **No batch tool yet.** `TIDEWATER_MAX_BATCH` is wired into config but there is
  no `clean_batch` MCP tool; records are cleaned one at a time. The cap exists so
  the limit is enforced the moment batching lands.
- **Replay depends on transform behaviour staying stable across versions.**
  Changing what a registered transform does will break replay of older lineages —
  surfaced as a `ReplayMismatch`, not a silent wrong answer, but it is a real
  compatibility obligation. (ADR 0003)
- **16-byte hashes detect tampering and drift, not a determined forger** who
  recomputes the whole chain. Signing the final lineage is a future option.
  (ADR 0002)
- **The `real` provider's response contract is provisional.** It assumes a
  `{"suggestions": [...]}` shape; see the TODO in `providers/real.py` pending a
  frozen endpoint spec.
- **No deployment wrapper.** systemd/container packaging is out of scope for this
  delivery.

---

*Tallowfield Technologies is an illustrative client; this repository is a self-directed reference implementation built to work end to end.*