# Agentic Data Cleaner -- notebook + Muse connector

Finds and fixes problems in tabular data (CSV, TSV, Excel, JSON, Parquet, ...): duplicates, placeholder nulls (`N/A`, `?`),
numbers/dates stored as text, spelling variants, encoding damage, outliers, personal data. Safe fixes apply automatically;
anything that deletes or invents data waits for a human yes. Every run produces the cleaned file, an audit report and a
standalone replay script.

One engine, three ways to use it:

```
datacleaner/engine.py  ──►  agentic_data_cleaner.ipynb   (generated by tools/build_notebook.py -- can't drift)
        │
        └──────────────►  server/  ──►  MCP endpoint  /mcp   (what Muse connects to)
                                   └──►  REST API      /v1/...  (OpenAPI at /docs)
```

## 1. Use the notebook

```bash
pip install -r requirements.txt jupyter
jupyter lab agentic_data_cleaner.ipynb     # Run All -- cleans input/uncleaned_data.csv into output/
```

Or as a library: `from datacleaner import AgenticDataCleaner; AgenticDataCleaner("data.csv").run(interactive=False)`.
A model is optional (`LLM_*` in `.env`); without one, deterministic rules plan everything.

## 2. Run the Muse connector locally

```bash
pip install -r requirements.txt
cp .env.example .env            # then edit: API_KEYS, PUBLIC_BASE_URL, ALLOWED_HOSTS
set -a; . ./.env; set +a
uvicorn server.app:app --port 8000
curl localhost:8000/healthz     # {"ok":true}
```

MCP endpoint: `http://localhost:8000/mcp` (Streamable HTTP, `Authorization: Bearer <key>`). REST docs: `/docs`.

### Tools Muse sees

| Tool | What it does |
|---|---|
| `clean_dataset` | One call: scan -> apply every safe fix -> export links. Returns `needs_your_decision` for risky items |
| `scan_dataset` | Load a file, list every problem, change nothing. Returns `dataset_id` |
| `get_cleaning_plan` | Proposed fix per problem, with risk and "applies automatically?" |
| `apply_cleaning_plan` | Apply low/medium items; high-risk items only via `approve_ids` (the user's explicit yes) |
| `apply_cleaning_action` | One specific action (Muse as the planner); risky ones need `confirm=true` |
| `preview_dataset` / `export_dataset` | Look at rows / get time-limited download links (csv, tsv, xlsx, json, jsonl, parquet) |
| `create_upload_url` | One-time URL for files too big to pass inline |
| `delete_dataset` / `list_cleaning_actions` | Delete now / list actions |

Input can be `file_base64`, `csv_text`, a public `file_url` (https only, SSRF-guarded) or an `upload_id`.

## 3. Deploy

Pick one -- both configs are in this folder:

```bash
# Fly.io (fly.toml):  fly launch --no-deploy  ->  fly secrets set API_KEYS=...  ->  fly deploy
# Render   (render.yaml):  push to GitHub -> New + -> Blueprint -> read the API key from the dashboard
```

Or anywhere with Docker (untested -- no Docker here): `docker build -t data-cleaner . && docker run -p 8000:8000 --env-file .env data-cleaner`

Then set the env vars to your real domain (`PUBLIC_BASE_URL`, `ALLOWED_HOSTS`) and **always set `API_KEYS`** (empty means open).
Generate a key with: `python -c "import secrets; print(secrets.token_urlsafe(32))"`.

**Before you submit, run the reviewer test against your live URL:**

```bash
python tools/preflight.py https://your-domain.example YOUR_API_KEY
```

It checks everything Muse's reviewers will touch (public pages, auth, MCP handshake, a real clean + download) and exits 0
only when all pass. **Scaling:** datasets live in process memory, so run one instance/worker; to go multi-instance, move
`server/store.py` and the meters in `server/quota.py` to Redis/S3.

## Security model (what is actually enforced, and tested)

* Bearer-key auth (constant-time compare); each client's datasets are isolated -- "not yours" and "doesn't exist" look identical.
* Unguessable dataset/download/upload tokens; datasets, artifacts and upload slots expire (default 1 h) and can be deleted on demand.
* Only allow-listed formats are parsed (csv/tsv/txt/json/jsonl/xlsx/parquet). No pickle, HDF5, SQLite, HTML/XML, legacy `.xls`;
  no server file paths; row/column/size caps; zip-bomb checks on `.xlsx`; Parquet metadata checked before reading.
* URL fetching: https/443 only, no credentials, every address must be public (DNS is resolved once and the connection is pinned
  to that IP), redirects re-validated, size and time capped.
* The actions are a fixed registry -- no code execution, no user regexes. A model (if you enable one) cannot lower risk, invent
  actions or touch missing columns; and `LLM_ENABLED` is **off** by default with cell values redacted when on.
* Errors return generic messages (stack traces stay in server logs); logs hold ids and counts, never data.
* DNS-rebinding protection on `/mcp` (`ALLOWED_HOSTS`), `nosniff`, `no-store`, strict CSP on the public pages.

## Tests

```bash
pip install -r requirements-dev.txt
pytest                # 89 tests: engine, planner guard rails, server, SSRF, tiers, live MCP client, notebook end-to-end
```

## What is *not* done / not verified (please read)

* **Not deployed and not submitted** -- I can't do either from here. You need a host, a domain, real legal text and to fill the form (`muse/submission.md`).
* **Muse's exact requirements are unverified.** I could not confirm how Muse authenticates to a connector (bearer key vs OAuth), how
  payment entitlement reaches your server, or the review criteria. Check Meta's current developer docs. Auth is one function
  (`server/app.py::_client_id`); entitlement is one function (`server/quota.py::tier_for`).
* **The Dockerfile is untested** (no Docker in the build environment); the app itself was tested via uvicorn + a real MCP client.
* **Scheduled cleaning is not built** -- it is the feature most likely to justify a subscription (see `docs/monetization.md`).
* Long calls: a 200k-row file took ~8 s in the sandbox. Confirm Muse's tool timeout before advertising that size.
* Legal pages are drafts, not legal advice.

See `CHANGELOG.md` for every bug fixed in the original notebook, and `docs/monetization.md` for the earning model.

Built by **Ishan Makkar** -- https://github.com/iishanmakkar/dc-mcp