# RefineData MCP Server

RefineData MCP is a [Model Context Protocol](https://modelcontextprotocol.io) server
that gives AI agents (Claude Code, Cursor, ChatGPT) a toolbox for data preparation,
anonymization, analysis, and visualization. It leans on **Polars** and **DuckDB** so
the heavy tools stream data out-of-core instead of loading everything into RAM.

## Tools

### Analysis & Profiling

| Tool | Purpose | Engine | Scales beyond RAM? |
|------|---------|--------|--------------------|
| `analyze_data` | Row counts, schema, numeric stats | DuckDB | ✅ out-of-core |
| `profile_data` | Per-column null rates, unique counts, sample values, date ranges | DuckDB | ✅ out-of-core |
| `sample_data` | Preview N rows (head/tail/random/stratified) | DuckDB | ✅ out-of-core |
| `simplify_quant_data` | Percentiles, skew, correlations | Polars (lazy/streaming) | ◐ streamed |
| `validate_schema` | Check columns and types against an expected schema | DuckDB | ✅ out-of-core |

### Cleaning & Transformation

| Tool | Purpose | Engine | Scales beyond RAM? |
|------|---------|--------|--------------------|
| `process_data` | Dedup + missing-value handling | Polars (lazy/streaming) | ◐ streamed |
| `convert_format` | CSV ↔ Parquet ↔ JSONL with compression options | Polars (lazy/streaming) | ◐ streamed |
| `merge_datasets` | Join two datasets on key columns | DuckDB | ✅ out-of-core |
| `detect_outliers` | IQR or z-score outlier extraction | DuckDB | ✅ out-of-core |

### ML / NLP

| Tool | Purpose | Engine | Scales beyond RAM? |
|------|---------|--------|--------------------|
| `filter_toxic_data` | Drop toxic rows (batched inference) | Detoxify | ✗ in-memory |
| `anonymize_data` | Mask PII in text columns | Presidio NLP | ✗ in-memory |

### Generation & Database

| Tool | Purpose | Engine | Scales beyond RAM? |
|------|---------|--------|--------------------|
| `generate_synthetic_data` | Schema-driven synthetic dataset generation | Polars | ✗ in-memory |
| `query_db` | Run SQL on PostgreSQL/SQLite → file | DuckDB | ✅ out-of-core |
| `scrape_web_data` | Extract the main HTML table from a URL | requests + BeautifulSoup | ✗ in-memory |

### Session Management

| Tool | Purpose |
|------|---------|
| `load_dataset` | Load a file into the in-process cache, returns a handle |
| `release_dataset` | Free a cached dataset from memory |
| `list_datasets` | List all currently loaded dataset handles |

### Visualization

| Tool | Purpose | Engine | Scales beyond RAM? |
|------|---------|--------|--------------------|
| `visualize_dataset` | Heatmap/scatter/boxplot/bar → image | Matplotlib/Seaborn | ✗ in-memory |

> ◐ = the streamable parts of the plan run in chunks; the result is materialized.
> The PII/toxicity/scraping/viz tools load the working set into memory by design.

## Installation

Core install (only `analyze_data` + `query_db`) is lightweight. Heavy features are
**optional extras**, so you don't pull in PyTorch/spaCy/Matplotlib unless you need them.

```bash
python -m venv venv && . venv/Scripts/activate   # Windows
# source venv/bin/activate                       # Linux/Mac

pip install -e .                # core (DuckDB + Polars)
pip install -e ".[scraping]"    # + web scraper
pip install -e ".[viz]"         # + visualizer
pip install -e ".[pii]"         # + PII anonymizer
pip install -e ".[toxicity]"    # + toxicity filter
pip install -e ".[all]"         # everything
```

If you call a tool whose extra isn't installed, it returns a clear message telling you
which `pip install` to run.

The PII tool also needs a spaCy model:

```bash
python -m spacy download en_core_web_sm
```

> Supported Python: **3.11+** for core/`[scraping]`/`[viz]`; the `[pii]`/`[toxicity]`
> extras are pinned to **3.11–3.12** until spaCy/PyTorch ship newer wheels.

## Configuration for MCP clients

Once published, run with zero local setup via [`uv`](https://docs.astral.sh/uv/) —
no clone, no venv:

```json
{
  "mcpServers": {
    "RefineDataMCP": {
      "command": "uvx",
      "args": ["--from", "refinedata-mcp[all]", "refinedata-mcp"]
    }
  }
}
```

Drop `[all]` (or swap in `[scraping]`, `[viz]`, etc.) to install only the
extras you need. `uvx` fetches the package into an isolated, cached
environment on first run.

### Running from a local checkout

```json
{
  "mcpServers": {
    "RefineDataMCP": {
      "command": "refinedata-mcp",
      "cwd": "/absolute/path/to/refinedata-mcp"
    }
  }
}
```

`refinedata-mcp` is installed as a console script. Alternatively use
`"command": "python", "args": ["-m", "refinedata_mcp"]`.

## Environment variables

| Variable | Default | Effect |
|----------|---------|--------|
| `REFINEDATA_LOG_LEVEL` | `INFO` | Log verbosity (logs go to stderr) |
| `REFINEDATA_AUDIT_LOG` | unset | Path to a JSONL audit log file (tool name, status, duration) |
| `REFINEDATA_MAX_FILE_SIZE_MB` | `1024` | Reject in-memory loads larger than this (0 = off) |
| `REFINEDATA_WORKDIR` | unset | Sandbox: restrict all file reads/writes to this dir |
| `REFINEDATA_ENABLED_TOOLS` | `all` | Comma-separated tool names to expose, or `all` |
| `REFINEDATA_MAX_CONCURRENT` | `0` | Max simultaneous tool calls (0 = unlimited) |
| `REFINEDATA_HTTP_TIMEOUT` | `15` | Web scraper request timeout (seconds) |
| `REFINEDATA_ALLOW_PRIVATE_URLS` | `false` | Allow scraping private/loopback IPs (SSRF override) |
| `REFINEDATA_DB_READ_ONLY` | `true` | Attach external databases read-only |
| `REFINEDATA_DB_TIMEOUT` | `60` | Best-effort DB query timeout (seconds, 0 = off) |
| `REFINEDATA_PII_LANGUAGE` | `en` | Presidio scan language |
| `REFINEDATA_PII_BATCH_SIZE` | `256` | Cells per spaCy `nlp.pipe()` batch in `anonymize_data` |
| `REFINEDATA_DETOXIFY_MODEL` | `original` | Detoxify model variant |

## Security notes

- The web scraper **blocks requests to private/loopback/link-local/metadata IPs**
  (SSRF guard). Override only for trusted local use with `REFINEDATA_ALLOW_PRIVATE_URLS=true`.
- `query_db` executes the SQL you pass it and attaches databases **read-only** by default.
- Set `REFINEDATA_WORKDIR` to confine file access to a single directory.
- Set `REFINEDATA_ENABLED_TOOLS` to expose only a subset of tools in multi-tenant setups.
- This server is built for a **single trusted user** (your agent). Do not expose it
  to untrusted input without adding allow-lists.

## Testing

```bash
pip install -e ".[dev]"
python -m pytest
```

Tests for optional features `importorskip` their dependencies, so the suite runs
(skipping what isn't installed) at any install level. See [ROADMAP.md](ROADMAP.md)
for what's done and what's next.