# Dataset-Pipeline-MCP

An MCP server that turns *"find me a dataset"* into *"find me a dataset AND
a correct, domain-aware preprocessing script for it"* — callable from any
MCP client (Claude Code, Claude Desktop) without leaving your terminal.

## Why this one, specifically

Kaggle ships an official MCP server, and there are several community
HuggingFace/Kaggle MCP connectors already. They cover search, download, and
(in one case) an EDA-notebook prompt. **None of them detect what kind of
data a dataset actually is and generate a matching preprocessing
pipeline** and time-series/sensor data in particular gets silently
treated as generic tabular data by every generic tool, which means wrong
resampling, ignored sensor drift, and no windowing.

This server's actual job is: **search → detect domain → generate a
correct, runnable preprocessing script**, with `time_series_sensor` as a
first-class domain alongside `tabular`, `nlp_text`, `image`, and `audio`.

## Tools

| Tool | Description |
|---|---|
| `search_datasets(query, max_results, include_kaggle)` | Searches HuggingFace Hub (no auth needed) and optionally Kaggle (needs credentials). Annotates every result with a lightweight detected `domain`. |
| `detect_domain(text, tags)` | Classifies free text/tags into a domain, with a confidence score and the signals that were matched — not a black box. |
| `generate_preprocessing_pipeline(dataset_id, domain, source, description_hint)` | Renders a complete, runnable Python script. `domain="auto"` triggers metadata lookup + detection. |
| `find_related_papers(topic, max_results)` | arXiv search for preprocessing/methodology context. |

Plus one resource (`domains://catalog`) and one prompt (`dataset_report`)
to demonstrate full MCP surface coverage, not just tools.

## Repository layout

```
server.py                    <- FastMCP wiring: tools, resource, prompt
core/
  domain_detector.py          <- pure heuristic classifier, fully unit-tested
  templates.py                 <- one preprocessing script generator per domain
connectors/
  huggingface.py               <- public search, optional token
  arxiv_search.py               <- public search, no auth
  kaggle_connector.py           <- optional, requires user-supplied credentials
scripts/
  verify_live.py               <- live end-to-end check against real APIs
tests/                        <- pytest suite, network-independent by design
Dockerfile
.mcp.json                     <- project-level Claude Code config
.env.example
mcp-config.example.json
requirements.txt / requirements-dev.txt
```

`core/` has zero network dependencies by design — it's the part that has to
be correct every time, so it's the part that's cheap to test exhaustively.
`connectors/` is where the world can fail, so every connector fails
*loudly and specifically*, but never *silently* or *fatally*.

## Setup

```bash
git clone https://github.com/pranjalisr/dataset-pipeline-mcp.git
cd dataset-pipeline-mcp
python -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate
pip install -r requirements.txt
```

No credentials are required for basic use — HuggingFace dataset search is
public. Copy `.env.example` to `.env` and fill in `HUGGINGFACE_TOKEN` /
Kaggle credentials only if you want higher rate limits or Kaggle search.
**`.env` is gitignored** — double check with `git check-ignore -v .env`
before committing if you're ever unsure.

## Running generated scripts locally

`generate_preprocessing_pipeline` produces a script that depends on
`pandas`, `numpy`, and `scipy` — these are **not** required to run the MCP
server itself (the server only generates this code as text, it never
imports these libraries), so they're kept in a separate
`scripts/requirements.txt` rather than the top-level one:

```bash
pip install -r scripts/requirements.txt
```

## Running it standalone

```bash
python server.py
```

This starts the server on `stdio`, which is how MCP clients launch local
servers. Running it directly just makes it sit and wait for a client to
connect — use one of the methods below to actually exercise it.

## Verifying live network calls work

The automated test suite intentionally does **not** depend on live
HuggingFace/arXiv/Kaggle access, so it passes even offline. To confirm the
real network calls work end-to-end:

```bash
python scripts/verify_live.py
python scripts/verify_live.py --query "wearable sensor human activity recognition"
```

This calls the tools through the real MCP protocol layer for every
supported domain plus a live Kaggle check when credentials are configured,
and prints a PASS/FAIL summary with the resolved domain and confidence for
each — including a hard failure if a result comes back with zero
confidence, since a generated script with no real domain signal isn't
trustworthy even though it's technically valid Python.

## Interactively testing tools with MCP Inspector

```bash
npx @modelcontextprotocol/inspector python server.py
```

Prints a local URL with a UI listing all tools/resources/prompts where you
can fill in arguments and see live JSON responses.

## Running with Docker

```bash
docker build -t dataset-pipeline-mcp .
docker run -i --rm dataset-pipeline-mcp
npx @modelcontextprotocol/inspector docker run -i --rm --env-file .env dataset-pipeline-mcp
```

## Connecting to Claude Code

This repo includes a project-level `.mcp.json`, so opening it in Claude
Code prompts you to trust the `dataset-pipeline` server automatically. To
wire it up manually instead:

```bash
claude mcp add --transport stdio dataset-pipeline -- python /absolute/path/to/dataset-pipeline-mcp/server.py
```

Once connected, just ask naturally — e.g. *"find me a wearable sensor
dataset for activity recognition and give me a preprocessing pipeline for
it"* — and the model decides which tools to call and in what order; you
never name a tool directly.

## Testing

```bash
pip install -r requirements-dev.txt
pytest tests/ -v
```

## The bug hunt

This project's test suite didn't just check coverage gaps — every bug
below was found by actually running the server against live data, a real
Docker container, or a real Claude Code session, not by reading the code
and assuming it was correct.

1. **Lazy-generator exception escape.** `huggingface_hub.list_datasets()`
   returns a lazy generator — the HTTP request only fires on iteration, so
   wrapping just the *call* in try/except let real API failures escape
   uncaught. Fixed by materializing the generator inside the try block.

2. **HuggingFace's `search` param is a repo-name substring match, not full
   text search.** A natural-language query like `"human activity
   recognition sensor"` returned zero results on a `200 OK` — matching
   datasets existed, but not with that literal string in their name. Fixed
   with a keyword-fallback search plus relevance ranking.

3. **Domain detection was reading empty YAML frontmatter.**
   `dataset_info().cardData` is the README's front matter, which authors
   almost never fill in — the real description lives in the Markdown body,
   a separate fetch (`DatasetCard.load().text`). Detection was running on
   empty input for most real datasets until this was fixed.

4. **Single-keyword overconfidence.** The confidence formula
   (`winning_score / total_score`) reports `1.0` whenever only one domain
   has *any* signal — even a single ambiguous keyword. A wearable *video*
   dataset scored 100% confidence for `time_series_sensor` off the word
   "wearable" alone. Fixed with an evidence floor that dampens confidence
   when total signal strength is thin, verified to leave strong
   multi-signal matches untouched.

5. **Kaggle SDK crash bug.** `import kaggle` calls `sys.exit(1)` internally
   when credentials aren't recognized — and `SystemExit` is **not** caught
   by `except Exception` (it inherits from `BaseException`). A single bad
   Kaggle token could have crashed the entire server process, not just
   failed that one call. Confirmed against the real installed package and
   fixed by explicitly catching `SystemExit` at every Kaggle API boundary.

6. **Image domain had no standalone keyword.** Unlike `tabular`/`nlp`/
   `audio`, which all have their own name as a strong signal, `image` only
   classified correctly by accident (via an unrelated `"x-ray"` match) on a
   real chest X-ray dataset. Fixed by adding `"image"` itself as a signal.

7. **Blank `HUGGINGFACE_TOKEN` breaks auth.** `.env.example` shows
   `HUGGINGFACE_TOKEN=` (blank) as a template — but `os.environ.get()`
   returns `""` for a set-but-empty variable, not `None`, so an empty
   string was passed as a literal bearer token (`Illegal header value
   b'Bearer '`) instead of being treated as "no token." Fixed in the
   HuggingFace API client setup.

8. **Client-side paste artifacts.** A real MCP Inspector session had its
   own placeholder hint text (`query: "wearable sensor"`) submitted
   literally instead of being replaced, polluting search results with
   unrelated matches on the stray word "query." Fixed with input
   sanitization that strips a recognized `<label>:` prefix and one layer
   of fully-wrapping quotes, verified not to touch genuine queries that
   happen to contain those words (`"search query logs dataset"` passes
   through unchanged).

Every fix above shipped with a regression test that reproduces the
original failure, not just a check that the happy path still works.

## Proof it works end-to-end

Beyond unit tests, this was verified through the full real stack: local
stdio, live HuggingFace search across all five domains, live arXiv search,
a live Kaggle call (crash-fixed and confirmed safe on invalid credentials),
a built and running Docker container connected via MCP Inspector, and
finally a real Claude Code session — asked in plain English, with no tool
names given — that correctly chained `search_datasets` →
`generate_preprocessing_pipeline`, found `DiFronzo/Human_Activity_Recognition`,
generated the `time_series_sensor` template, then independently discovered
the template's assumptions didn't match that dataset's real file layout
(separate accelerometer/gyroscope files + a label-segment file, no
timestamps) and rewrote the script to handle it correctly — producing
3,289 labeled windows across 5 balanced activity classes with a proper
subject-wise train/test split.

## Known limitations

- Domain detection is keyword/heuristic-based, not a trained classifier —
  transparent and fast, but a genuinely ambiguous or sparsely-described
  dataset can be misclassified. `detect_domain` always returns its
  confidence and matched signals so a caller can tell when to double-check.
- `search_datasets`'s per-result `domain` field is a lightweight, free
  estimate (id words + whatever description/tags came back in the list
  response) — `generate_preprocessing_pipeline(domain="auto")` does a
  deeper per-dataset fetch and is the one to trust for a real
  classification.
- Kaggle search requires the user's own API credentials
  (`KAGGLE_API_TOKEN` recommended); this server never requests, stores, or
  proxies them beyond reading environment variables. The full success path
  with a genuinely valid token hasn't been verified in this project's own
  testing — only the credential-missing and credential-invalid paths have
  live coverage.
- Preprocessing templates are strong starting points, not final pipelines —
  they're meant to be read and adapted (target columns, window sizes,
  actual file formats), not run blindly in production.