dashai-mcp

An MCP server for dashAI, the open source Machine Learning workbench led by the University of Chile (FCFM), built by students of DCC UChile and UTFSM, with CENIA and IMFD.

Unofficial and independent. This is a third-party project. It is not affiliated with, endorsed by, or maintained by the dashAI project or the institutions that develop it.

It gives an agent the same surface dashAI gives a person through its GUI: look at datasets, see which models are available, train, follow queued work, and read the metrics.

"Train a random forest on dataset 3 predicting 'species' and tell me the F1"

Status

v0.2.2 — verified against a running dashAI 0.9.7.post1. Deterministic tests (including the predict two-step path) plus live runs: dashai_train_model → dashai_job_status → dashai_get_run with metrics, and dashai_predict → dashai_job_status finished.

Live tabular run (seed students)

Public dashAI seed only. Target placement_status (~83% majority).
exam_score was left out of the inputs (it leaks the label). Split 70/15/15. Goal metric: BalancedAccuracy — Accuracy and F1 on the majority class are traps.

The dummy wins F1 by always answering the majority class. The linear model beats the forest on the metrics that actually measure separation. Each row is its own 70/15/15 draw (not the same test rows) — still enough to stop treating the forest as the default. If a client reports only F1 here, it is lying.

dashai_predict on the finished forest run returned prediction_id and the job finished. That scores the same seed dataset the model was trained on, not a held-out file — do not read it as a generalization check. dashai_get_prediction returns {n, n_classes, class_counts} only.

Live image run (seed cifar10-subset)

Public seed: 200 images, frog vs truck (100/100). LeNet5ImageClassifier, CPU, 32×32. Split 70/15/15 → test n=30. Chance is 0.5.

*Accuracy, not BalancedAccuracy — on the one-class val/test they collapse. Chance on this seed is 0.5. Dummy tabular does not apply (image task).

The image path works. The 0.867 is not a result: dashAI defaulted shuffle=False, so val/test were 30 trucks and zero frogs. MCC 0 is sklearn on a one-class split. After this server sent shuffle=true + stratify=true, both classes are in val/test and MCC is no longer 0. Train hits 1.0 (140 images memorized). Val 0.767 is the honest-ish number; test 0.900 is a 30-row lottery. Do not publish a leaderboard line.

Verifying against a live instance surfaced gaps between dashAI's documentation and its actual behaviour. Each one has its own regression test. A sixth — sequential splits with shuffle=False — only showed up live because MCC came back 0 next to a moving Accuracy. dashai_predict sending run_id to PredictJob — only showed up live (KeyError: 'prediction_id') because there was no predict test.

Install

pip install dashai-mcp
# or: uv pip install dashai-mcp

In your MCP client configuration:

{
  "mcpServers": {
    "dashai": {
      "command": "dashai-mcp"
    }
  }
}

dashAI has to be running separately (dashai, or the desktop app). It is looked up at http://localhost:8000 by default.

Tools

Six things dashAI's documentation (or defaults) get wrong

Found by running against a real instance. If you are writing a client for this API, these will bite you:

The component registry also has 13 types, not the four the documentation suggests: Task, GenerativeTask, Model, GenerativeModel, DataLoader, DatasetSource, Metric, Optimizer, Job, LocalExplainer, GlobalExplainer, Explorer, Converter.

And GET /run/{id} returns split_indexes with the full list of indices: on a 10,000-row dataset that is 59 KB, 99% of the response. This server replaces it with the per-split counts, bringing the response down to ~1 KB.

Three design decisions

1. Ten tools, not 142

dashAI exposes 142 REST endpoints. Generating one tool per endpoint is mechanical and it is a mistake: a model with 140 tools burns context reading the catalogue and chooses worse. These ten cover the actual working path.

2. dashai_train_model collapses three calls

In the raw API, training is a chained sequence:

POST /model-session/   → creates the experiment
POST /run/             → creates the run
POST /job/             → enqueues the ModelJob

With required fields the GUI fills in on its own and that are undocumented — plot_history_path, plot_slice_path, plot_contour_path, plot_importance_path. On top of that, splits travels as a JSON string, not an object, even though dashAI's documentation shows it as an object: the backend's Pydantic schema declares it str. That kind of detail is exactly what makes an agent fail against the raw API.

Here it is a single call, and it does not block: training can take hours, so it returns the job_id immediately and progress is polled with dashai_job_status.

dashai_predict does the same for the two-step GUI path: POST /predict/ then POST /job/ with prediction_id.

3. No tool deletes anything

dashAI's API has no authentication — checked endpoint by endpoint. That is coherent for something local-first, but it means there is no barrier between a misread sentence and an irreversible DELETE /dataset/{id}. Deleting is done from the GUI, looking at what is being deleted.

For the same reason, the server refuses to point at a non-local host:

DASHAI_BASE_URL points to 'ml.example.com', which is not local, and dashAI's API
has no authentication: exposing it to the network leaves the backend open to
anyone who can reach it.

This can be disabled on purpose with DASHAI_ALLOW_REMOTE=1, if the target is protected some other way.

Configuration

dashai_get_prediction needs pyarrow in the MCP process to turn the Arrow file into class counts (pip install 'dashai-mcp[counts]', or install the MCP into the same env as dashAI). Without it the tool still returns status and refuses to dump rows.

Development

python -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest tests/ -q

The tests stub the HTTP responses with respx: they need neither a dashAI instance nor credentials. They test the contract — which calls are made, in what order, with what body, and what the agent is told when something fails.

Verifying against a live instance

A client tested only against stubs is a hypothesis. scripts/smoke_live.py exercises every tool against a real running dashAI, through the same code paths an agent uses:

python scripts/smoke_live.py            # read-only tools
python scripts/smoke_live.py --train    # + a real train -> predict loop

The --train loop creates a model session, a run and a prediction on the target instance — point it at a scratch instance, not a production one. Exit code 0 only if every exercised tool worked. This is how each release gets verified; the version it was last run against is what dashai_server_info reports under compatibility.verified_against.

API compatibility check

dashAI exposes no version endpoint, so dashai_server_info reads the instance's openapi.json and compares the API surface against what this server actually calls: are the endpoints still there, and does POST /model-session/ require fields this server does not send? The verdict comes back as compatibility.status — ok, mismatch (with the exact differences named) or unknown (schema unreadable; everything else may still work). The case it exists for is real: dashAI's development branch already adds an evaluation_strategy field to model sessions.

A note on the SDK

Requires the MCP Python SDK 2.x. Version 2.0 removed mcp.server.fastmcp; it is now mcp.server.mcpserver.MCPServer, and annotations are ToolAnnotations objects instead of dictionaries. Most tutorials still show the 1.x API.

License

MIT, same as dashAI. See the note at the top on affiliation.