Nightingale's Hoard

Native data and calculation modules

Nightingale incorporates all of Laplace's arithmetic, symbolic math, units, dates, statistics, SQL-over-files, charts and calculation history. Data and Calculation are independent switches, both OFF on first installation or migration. Start the app explicitly from Hoard Hub, then enable only what you need. Calculation works without loading the data workbench or DuckDB. Full native interfaces live under /data/ and /calculation/; one server owns both.

The static MCP catalog contains 38 native tools plus two module controls. Existing Nightingale tools retain their names; the three collisions are laplace_data_list, laplace_data_query and laplace_data_chart. Existing native data directories can be adopted in place through data/modules.json without copies or format changes. See architecture and provenance.

The data workbench for a local language model: ingest a messy file, clean it with versioned steps you can undo, check it against rules you define, chart and dashboard it, train a quick model, and ask it a question — all on your own machine, and all reachable by an assistant through MCP exactly the way you'd use it yourself.

Everything stays local: DuckDB for the data, SQLite for metadata (sources, versions, quality rules, charts, dashboards, models, the analysis log), no accounts, no network calls except to a file URL you gave it.

Part of the Hoard family (see faustus-plugin.json). Calculation preserves Laplace's read-only file catalog alongside Nightingale's reversible workbench.

Related MCP server: Chart Canvas MCP Server

Why this exists

Cleaning a real dataset is normally a one-way trip: you overwrite the file, or you keep five half-named copies "just in case." Nightingale keeps the trip reversible. Every cleaning step — filter, rename, cast a type, fill a null, derive a column, join two datasets, twenty-one kinds in total — creates a new, real, materialized version. Undo is instant. A step you didn't like never touched the version before it. And if the source file changes later, "refresh" re-ingests it and replays your whole recipe on the fresh data automatically, stopping cleanly if a step no longer applies.

What's implemented

For an imported .xlsx, workbook-preserving export can write changed dataset cells into a copy of the original workbook while retaining unchanged cells and workbook features. One request can update several imported tabs in that copy. It includes a source inventory and lineage receipt; formulas cannot be overwritten, and validation rules are preserved but not enforced. See the workbook export guide for the API, MCP, and supported boundaries.

The data grid formats every cell by its DuckDB column type and the UI's own language setting — dates and timestamps render as dates (a timestamp at exact midnight drops its time-of-day), and numbers get thousands separators and sensible precision (2 decimals for DECIMAL columns, up to 4 significant digits for other numeric types) — while the exact raw value stays one hover away (a tooltip) and a double-click away (copies it to the clipboard).

Lab, in the UI

Use cases

• Drop in a messy CSV a colleague sent you, watch the workbench guess types and encodings, clean it with a handful of steps, and export something you'd actually hand to someone else.

• Ask an assistant to "look at this data and tell me what's off" — it can profile, run quality checks, and cite the exact N-000123 log entry for whatever it found.

• Keep a recipe against a source file that updates weekly: refresh re-applies every step to the new data in one call, runs saved quality rules, and reports failing rules with sample rows. Pass check_quality=false to skip the checks for a particular refresh.

• Train a same-afternoon model on a dataset you just finished cleaning, without leaving the app or writing a notebook.

Quick start

Requires Python 3.11+ and Node 22+ (Node only to build the client).

git clone <this repo> nightingale-hoard
cd nightingale-hoard
python -m venv venv
venv/bin/pip install -r requirements.txt      # Windows: venv\Scripts\pip install -r requirements.txt
venv/bin/pip install -r requirements-lab.txt  # optional: Optuna/SHAP/XGBoost/LightGBM for the Lab package
npm install
npm run build
venv/bin/python -m nightingale --demo         # Windows: venv\Scripts\python -m nightingale --demo

Open http://127.0.0.1:5189 and enable Data or Calculation in Functions. --demo seeds three invented datasets in Data when enabled (a messy Spanish-format sales CSV, a customer table, a seasonal sensor time series with a few injected anomalies) into a separate data-demo/ folder, so it never touches real data. Drop --demo for a clean workbench.

• python scripts/launch.py starts the app on a free port and opens the browser.

• python scripts/dev.py runs uvicorn with --reload plus the Vite dev server (proxying /api).

Configuration (environment)

Access from your phone (behind a tunnel)

The server binds 127.0.0.1 and only answers requests whose Host is localhost, 127.0.0.1 or [::1]. To reach it from your phone through a tunnel, list the extra host names in NIGHTINGALE_ALLOWED_HOSTS, comma-separated, exact names or *.suffix: NIGHTINGALE_ALLOWED_HOSTS=my-pc.example,*.ts.net. Port and letter case are ignored, and the Origin of API calls must resolve to one of those hosts too. Once opened through the tunnel, the browser offers to install it as a PWA.

Connect to Faustus

Load faustus-plugin.json in Faustus to discover the static MCP catalog. Start the app explicitly from Hoard Hub and enable the requested module. The bridge calls the running app's HTTP API, the same as the browser UI, and never opens the database itself.

API

All JSON; errors are { "error": "..." }.

• GET /api/health, GET /api/status

• GET /api/sources, POST /api/sources/ingest (kind: "hoard" with preset or app + tool + args + list_path), GET /api/sources/hoard-presets

• GET /api/datasets, POST /api/datasets/{name}/refresh, DELETE /api/datasets/{name}, GET /api/datasets/{name}/dependents

• GET /api/datasets/{name}/{profile,preview,lineage,recipe,correlation}

• POST /api/datasets/{name}/{transform,undo,redo,join-preview}

• POST /api/query

• GET/POST /api/datasets/{name}/quality, POST /api/datasets/{name}/quality/run, DELETE /api/quality/{id}

• GET/POST /api/charts, GET/DELETE /api/charts/{id}

• GET/POST /api/dashboards, GET /api/dashboards/{id}, POST /api/dashboards/{id}/items\n- GET /api/dashboards/action-schemas, POST /api/dashboards/{id}/{actions,undo,redo}, GET /api/dashboards/{id}/history - typed, reversible edits (docs/DASHBOARD-ACTIONS.md)

• GET/PUT /api/dac/semantic, POST /api/dac/semantic/validate, GET /api/dac/semantic/suggest?dataset=

• GET/POST /api/dac/dashboards, GET/PUT/DELETE /api/dac/dashboards/{slug}, POST /api/dac/dashboards/{slug}/{rename,validate,render,export}, GET /api/dac/dashboards/{slug}/{history,diff}, POST /api/dac/dashboards/import — see docs/DASHBOARDS_AS_CODE.md

• POST /api/models/{train,cluster,pca,anomaly,forecast}, GET /api/models

• POST /api/export

• GET /api/log, GET /api/log/{id}

• POST /api/ask, GET /api/ask/available

• GET /api/agent/tools (catalog + instructions), POST /api/agent/call (Bearer token from <DATA_DIR>/mcp-token)

MCP tools

See docs/MCP.md for the Data module reference (18 tools, data_ingest through data_ask) and a minimal walk-through. The shipped instructions tell the assistant to preview a transform before applying it when the effect isn't obviously safe, to never assume a dataset name (data_list first), and to cite the analysis-log id (N-000123) when reporting a result back. Calculation adds 20 native tools documented in its reference; three collisions use the laplace_ prefix listed above. nightingale_modules and nightingale_set_module complete the 40-tool catalog.

Tests

venv/bin/python -m pytest -q     # Windows: venv\Scripts\python -m pytest -q

Tests cover the workbench engine (every ingestion path, every transform step, versioning and undo/redo/replay), quality rules, charts and dashboards, models (including the id-like-column exclusion, the duplicate-timestamp forecasting fix, and that both the default new-dataset output and the write_to="new_version" opt-in preserve every copied column's exact type), dataset deletion (dependents check, cascade, force), the HTTP API, agent tools through /api/agent/call, the request guard, the PWA endpoints, and a subprocess end-to-end test through the MCP stdio bridge.

License

MIT — Luis María Salete Cuartero.

Shared services (HoardLink 0.8.1)

URL imports use the shared Web client and the Hub catalogue bridge. Import and analysis behavior remains in the app.

The vendored copy is maintained by HoardLink’s sync script. Windows validation and the family service contract are documented in HoardLink’s docs/commons/windows-validation.md and docs/commons/services.md.

Faustus panel

A «Faustus» button opens the family's shared assistant panel without leaving the app. Faustus keeps the conversations (also listed in Faustus Studio), runs the agent with the globally selected model and uses this app's own tools. Each turn carries the dataset, dashboard, dashboard-as-code, Lab model or log entry on screen; unsaved YAML of a dashboard-as-code or the semantic layer, and the new-chart and new-rule forms. After a turn the lists and the open view refresh in place, without discarding what you were typing. The panel comes from Hoard Link (install_spa) and talks to Faustus through this app's relay, so no token reaches the browser.

Owned data and compatibility

Native module data can be relocated into data/native/<module>/data, with per-module configuration beside it. Writer fiction uses the owning desktop profile writers-hoard/fiction/data. Move complete data trees, including SQLite WAL/SHM, media, downloaded models and browser sessions; update saved absolute paths before enabling a module. The compatibility manifests in compatibility/<id>/faustus-plugin.json preserve native IDs, tool schemas and tokens without a retired checkout. Add each merged application's compatibility directory to Hub roots before the general repository root. Discovery and compatibility bridges never start an application. All modules remain OFF during relocation.

Legacy in-place adoption scripts are optional import helpers; keeping the old repository is unnecessary once relocation and route updates have been verified.