ICRISAT Data Hub

A lightweight, federation-first data hub for ICRISAT teams, modelled on the CGIAR Climate Data Hub asset-mapping approach:

teams drop files  ->  inbox (watched)  ->  ingest pipeline  ->  catalogue  ->  MCP server  ->  any LLM
   (no skills needed)                    (auto: read, profile,              (SQLite + assets.json)
                                         register, flag gaps)

Phase 1 supports CSV, Excel (.xlsx/.xls), and Word (.docx) files. External data is registered as YAML pointers (federation: the hub points to data where it lives rather than copying it).

For teams: how to share data

Every dataset goes upload → describe → curator review → published. Nothing is public until the curator for its category approves it.

1. Web page (recommended, anyone with an email). Sign in, open Upload, drop one or more files, pick a category, fill the short form (description, team, licence, who can download) and press Submit for review. The curator for that category is emailed. Several files uploaded together share one description and become separate datasets. Missing details? Use Upload now, add details later and finish them on the dataset's Edit page.

2. Shared drop folder (ICRISAT network). Copy files into data/inbox/. Add yield_trials_2025.meta.yaml next to a file (or one metadata.yaml for a batch) with title, description, team, owner, owner_email, category, license, access and it goes straight to the review queue; otherwise it waits as Needs details with a pre-filled *_metadata_review.yaml to complete.

Statuses: Needs details → In review → Published (or Returned with the curator's comment → fix → resubmit). Published datasets can be Withdrawn.

Who can download (set on upload, changeable on the dataset's Share panel):

Descriptions of published datasets are always visible (CGIAR Open & FAIR policy); only the files are gated. The public GitHub Pages dashboard and the HTTP MCP server show published + open datasets only.

Roles: admins manage the Admin page — add categories, add/remove curators per category (paste several emails at once) and other admins; changes apply immediately and are logged. config/hub.yaml → access_control holds the bootstrap admins (not removable from the UI, so nobody is locked out) and any curators you prefer to keep in config. A category with no curators goes to the admins.

Scripts / API: My datasets → API tokens creates a personal token (shown once, stored hashed, revocable). Then:

curl -H "Authorization: Bearer $HUB_TOKEN" -F files=@trials.csv      -F category="Soil & Agronomy" -F team="Soil Intelligence"      -F description="On-farm trials 2025" -F license=CC-BY-4.0 -F access=open      https://<hub>/upload

returns JSON per file (dataset id, status, missing fields). The same header works for GET /api/assets and GET /datasets/<id>/download.

Access report: every download is logged; owners and curators see counts in the dataset's History and can download an access report CSV (grants, requests, downloads).

Related MCP server: Pipetable

Setup

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Running the pieces

All python -m hub... commands run from the project root with PYTHONPATH=src (the scripts set this for you).

Connecting an LLM (MCP)

Claude Desktop / opencode (stdio) - add to your MCP config:

{
  "mcpServers": {
    "icrisat-data-hub": {
      "command": "C:\\Users\\HDesalegn\\ICRISAT\\icrisat-data-hub\\.venv\\Scripts\\python.exe",
      "args": ["-m", "hub.mcp_server.server"],
      "env": { "PYTHONPATH": "C:\\Users\\HDesalegn\\ICRISAT\\icrisat-data-hub\\src" }
    }
  }
}

Other machines/LLMs (HTTP) - start with --http and point clients at http://<host>:8100/mcp.

MCP tools

YAML source pointers (federation)

Each external dataset/API/database gets one small YAML in sources/ - see sources/README.md and sources/examples/. The hub registers and exposes these through the MCP server without copying the data.

Deploy online

The repo is deploy-from-GitHub ready (Dockerfile included). Recommended host: Railway (persistent volume, permanent domain).

1. railway.app -> New Project -> Deploy from GitHub repo -> icrisat-data-hub.

2. Volume mounted at /data — raw uploads and the SQLite catalogue live there; back it up (e.g. nightly copy of /data/catalog/catalog.db).

3. Sign-in (WorkOS AuthKit, free tier): create a WorkOS project -> Authentication -> enable Email + Magic Auth (and Google/Microsoft if wanted) -> Redirects -> add https://<your-domain>/auth/callback.

4. Variables:

   Variable	Value
   WORKOS_API_KEY, WORKOS_CLIENT_ID	from the WorkOS dashboard
   HUB_BASE_URL	https://<your-domain>
   HUB_SECRET_KEY	long random string (signs session cookies)
   SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD, SMTP_FROM	any SMTP relay, for review/access emails (optional — logged if unset)
   GITHUB_TOKEN, GITHUB_REPO, GITHUB_BRANCH	optional: push the public dashboard JSON after each publish

5. Settings -> Networking -> Generate Domain, then share https://<your-domain>.

HUB_DEV_LOGIN=1 (used by scripts\start_web.cmd) gives a password-less email login for local development only; it is ignored when WorkOS is set.

GitHub Pages catalogue app

A static dashboard lives in docs/ and is served at https://henokd11.github.io/icrisat-data-hub/. It reads docs/data/*.json. A GitHub Action regenerates them automatically whenever the catalogue or the YAML pointers change on main; to refresh by hand:

set PYTHONPATH=src
.venv\Scripts\python -m hub.pages
git add docs/data && git commit -m "Refresh catalogue app data" && git push

Layout

config/hub.yaml          hub-wide settings (paths, vocab, ports)
data/inbox/              drop zone (watched)
data/processed/          ingested files, organised by team/month
data/failed/             rejected files + .error.txt
data/catalog/            catalog.db (SQLite) + assets.json (exported catalogue)
docs/                    GitHub Pages app (dashboard + data/*.json)
sources/                 YAML pointers to federated external data
src/hub/                 pipeline, watcher, web app, MCP server, pages exporter
tests/                   pytest suite