# ICRISAT Data Hub

A lightweight, federation-first data hub for ICRISAT teams, modelled on the
CGIAR Climate Data Hub asset-mapping approach:

```
teams drop files  ──►  inbox (watched)  ──►  ingest pipeline  ──►  catalogue  ──►  MCP server  ──►  any LLM
   (no skills needed)                      (auto: read, profile,               (SQLite + assets.json)
                                           register, flag gaps)
```

Phase 1 supports **CSV, Excel (.xlsx/.xls), and Word (.docx)** files.
External data is registered as **YAML pointers** (federation — the hub points
to data where it lives rather than copying it).

## For teams: how to share data (30 seconds, no skills needed)

**Option A — drop folder.** Copy your file into `data/inbox/` (this can be a
shared network drive or a OneDrive-synced folder). That's it. The pipeline
picks it up automatically, catalogues it, and flags anything it couldn't
figure out for review.

**Option B — web page.** Run `scripts\start_web.cmd` once, open
http://localhost:8010, drag & drop, optionally fill 5 small fields.

**Option C — with metadata.** Drop a tiny sidecar next to your file:

```
yield_trials_2025.csv
yield_trials_2025.meta.yaml     # title, team, owner, description, tags, ...
```

or one `metadata.yaml` for a batch of files. Incomplete uploads get a
pre-filled `*_metadata_review.yaml` written next to the processed file —
complete it and drop it back in the inbox (or edit via the review CLI).

## Setup

```
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
```

## Running the pieces

| Component | Command | Purpose |
|---|---|---|
| Inbox watcher | `scripts\start_watcher.cmd` or `python -m hub.watcher` | auto-ingest dropped files |
| One-off scan | `python -m hub.ingest.pipeline --scan` | scheduled/manual ingest |
| Web upload + catalogue | `scripts\start_web.cmd` or `python -m hub.web.app` | http://localhost:8010 |
| MCP server (stdio) | `python -m hub.mcp_server.server` | local LLM agents |
| MCP server (HTTP) | `python -m hub.mcp_server.server --http` | http://localhost:8100/mcp for other machines |
| Validate YAML pointers | `python -m hub.sources` | check `sources/*.yaml` |

All `python -m hub...` commands run from the project root with
`PYTHONPATH=src` (the scripts set this for you).

## Connecting an LLM (MCP)

**Claude Desktop / opencode (stdio)** — add to your MCP config:

```json
{
  "mcpServers": {
    "icrisat-data-hub": {
      "command": "C:\\Users\\HDesalegn\\ICRISAT\\icrisat-data-hub\\.venv\\Scripts\\python.exe",
      "args": ["-m", "hub.mcp_server.server"],
      "env": { "PYTHONPATH": "C:\\Users\\HDesalegn\\ICRISAT\\icrisat-data-hub\\src" }
    }
  }
}
```

**Other machines/LLMs (HTTP)** — start with `--http` and point clients at
`http://<host>:8100/mcp`.

### MCP tools

| Tool | What it does |
|---|---|
| `list_assets` | List catalogue, filter by status/team/type/tag |
| `search_assets` | Keyword search over titles, descriptions, column names, document text |
| `get_asset` | Full metadata + per-sheet schema (columns, dtypes, samples) |
| `preview_asset` | First N rows of a CSV/Excel asset |
| `query_asset` | Read-only SQL (DuckDB) over a CSV/Excel asset |
| `read_document` | Extracted text of a Word asset |
| `list_sources` | Federated external sources (YAML pointers) |
| `get_catalog_summary` | Portfolio stats (counts by status/type/team/access) |

## YAML source pointers (federation)

Each external dataset/API/database gets one small YAML in `sources/` — see
`sources/README.md` and `sources/examples/`. The hub registers and exposes
these through the MCP server without copying the data.

## GitHub Pages catalogue app

A static dashboard lives in `docs/` and is served at
https://henokd11.github.io/icrisat-data-hub/. It reads `docs/data/*.json` —
regenerate and commit them after the catalogue changes:

```
set PYTHONPATH=src
.venv\Scripts\python -m hub.pages
git add docs/data && git commit -m "Refresh catalogue app data" && git push
```

## Layout

```
config/hub.yaml          hub-wide settings (paths, vocab, ports)
data/inbox/              drop zone (watched)
data/processed/          ingested files, organised by team/month
data/failed/             rejected files + .error.txt
data/catalog/            catalog.db (SQLite) + assets.json (exported catalogue)
sources/                 YAML pointers to federated external data
src/hub/                 pipeline, watcher, web app, MCP server
tests/                   pytest suite
```