# datamgr — dataset registry for empirical research

A lightweight registry that catalogs your research datasets so an AI (or you)
can discover them without reading every file. Each dataset is one folder with a
`manifest.yaml`; a flat index (`_registry.json`) makes search cheap no matter
how many datasets you have.

> Architecture: one copy of logic in `datamgr.core`; the `dm` CLI and the MCP
> server are both thin front-ends over it.

## Install (editable)

```bash
cd data-manage
pip install -e .
```

## Quick start

```bash
# 1. point datamgr at a central warehouse (created if missing)
dm init-root D:/datasets

# 2a. import a raw data folder -- harvests variable names + labels from a .dta,
#     copies the file into a new clean-id folder under the warehouse
dm import "原始数据包" --id csmar_annual --name "CSMAR 上市公司年度财务"

# 2b. (or) scaffold an empty dataset and fill the manifest by hand
dm init csmar_annual --name "CSMAR 上市公司年度财务"

# 3. rebuild the index
dm refresh

# 4. discover
dm list                          # all datasets, one line each
dm show csmar_annual             # full manifest, rendered
dm search 资产负债率               # fuzzy-search across all variables/labels
dm variables --role control      # list all control variables available
dm variables --label 占比          # filter variables by label
dm stats csmar_annual --refresh  # compute rows/cols/n_firms, cache to manifest
dm verify                        # validate manifests + check file paths exist

# 5. pull data into an analysis working directory (backup + provenance)
dm export csmar_annual control_vars --to D:/my_paper
#   -> D:/my_paper/data/*.dta        (copied files, ready for `use "data/xxx.dta"`)
#   -> D:/my_paper/data_sources.txt  (where each file came from, for traceability)
```

## Commands

| command | purpose |
|---|---|
| `dm init-root [PATH]` | set / show the central warehouse location |
| `dm init <id>` | scaffold an empty dataset folder + manifest template |
| `dm import [<folder>] [--id X]` | build a manifest by harvesting variables from a `.dta`/`.csv`; with `--id` it copies data into a clean-id folder |
| `dm refresh` | rebuild the `_registry.json` index (reads manifests only, never data files) |
| `dm list [--tag T]` | list all datasets, one line each |
| `dm show <id>` | print a dataset's full manifest |
| `dm search <query> [--tag T]` | fuzzy-search datasets by variable name/label, name, description, tag |
| `dm variables [--role R] [--label L]` | list all variables across the index (for picking controls/instruments) |
| `dm stats <id> [--refresh]` | show cached stats, or recompute from the data file |
| `dm export <id...> --to <dir>` | copy dataset files into a working dir + record provenance |
| `dm verify` | validate manifests + check referenced files exist |


## The manifest (`manifest.yaml`)

This is the heart of the system. Fields:

| field | purpose |
|---|---|
| `id` | dataset id, must equal the folder name |
| `name` | human-readable name |
| `description` | free text — searchable |
| `version`, `source` | provenance |
| `unit_of_observation` | granularity, e.g. `公司-年` |
| `time_span` | `[start, end]` years |
| `identifiers` | `{id, time}` — fed straight to Stata `xtset` |
| `variables` | list; each has `name/label/type/role` |
| `tags`, `dependencies` | search facets & lineage |
| `files` | relpaths for `raw/ pipeline/ processed` |
| `stats` | cached summary (rows, cols, ...), computed at refresh |
| `notes` | any gotchas /口径变更 |

**`variables[].role`** ∈ `{id, time, x, y, control, weight, instrument, other}`
is the key signal that lets an AI judge "can this dataset run the regression I
want", which plain variable names can't convey.

## Why it stays fast as datasets grow

- Search reads the in-memory index, never the folder tree at query time.
- The index stores only search-relevant fields (names, roles, tags, spans) — KB
  per dataset. A full manifest is loaded only by `dm show` / `get_dataset`.
- Dataset files (`.dta`, etc.) are read **only** when you explicitly refresh
  stats. `list`/`search`/`show` never touch them.

## Layout

```
src/datamgr/
  config.py        warehouse root resolution
  core/
    manifest.py    schema + load/validate/save
    registry.py    scan -> _registry.json index
    search.py      fuzzy search over the index (rapidfuzz)
    stats.py       compute rows/cols/n_firms from a .dta/.csv (explicit --refresh only)
    importing.py   harvest variables from a .dta/.csv -> manifest scaffold
    exporting.py   copy data files to a working dir + provenance txt
  cli/main.py      `dm` command
  mcp/server.py    read-only MCP server for AI discovery (stdio)
```

## MCP server (AI discovery)

A read-only MCP server lets an AI client (e.g. zcode) discover and inspect
datasets without touching data files. Exposes 5 tools over stdio:

| tool | purpose |
|---|---|
| `list_datasets(tag)` | all datasets (id/name/vars/rows/span/tags) |
| `search_datasets(query)` | fuzzy search across variables/labels/names |
| `list_variables(role, label)` | flat variable list for picking controls |
| `get_dataset(id)` | full manifest incl. **absolute file paths** (locate the .dta on disk) |
| `get_dataset_stats(id)` | cached stats only (never reads data files) |

Run it directly, or register with an MCP-aware client:

```bash
# direct
python -m datamgr.mcp.server
# or via entry point
datamgr-mcp
```

ZCode workspace registration (`.zcode/config.json`):

```json
{
  "mcp": {
    "servers": {
      "datamgr": {
        "command": "python",
        "args": ["-m", "datamgr.mcp.server"],
        "env": { "DATAMGR_ROOT": "D:/datasets" }
      }
    }
  }
}
```

The server is deliberately read-only: importing data and recomputing stats are
done from the `dm` CLI by a human, never by the AI.

## Deployment & daily use

See **[DEPLOYMENT.md](DEPLOYMENT.md)** for:
- deploying to another machine or MCP client (zcode / Claude / Cursor)
- daily workflow when new data arrives (double-click `.bat` scripts, no CLI needed)
- troubleshooting and a full migration checklist