Skip to main content
Glama

datamgr — dataset registry for empirical research

A lightweight registry that catalogs your research datasets so an AI (or you) can discover them without reading every file. Each dataset is one folder with a manifest.yaml; a flat index (_registry.json) makes search cheap no matter how many datasets you have.

Architecture: one copy of logic in datamgr.core; the dm CLI and the MCP server are both thin front-ends over it.

Install (editable)

cd data-manage
pip install -e .

Related MCP server: Secure Schema MCP

Quick start

# 1. point datamgr at a central warehouse (created if missing)
dm init-root D:/datasets

# 2a. import a raw data folder -- harvests variable names + labels from a .dta,
#     copies the file into a new clean-id folder under the warehouse
dm import "原始数据包" --id csmar_annual --name "CSMAR 上市公司年度财务"

# 2b. (or) scaffold an empty dataset and fill the manifest by hand
dm init csmar_annual --name "CSMAR 上市公司年度财务"

# 3. rebuild the index
dm refresh

# 4. discover
dm list                          # all datasets, one line each
dm show csmar_annual             # full manifest, rendered
dm search 资产负债率               # fuzzy-search across all variables/labels
dm variables --role control      # list all control variables available
dm variables --label 占比          # filter variables by label
dm stats csmar_annual --refresh  # compute rows/cols/n_firms, cache to manifest
dm verify                        # validate manifests + check file paths exist

# 5. pull data into an analysis working directory (backup + provenance)
dm export csmar_annual control_vars --to D:/my_paper
#   -> D:/my_paper/data/*.dta        (copied files, ready for `use "data/xxx.dta"`)
#   -> D:/my_paper/data_sources.txt  (where each file came from, for traceability)

Commands

command

purpose

dm init-root [PATH]

set / show the central warehouse location

dm init <id>

scaffold an empty dataset folder + manifest template

dm import [<folder>] [--id X]

build a manifest by harvesting variables from a .dta/.csv; with --id it copies data into a clean-id folder

dm refresh

rebuild the _registry.json index (reads manifests only, never data files)

dm list [--tag T]

list all datasets, one line each

dm show <id>

print a dataset's full manifest

dm search <query> [--tag T]

fuzzy-search datasets by variable name/label, name, description, tag

dm variables [--role R] [--label L]

list all variables across the index (for picking controls/instruments)

dm stats <id> [--refresh]

show cached stats, or recompute from the data file

dm export <id...> --to <dir>

copy dataset files into a working dir + record provenance

dm verify

validate manifests + check referenced files exist

The manifest (manifest.yaml)

This is the heart of the system. Fields:

field

purpose

id

dataset id, must equal the folder name

name

human-readable name

description

free text — searchable

version, source

provenance

unit_of_observation

granularity, e.g. 公司-年

time_span

[start, end] years

identifiers

{id, time} — fed straight to Stata xtset

variables

list; each has name/label/type/role

tags, dependencies

search facets & lineage

files

relpaths for raw/ pipeline/ processed

stats

cached summary (rows, cols, ...), computed at refresh

notes

any gotchas /口径变更

variables[].role{id, time, x, y, control, weight, instrument, other} is the key signal that lets an AI judge "can this dataset run the regression I want", which plain variable names can't convey.

Why it stays fast as datasets grow

  • Search reads the in-memory index, never the folder tree at query time.

  • The index stores only search-relevant fields (names, roles, tags, spans) — KB per dataset. A full manifest is loaded only by dm show / get_dataset.

  • Dataset files (.dta, etc.) are read only when you explicitly refresh stats. list/search/show never touch them.

Layout

src/datamgr/
  config.py        warehouse root resolution
  core/
    manifest.py    schema + load/validate/save
    registry.py    scan -> _registry.json index
    search.py      fuzzy search over the index (rapidfuzz)
    stats.py       compute rows/cols/n_firms from a .dta/.csv (explicit --refresh only)
    importing.py   harvest variables from a .dta/.csv -> manifest scaffold
    exporting.py   copy data files to a working dir + provenance txt
  cli/main.py      `dm` command
  mcp/server.py    read-only MCP server for AI discovery (stdio)

MCP server (AI discovery)

A read-only MCP server lets an AI client (e.g. zcode) discover and inspect datasets without touching data files. Exposes 5 tools over stdio:

tool

purpose

list_datasets(tag)

all datasets (id/name/vars/rows/span/tags)

search_datasets(query)

fuzzy search across variables/labels/names

list_variables(role, label)

flat variable list for picking controls

get_dataset(id)

full manifest incl. absolute file paths (locate the .dta on disk)

get_dataset_stats(id)

cached stats only (never reads data files)

Run it directly, or register with an MCP-aware client:

# direct
python -m datamgr.mcp.server
# or via entry point
datamgr-mcp

ZCode workspace registration (.zcode/config.json):

{
  "mcp": {
    "servers": {
      "datamgr": {
        "command": "python",
        "args": ["-m", "datamgr.mcp.server"],
        "env": { "DATAMGR_ROOT": "D:/datasets" }
      }
    }
  }
}

The server is deliberately read-only: importing data and recomputing stats are done from the dm CLI by a human, never by the AI.

Deployment & daily use

See DEPLOYMENT.md for:

  • deploying to another machine or MCP client (zcode / Claude / Cursor)

  • daily workflow when new data arrives (double-click .bat scripts, no CLI needed)

  • troubleshooting and a full migration checklist

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    A read-only MCP server that provides AI coding tools with database schema structure (tables, columns, keys, relationships) without exposing row data.
    3
    Apache 2.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    A read-only MCP server for code reading with intelligent caching, line-range selection, and language detection, enabling AI assistants to efficiently and safely explore file systems.
    MIT
  • F
    license
    A
    quality
    C
    maintenance
    A read-only MCP server that enables AI assistants to search files, list directories, retrieve system info, and get file metadata on the local file system.
    4
    -