Skip to main content
Glama
Sign In
datamgr

datamgr

by philogeneperessini-ui
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

datamgr — dataset registry for empirical research

A lightweight registry that catalogs your research datasets so an AI (or you) can discover them without reading every file. Each dataset is one folder with a manifest.yaml; a flat index (_registry.json) makes search cheap no matter how many datasets you have.

Architecture: one copy of logic in datamgr.core; the dm CLI and the MCP server are both thin front-ends over it.

Install (editable)

cd data-manage
pip install -e .

Related MCP server: local-tools-mcp-server

Quick start

# 1. point datamgr at a central warehouse (created if missing)
dm init-root D:/datasets

# 2a. import a raw data folder -- harvests variable names + labels from a .dta,
#     copies the file into a new clean-id folder under the warehouse
dm import "原始数据包" --id csmar_annual --name "CSMAR 上市公司年度财务"

# 2b. (or) scaffold an empty dataset and fill the manifest by hand
dm init csmar_annual --name "CSMAR 上市公司年度财务"

# 3. rebuild the index
dm refresh

# 4. discover
dm list                          # all datasets, one line each
dm show csmar_annual             # full manifest, rendered
dm search 资产负债率               # fuzzy-search across all variables/labels
dm variables --role control      # list all control variables available
dm variables --label 占比          # filter variables by label
dm stats csmar_annual --refresh  # compute rows/cols/n_firms, cache to manifest
dm verify                        # validate manifests + check file paths exist

# 5. pull data into an analysis working directory (backup + provenance)
dm export csmar_annual control_vars --to D:/my_paper
#   -> D:/my_paper/data/*.dta        (copied files, ready for `use "data/xxx.dta"`)
#   -> D:/my_paper/data_sources.txt  (where each file came from, for traceability)

Commands

command

purpose

dm init-root [PATH]

set / show the central warehouse location

dm init <id>

scaffold an empty dataset folder + manifest template

dm import [<folder>] [--id X]

build a manifest by harvesting variables from a .dta/.csv; with --id it copies data into a clean-id folder

dm refresh

rebuild the _registry.json index (reads manifests only, never data files)

dm list [--tag T]

list all datasets, one line each

dm show <id>

print a dataset's full manifest

dm search <query> [--tag T]

fuzzy-search datasets by variable name/label, name, description, tag

dm variables [--role R] [--label L]

list all variables across the index (for picking controls/instruments)

dm stats <id> [--refresh]

show cached stats, or recompute from the data file

dm export <id...> --to <dir>

copy dataset files into a working dir + record provenance

dm verify

validate manifests + check referenced files exist

The manifest (manifest.yaml)

This is the heart of the system. Fields:

field

purpose

id

dataset id, must equal the folder name

name

human-readable name

description

free text — searchable

version, source

provenance

unit_of_observation

granularity, e.g. 公司-年

time_span

[start, end] years

identifiers

{id, time} — fed straight to Stata xtset

variables

list; each has name/label/type/role

tags, dependencies

search facets & lineage

files

relpaths for raw/ pipeline/ processed

stats

cached summary (rows, cols, ...), computed at refresh

notes

any gotchas /口径变更

variables[].role ∈ {id, time, x, y, control, weight, instrument, other} is the key signal that lets an AI judge "can this dataset run the regression I want", which plain variable names can't convey.

Why it stays fast as datasets grow

  • Search reads the in-memory index, never the folder tree at query time.

  • The index stores only search-relevant fields (names, roles, tags, spans) — KB per dataset. A full manifest is loaded only by dm show / get_dataset.

  • Dataset files (.dta, etc.) are read only when you explicitly refresh stats. list/search/show never touch them.

Layout

src/datamgr/
  config.py        warehouse root resolution
  core/
    manifest.py    schema + load/validate/save
    registry.py    scan -> _registry.json index
    search.py      fuzzy search over the index (rapidfuzz)
    stats.py       compute rows/cols/n_firms from a .dta/.csv (explicit --refresh only)
    importing.py   harvest variables from a .dta/.csv -> manifest scaffold
    exporting.py   copy data files to a working dir + provenance txt
  cli/main.py      `dm` command
  mcp/server.py    read-only MCP server for AI discovery (stdio)

MCP server (AI discovery)

A read-only MCP server lets an AI client (e.g. zcode) discover and inspect datasets without touching data files. Exposes 5 tools over stdio:

tool

purpose

list_datasets(tag)

all datasets (id/name/vars/rows/span/tags)

search_datasets(query)

fuzzy search across variables/labels/names

list_variables(role, label)

flat variable list for picking controls

get_dataset(id)

full manifest incl. absolute file paths (locate the .dta on disk)

get_dataset_stats(id)

cached stats only (never reads data files)

Run it directly, or register with an MCP-aware client:

# direct
python -m datamgr.mcp.server
# or via entry point
datamgr-mcp

ZCode workspace registration (.zcode/config.json):

{
  "mcp": {
    "servers": {
      "datamgr": {
        "command": "python",
        "args": ["-m", "datamgr.mcp.server"],
        "env": { "DATAMGR_ROOT": "D:/datasets" }
      }
    }
  }
}

The server is deliberately read-only: importing data and recomputing stats are done from the dm CLI by a human, never by the AI.

Deployment & daily use

See DEPLOYMENT.md for:

  • deploying to another machine or MCP client (zcode / Claude / Cursor)

  • daily workflow when new data arrives (double-click .bat scripts, no CLI needed)

  • troubleshooting and a full migration checklist

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    This server cannot be deployed

    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Resources

    Related MCP Connectors

    Related MCP Servers

    • Secure Schema MCP

      Siddharth-coder13
      A
      license
      A
      quality
      D
      maintenance
      A read-only MCP server that provides AI coding tools with database schema structure (tables, columns, keys, relationships) without exposing row data.
      3
      Apache 2.0

    • local-tools-mcp-server

      fenhuayue
      F
      license
      A
      quality
      D
      maintenance
      A read-only MCP server that enables AI assistants to search files, list directories, retrieve system info, and get file metadata on the local file system.
      4
      -

    • FileManager MCP Server

      Surf-Team
      F
      license
      Not graded
      quality
      D
      maintenance
      Read-only MCP server for AI clients to browse and search project files securely, with configurable permissions, virtual paths, and key-based access.
      -

    • osint-workbench

      rmednitzer
      A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI assistants to search and retrieve stored open-source information through read-only MCP tools, including evidence retrieval by document or snapshot ID with byte-integrity verification.
      1
      Apache 2.0