DAAS - Data As a Service

Layered data platform for financial, economic, and statistical data — a single SQLite file (daas.db) behind a consolidated MCP server, with data fetch delegated down to the fd-open-data-mcp upstream.

What is this? A local data platform that turns Python data libraries (akshare, yfinance, edgar, edinet-tools, dartlab, world_bank_data, ckanapi) into a queryable, indicator-computing, dashboard-ready store backed by one SQLite file. You drive it through Claude Code skills (thin shells that call workflow manifests) or through the consolidated fd-daas-mcp MCP server — both paths read/write the same database.

Upstream: The fd-open-data-mcp data-fetcher is a sibling repo at ~/finddata/fd-open-data-mcp (cloned automatically by install.sh).

Docs site: The full, role-based documentation lives at docs-site/ (MkDocs Material, EN+ZH bilingual). Read it locally with uv run mkdocs serve (browses at /DAAS/), or build strictly with uv run mkdocs build --strict. See docs-site/README.md for build/serve/deploy.

Architecture

Strict downward dependency — a layer never reaches up.

L3  user MCP compositions  (composite manifests, served in-proc on fd-daas-mcp)
L2  workflow manifests      (daas.db `workflows` table + engine, run via workflow_run)
L1  fd-daas-mcp            (consolidated infra: daas/cron/alerts/dashboard/composite/research/pdf/gateway/workflow)
L0  fd-open-data-mcp       (sole data-fetch upstream; concept-based semantic fetcher + entity master)

• L0 — fd-open-data-mcp (sibling repo): the sole data-fetch surface. A concept-based semantic fetcher with ranking/failover/caching; holds the entity master (entities, entity_datasource_links). Served HTTP at :8300 (stdio fallback). Replaces the 11 former per-source data-fetch MCPs.

• L1 — fd-daas-mcp (this repo): the consolidated stdio server, sole entry in repo-root .mcp.json. Exposes 161 tools across 9 groups (daas · cron · alerts · dashboard · composite · research · pdf · gateway · workflow) behind one server and one fd-daas-mcp Click CLI. The thin consolidation layer is fd-daas-mcp/daas/fd_daas_mcp/ (server.py/registry.py/cli.py/selfcheck.py); each group's tool code lives in-package at fd-daas-mcp/<group>-mcp/.

• L2 — workflow manifests: manifests live in the workflows table in daas.db (registered via workflow_register, run via workflow_run). build_workflow_from_goal decomposes a natural-language goal into a manifest via an LLM.

• L3 — user MCP composition: a composite manifest ({name, upstreams, tools, workflows, prompt}) curates a named MCP surface served in-proc on the consolidated server. CRUD via composite_*_manifest.

The fetch skills (fd-daas-based-data-fetch, fd-daas-fetch-data, fd-daas-research) are thin shells: parameter-gathering → workflow_run(name, params) → checkpoint handling. They no longer call Python data libraries directly — fetch goes down through L1→L0.

For the full architecture, conventions, and the daas.db schema reference, see CLAUDE.md and construction/mcp.md.

Install & Quick Start

Requirements: Python 3.10+ and uv. dartlab fetches need 3.12 — run them with uv run --python 3.12 --with dartlab ....

One-click install (clones DAAS + upstreams, provisions venvs, inits daas.db, localizes .mcp.json):

curl -fsSL https://raw.githubusercontent.com/FindDataTechnology/fd-daas-mcp/master/install.sh | sh

Env overrides: DAAS_DEST (default ~/code/DAAS), DAAS_BRANCH, FINDDATA_HOME (default ~/finddata). Manual steps below.

# 1. Provision the root venv (data libs are declared deps)
uv sync

# 2. Provision the database - creates daas.db (full schema + dep-free starter
#    catalog of sources). DAAS_DATABASE_URL is OPTIONAL: unset, it defaults to
#    ./daas.db (writable cwd) or ~/.fd-daas-mcp/daas.db. Set it only to relocate.
fd-daas-mcp/.venv/bin/fd-daas-mcp init       # one-shot provision + seed
fd-daas-mcp/.venv/bin/fd-daas-mcp doctor      # read-only health check (path, schema, row counts)

# 3. Configure credentials - create a repo-root .env for the source keys you need
#    (listed in Environment Variables below). Scripts auto-load .env; no manual export.

# 4. Compute an existing indicator (upserts into observations)
uv run python .claude/skills/fd-daas-based-data-fetch/scripts/run_indicator.py SPY_ma5

# 5. Query daas.db directly (db lives at the resolved DAAS_DATABASE_URL, default ./daas.db)
sqlite3 daas.db "SELECT name, datasource, op FROM indicator_rules LIMIT 10"
sqlite3 daas.db "SELECT source, COUNT(*) FROM observations GROUP BY source"

# 6. Launch / health-check the consolidated server
fd-daas-mcp/bin/fd-daas-mcp-server                       # stdio server (what .mcp.json launches)
fd-daas-mcp/.venv/bin/python -m daas.fd_daas_mcp.selfcheck   # registry + tool health check (target: failed=0)

# 7. Run a workflow manifest (L2 — the fetch path goes L1→L0)
fd-daas-mcp/.venv/bin/python -c "
from daas_mcp_workflow_tools import workflow_run   # or via the MCP tool
print(workflow_run('fetch-and-persist', params_json='{\"entity\":\"SPY\",\"indicator\":\"ma5\"}'))
"

The Quick Start commands above have been verified against this repo: SPY_ma5 is a real indicator_rules row, and the fd-daas-mcp registry reports 161 tools across 9 sources (failed=0, skipped_optional=1 for the optional pdf group).

Project Structure

daas/
├── .claude/skills/          # Claude Code skills (fd-daas-based-data-fetch is the core fetch shell)
├── fd-daas-mcp/             # Consolidated MCP server — sole .mcp.json entry (161 tools, 9 groups)
│   ├── alerts-mcp/          #   alert rule engine + 7 notification channels
│   ├── composite-mcp/       #   user MCP composition (curate tools + embed workflows + prompt)
│   ├── cron-mcp/            #   task + schedule registry (DB-backed)
│   ├── daas-mcp/            #   datasource/function/indicator/entity catalog + compute + rules
│   ├── dashboard-mcp/       #   standalone-HTML dashboard registry + query
│   ├── gateway-mcp/         #   L0 upstream registry + call routing (former leader gateway half)
│   ├── workflow-mcp/        #   manifest-based multi-step data workflows (former leader workflow half)
│   ├── pdf-mcp/             #   local PDF/text semantic search (sqlite-vec) [optional]
│   ├── research-mcp/        #   persisted research bundle (collections + indicators + dashboard + report)
│   ├── bin/fd-daas-mcp-server      # launcher
│   └── daas/fd_daas_mcp/   # server.py / registry.py / cli.py / selfcheck.py
├── daas.db                  # Shared SQLite database (ships as a demo dataset: registry + observations + scraw_*)
├── dashboards/              # Standalone HTML dashboards (+ index.html, daas.md)
├── construction/            # Architecture docs (mcp.md — layered L0/L1/L2/L3)
└── .env                     # DAAS_DATABASE_URL, proxy, source auth keys, LLM config, ...

daas.db Data Model

One SQLite file at the path in DAAS_DATABASE_URL (relative sqlite:/// paths resolve against repo root; PRAGMA foreign_keys=ON for FK cascade, PRAGMA journal_mode=WAL + busy_timeout=10000 to dodge "database is locked"). Tables group by role:

Query it directly from the repo root: sqlite3 daas.db "SELECT …".

Skills (.claude/skills/)

Skills are plain Markdown (SKILL.md) + Python scripts. The fetch skills are thin shells that gather parameters and call workflow_run — they no longer call Python data libraries directly (fetch goes L1→L0).

MCP Tool Groups (fd-daas-mcp)

The consolidated server exposes 161 tools across 9 groups (failed=0, skipped_optional=1 for the optional pdf group). Catalog is group-level (per-tool detail via the server's own introspection / selfcheck).

The legacy leader group is dissolved: its gateway-routing half became gateway_*, its workflow-manifest half became workflow_*. Harness-registry / snapshot / provenance capabilities are deleted.

Launch: fd-daas-mcp/bin/fd-daas-mcp-server (stdio). Both the server and the fd-daas-mcp CLI consume registry.build(), so the two surfaces cannot drift.

Environment Variables

A single repo-root .env holds all config; scripts and the MCP server auto-load it. (Keys marked optional are only needed for the features they enable.)

For AI Agents

If you are an AI agent (e.g. Claude Code) operating in this repo:

• Fetch data through the workflow path. Use fd-daas-based-data-fetch: resolve the entity + indicator against daas.db via sqlite3, then workflow_run(name, params) — the manifest routes the fetch down through gateway_call → fd-open-data-mcp (L0) and persists into scraw_<slug> / observations. For multi-step fetches, build_workflow_from_goal emits a manifest.

• Workflow: resolve → fetch (via L0) → persist. Resolve entity+indicator in daas.db; fetch via the gateway; persist into scraw_<slug> (raw) or observations (computed indicator).

• Use the MCP server for everything else — catalog browsing, creating indicators/collections/rules, cron scheduling, alerts, building/finding dashboards, PDF semantic search, composite authoring, research bundles. These are the fd-daas-mcp tools (161 across 9 groups).

• Query daas.db with sqlite3 from the repo root (sqlite3 daas.db "…"). Use PRAGMA foreign_keys=ON for FK cascade.

• Authoritative architecture + schema reference: CLAUDE.md (it has a ## daas.db section listing every table) and construction/mcp.md (the layered L0/L1/L2/L3 reference).

License

Apache 2.0.