Skip to main content
Glama
calvinchengx

data-agent-service

by calvinchengx

Data Agent Service

CI Docs release License: Apache 2.0

python coverage go coverage witnesses

Ask your data warehouse a question in English. Get an answer you can defend.

Natural-language questions over the warehouses, databases and APIs you already have — Fabric and Azure SQL, PostgreSQL, Databricks, and any REST service including a retrieval one — grounded in the glossary, metrics and schema held in OpenMetadata. Each query is authorized twice: role rules in the service, then the source itself under the caller's own identity. Any MCP client, unchanged against real Azure.

📖 Documentation site — the full reference, also browsable as Markdown in docs/.

Why this exists

"Which support team resolves tickets fastest?" has a right answer only if everyone agrees what resolves means. Point a general-purpose SQL agent at the warehouse and it will infer that from column names, fluently and with no warning when it is wrong.

On this repository's own seeded data, that inference names the wrong team. Wall-clock elapsed time says Frontline is fastest and Billing is worst. The business's actual definition — Resolution Time, which excludes hours spent waiting on the customer — reverses it: Billing is fastest. A wrong winner is not a rounding error, and nothing in the answer would tell you it happened.

This service is built so that class of error is structurally hard rather than merely unlikely.

What you get

How

Proof

Meaning comes from your catalog, not the model

Glossary terms, metric formulas and column descriptions are read from OpenMetadata at query time. Business semantics are never baked into a prompt

make eval — an ablation scores the same questions with the catalog withheld

Every answer runs as the person asking

The user's token is exchanged on-behalf-of all the way to the engine, so row and column permissions are the engine's decision, not the agent's

make test — two personas, same question, different rows

It cannot write, wander, or work around a refusal

One read-only SELECT, parsed rather than pattern-matched; schema allow-list; row ceiling applied for you; a refusal is reported, not routed around

make conformance — a 39-assertion contract the executor must satisfy

It answers with its reasoning attached

The figure, the definition applied, the tables it came from, and any caveat the catalog raised

make ask Q="..."

A question that keeps recurring becomes a governed dashboard

The promoter counts templates, not questions — no natural language is stored. A released candidate is published to Power BI, Superset and Tableau from one Plan, each verified against the SQL it came from before it reaches the catalog

docs/14-publishing.md

Bring your own model, and your own LLM gateway

Two wire protocols rather than one integration per vendor: Anthropic Messages and OpenAI chat completions reach LiteLLM, TrueFoundry, Azure OpenAI, OpenRouter, vLLM and the rest. A backend declares what it cannot do, and a deployment has to accept that in writing

docs/21-llm-backends.md — one contract, both protocols, in CI

Any MCP client, no custom code

Claude, Cursor, VS Code and the SDKs connect over standard MCP with OAuth discovery

docs/09-mcp-clients.md

Runs on your laptop, deploys to real Azure unchanged

The whole stack runs on the emulator family; switching to Fabric, APIM and Entra is configuration, not a code path

docs/10-production.md

Nothing here is claimed without something that checks it

Every capability carries a command that proves it; where something is designed but not built, the docs say so

the witnesses badge above, live from docs/witnesses.json, in CI on every push

Status: Phases 0-16 landed — see docs/00-plan.md.

Related MCP server: Interactive Database Analyst via MCP

Quick start

make doctor   # toolchain, docker, ~14 GB memory
make up       # entra, keyvault, arm, fabric (+ SQL Server), OpenMetadata 1.13.2, apim
make status   # "stack OK" is the verdict

Then make seed, make test, make eval, make load, make ask Q="…" — or make stack to do the whole bring-up from nothing, which is what CI runs.

What is here

Path

Purpose

docker-compose.yml

Emulators and OpenMetadata are pinned, published images used as-is. Only this repo's own services build — plus superset, which adds the database drivers Apache Superset's image deliberately ships without

.env.example

Every DAS_* setting; copy to .env (local) or .env.prod (real Azure)

docs/00-plan.md

Architecture, decisions, phases, evaluation, load, authz, extension

docs/

Quickstart, architecture, authorization, classification, evaluation, load, MCP clients, adding a source, production, CI

services/

The warehouse-query executor (Python and Go), and the contract both answer to

promoter/, publisher/

What keeps being asked, and the Plan each dashboard target spells in its own language

publisher-go/

A second generator held to the same recorded bytes — deterministic output, checked rather than described

agent/, evals/, e2e/

The agent, the accuracy suite, and the witnesses

seed/

Datasets, warehouse provisioning, OpenMetadata semantics, identity setup

infra/terraform/

Terraform for real Azure; docs/10-production.md is the runbook

.github/workflows/ci.yml

Six jobs; docs/11-ci.md says what each proves

website/

The docs site — Astro + Starlight, generated from docs/, which stays the source of truth

scripts/

doctor.sh, status.sh, check-discipline.sh, preflight.py, and the gates make lint runs

Discipline

  1. Emulators and OpenMetadata are never modified; suspected bugs go to docs/upstream-issues.md.

  2. No emulator-only code paths. Standard protocols only (OIDC/OAuth2 incl. OBO, managed-identity App Service protocol, TDS FedAuth, ARM, Graph, OM REST/MCP). ENV=prod swaps .env and nothing else.

Emulator family

Built on entra-emulator, azure-keyvault-emulator, arm-emulator, fabric-emulator, azure-apim-emulator; composed per azure-emulators.

data-agent-voice is the voice front end over this service: it consumes this MCP surface, and every question still runs as the asking user. Its ledger and this one answer adjacent questions, so a claim about the whole path is only as strong as the weaker of the two.

Tier: intermediate — emulates nothing, and is consumed by data-agent-voice.

License

Apache-2.0.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables natural language querying of SQL databases with robust safety guarantees including read-only enforcement, AST validation, and row caps.
    -
  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables natural-language querying of PostgreSQL databases with schema grounding and self-correcting error recovery. Provides a live audit trace and verifies results through exploratory decomposition and empty-result sanity checks.
    -
  • F
    license
    Not graded
    quality
    C
    maintenance
    Provides a guarded semantic layer for text-to-SQL, offering tools for schema discovery, semantic profiling, and query verification to enable safe and reliable SQL generation on enterprise databases.
    -
  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables natural-language querying of SQLite databases through a governed semantic layer, with citations and typed abstention for PII or uncertified data.
    -