data-quality-cde-mcp-server
This MCP server automates data quality checks on live Cloudera Data Warehouse (CDW) tables using CDE Airflow jobs generated from ODCS data contracts, without requiring WAP staging. You can:
Generate DAGs: Parse an ODCS contract (schema + quality rules like completeness, uniqueness, range, freshness, custom SQL, etc.) to generate an Airflow DAG, and preview the SQL checks before deployment.
Deploy: Deploy the DAG and required resources to CDE as an Airflow job, with options to force recreate and auto-trigger after deployment.
Manage Jobs: List, describe, and delete CDE jobs, and list file resources.
Run & Monitor: Trigger manual runs, list job runs, get run status, and check the latest Airflow DAG run status.
Diagnose: View redacted CDE/CDW configuration and test CDE endpoint connectivity.
The generated DAGs include audit logging and support flexible cron scheduling (defaulting to daily at 03:00 UTC).
Provides tools for managing Cloudera Data Engineering (CDE) jobs, including generating DAGs from data contracts, deploying Airflow jobs, listing jobs and resources, triggering runs, and monitoring job status.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@data-quality-cde-mcp-server Deploy the flights contract to CDE"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Data Quality CDE MCP Server
MCP server that generates and deploys Cloudera Data Engineering (CDE) Airflow jobs from ODCS data contracts. It validates live table data in CDW against schema and data-quality rules — without Write-Audit-Publish (WAP) staging branches.
Built on patterns from data-quality (CDE deployment, CDW connection, SQL operators) and data-contract-mcp-server (ODCS contract format, MCP tool design).
What it does
Parse an ODCS-compatible data contract (
schema_objects+qualityrules)Generate an Airflow DAG that runs daily (default
0 3 * * *) using:SQLExecuteQueryOperator— audit table setupSQLCheckOperator— schema + quality checks against the live tableBranchPythonOperator— pass/fail gate with audit logging
Deploy to CDE — upload DAG +
lib/+sql/resource, create Airflow jobMonitor CDE job runs and Airflow execution status
Related MCP server: oracle-eim-dq
MCP tools (14)
Tool | Description |
| Show current CDE/CDW settings |
| Probe CDE CLI endpoint + list jobs |
| Generate DAG Python from contract JSON |
| Generate Spark orchestrator DAG + bundle preview |
| Preview SQL checks before deploy |
| Generate + deploy CDE job ( |
| List CDE jobs |
| Job details |
| List file resources |
| Manual job run |
| List runs for a job |
| Run status by id |
| Latest Airflow run for a CDE job |
| Delete CDE job |
Configuration
Copy .env.example to .env:
CDE_USER=your_cde_workload_user
CDE_PASSWORD=your_cde_workload_password
CDE_ENDPOINT_URL=https://your-cde-service.cloudera.site
CDW_CONNECTION_ID=CDW
DEFAULT_DAG_SCHEDULE=0 3 * * *Parameter | Description |
| CDE workload username |
| CDE workload password |
| Jobs API URL from CDE cluster details ( |
| Optional Knox token URL (auto-derived if unset) |
|
|
| Path to |
| Airflow connection id for Hive/Impala in CDW (default: |
| Cron schedule (default: daily at 03:00 UTC) |
Agent Studio: The CDE CLI is not installed in CrewAI/Agent Studio. Use CDE_TRANSPORT=api (default) with the Jobs API URL — not the Grafana or service-only URL.
Local laptop: Use CDE_TRANSPORT=cli if you have the cde binary configured, or keep api everywhere for consistency.
Prerequisite: For deploy tools, generated DAG bundles are uploaded via REST. The Airflow CDW connection must exist in your CDE environment.
Cloudera Agent Studio (recommended)
Agent Studio only supports stdio MCP servers launched with uvx (Python) or npx (Node.js). Use a git URL so the runtime can install the package; do not use uv run unless the repo is checked out on the same machine.
{
"mcpServers": {
"data-quality-cde-mcp-server": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/frothkoetter/data-quality-cde-mcp-server.git@master",
"run-server"
],
"env": {
"CDE_USER": "<your_cde_workload_user>",
"CDE_PASSWORD": "<your_cde_workload_password>",
"CDE_ENDPOINT_URL": "https://<your-cde-service>.cloudera.site",
"CDW_CONNECTION_ID": "CDW"
}
}
}
}See also conf.yaml.example for YAML-style Agent Studio registration.
Registration tips
Push this repo to GitHub first —
uvx --from git+https://...cannot resolve a local-only repo.Use placeholder CDE credentials during catalog registration; provide real values when attaching the MCP server to a workflow agent.
If you see "We could not figure out the tools offered by the MCP server", the server may still work in workflows — Agent Studio documents occasional tool-discovery failures. Add the MCP server to your agent manually and select tools there.
Avoid writing to stdout from wrapper scripts — stdio transport uses stdout for JSON-RPC.
Ensure the Agent Studio environment has
uvxon PATH (which uvx) and can reachCDE_ENDPOINT_URL.
See Cloudera MCP registration docs.
Troubleshooting MCP server load failures
If CrewAI Agent Studio, Cursor, or another MCP host fails to connect to this server via uvx, check the following.
Wrong git branch in uvx --from
This repo's default branch is master (not main). Agent Studio registration fails if the git URL points at a non-existent branch:
git+https://github.com/frothkoetter/data-quality-cde-mcp-server.git@masterFastMCP / mcp import error
uvx installs the latest mcp package (currently 2.x). MCP Python SDK 2.0 renamed FastMCP to MCPServer. Older server builds crash on startup with:
ImportError: cannot import name 'FastMCP' from 'mcp.server'This server includes the dual-import fix (MCPServer + FastMCP fallback). Ensure you use a build from main after the MCP SDK 2.x compatibility update.
Verify locally:
CDE_USER="test" CDE_PASSWORD="test" CDE_ENDPOINT_URL="https://example.com" \
uv run python scripts/test_mcp_stdio.py uvx --from . run-serverExpected output: OK initialize and OK tools/list: 13 tools.
Missing CDE credentials
CDE credentials are not required at startup (tool discovery works without them). They are required when calling deploy/monitor tools such as deploy_contract_dq_job or list_cde_jobs. Set CDE_USER, CDE_PASSWORD, and CDE_ENDPOINT_URL in the MCP host env block before using those tools.
FileNotFoundError: cde in Agent Studio
CrewAI Agent Studio does not install the CDE CLI. The server now uses the CDE REST API by default (CDE_TRANSPORT=api). Ensure:
CDE_ENDPOINT_URLis the Jobs API URL from your virtual cluster details (https://<vc>.<host>/dex/api/v1) — not a generic service URL.CDE_TRANSPORT=api(default) — do not setcliin Agent Studio.CDE_USER/CDE_PASSWORDare valid CDP workload credentials.
Test connectivity with the diagnose_cde_connectivity MCP tool after re-registering.
Agent Studio / CrewAI config
Ensure uvx is on the Agent Studio host PATH (which uvx).
Cursor / MCP setup
Add to your MCP config (see mcp.json.example):
{
"mcpServers": {
"data-quality-cde-mcp-server": {
"command": "uvx",
"args": ["--from", "git+https://github.com/frothkoetter/data-quality-cde-mcp-server.git@master", "run-server"],
"env": {
"CDE_USER": "<user>",
"CDE_PASSWORD": "<password>",
"CDE_ENDPOINT_URL": "https://<cde-host>",
"CDW_CONNECTION_ID": "CDW"
}
}
}
}Local development:
cd data-quality-cde-mcp-server
uv sync
uv run run-serverExample contract
See examples/flights_contract.json — based on the airlinedata.flights table from the data-quality project:
{
"contract_id": "flights-contract",
"version": "1.0",
"schema_objects": [{
"name": "flights",
"physical_name": "airlinedata.flights",
"properties": [
{"name": "uniquecarrier", "isRequired": true},
{"name": "origin", "isRequired": true}
]
}],
"quality": [
{"name": "carrier_completeness", "rule_type": "completeness", "element": "uniquecarrier"},
{"name": "delay_range", "rule_type": "range", "element": "arrdelay", "threshold": "600"}
]
}Generated DAG flow
ensure_audit_table → column/table/custom checks → evaluate_contract_gate
├─ dq_success → persist_audit
└─ contract_violation_alert (AirflowException + audit log)Uses Airflow Common SQL provider operators per Astronomer's SQL data quality guide:
Operator | Used for | Contract mapping |
| Null, unique, min, max, distinct per column |
|
| Row count, freshness, composite PK | Default |
| Custom SQL | Rules with a |
Column check types (SQLColumnCheckOperator)
Check type | SQL abstraction | Example contract rule |
|
|
|
|
|
|
|
|
|
|
|
|
Qualifiers: equal_to, greater_than, geq_to, less_than, leq_to (with optional tolerance).
Table checks (SQLTableCheckOperator)
Check | Example |
Row count |
|
Freshness |
|
Composite PK | Custom |
Custom checks (SQLCheckOperator)
Any rule with a query field, or checks spanning multiple tables. Returns a single row; task fails if any value is Python-falsy (e.g. 0).
Supported quality rule types
Rule type | Generated SQL |
| Null count = 0 |
| No duplicate values |
| Column <= threshold |
| Column >= threshold |
| Max timestamp within N hours |
| COUNT(DISTINCT col) >= threshold |
Custom | Use |
Schema checks always include: min row count, required-column null checks, primary-key uniqueness.
Deploy from MCP
deploy_contract_dq_job(
contract_json=<ODCS contract>,
cde_user=...,
cde_password=...,
cde_endpoint_url=...,
cdw_connection_id="CDW",
trigger_after_deploy=true
)This creates:
CDE resource:
{contract-id}-dq-resource(DAG + lib/ + sql/)CDE Airflow job:
{contract-id}-dq-jobDAG id:
dag_contract_dq_{contract_id}
Spark 3.4 orchestration (execution_mode="spark")
Use when checks should run in Spark instead of Airflow SQL operators:
deploy_contract_dq_job(
contract_json=<ODCS contract>,
execution_mode="spark",
spark_runtime_image="dex-spark-runtime", # optional Spark 3.4 runtime resource
schedule="0 * * * *",
trigger_after_deploy=true
)This creates:
Same CDE resource with
contract_dq_spark.py,contract_config.json, and orchestrator DAGSpark child job:
{contract-id}-spark-dq-job(triggered by Airflow, not scheduled)Airflow orchestrator job:
{contract-id}-dq-jobwith schedule; DAG ensures Icebergdq_audit_log/dq_audit_metrictables then callsCDEJobRunOperatorSpark job runs contract checks and writes audit log + Griffin metrics rows
Optional env: CDE_SPARK_RUNTIME_IMAGE, CDE_SPARK_NUM_EXECUTORS, CDE_SPARK_EXECUTOR_MEMORY,
CDE_SPARK_DRIVER_MEMORY, CDE_SPARK_CONF (JSON).
Monitor:
list_cde_job_runs(job_name="flights-contract-dq-job")
get_airflow_dag_run_status(job_name="flights-contract-dq-job")Schedule / duplicate runs: Deploy leaves the CDE schedule paused by default. Unpausing can immediately enqueue a missed-interval (backfill) run. Never combine unpause with a manual trigger in the same step — that starts two runs within seconds.
default deploy → schedule paused; unpause manually or pass
activate_schedule=truetrigger_after_deploy=true→ one isolated manual run; schedule re-paused after triggertrigger_cde_job→ same isolated manual run (schedule stays paused)All generated DAGs set
max_active_runs=1to serialize overlapping DAG runs
Differences from data-quality (WAP project)
Feature | data-quality | This MCP server |
WAP / Iceberg branches | Yes | No |
DataSketches baseline | Yes | No (contract-driven SQL) |
Data ingest staging | Yes | No — checks live table |
Deployment | Shell script | MCP tools + CDE CLI |
Contract source | Airflow Variables | ODCS JSON / Atlas contracts |
Schedule |
| Default |
Development
uv sync
uv run pytest
uv run ruff check src testsLicense
Apache-2.0
Available Tools
13 toolsdelete_cde_jobB
Delete a CDE Airflow job. WRITE OPERATION
Inputs: job_name (required), cde_user, cde_password, cde_endpoint_url. Returns: result
deploy_contract_dq_jobA
Generate DAG from data contract and deploy to Cloudera CDE as an Airflow job.
Creates/updates a CDE files resource (lib/, sql/, DAG) and an Airflow job. Checks live table data against schema and quality rules — no WAP/staging branch.
Args: contract_json: ODCS-compatible contract JSON. cde_user: CDE workload username. cde_password: CDE workload password. cde_endpoint_url: CDE REST/CLI endpoint URL. cdw_connection_id: Airflow CDW connection id (default CDW). schedule: Cron schedule (default daily 0 3 * * *). audit_table: Optional audit table qualified name. force_recreate: Delete and recreate job/resource before deploy. trigger_after_deploy: Trigger a manual CDE job run after deploy.
Inputs: contract_json (required), cde_user, schedule, audit_table, +5 more. Returns: result
describe_cde_jobC
Describe a CDE job by name.
Inputs: job_name (required), cde_user, cde_password, cde_endpoint_url. Returns: result
diagnose_cde_connectivityB
Probe CDE CLI connectivity (configure endpoint + list jobs).
Args: cde_user: CDE workload username (overrides CDE_USER env). cde_password: CDE workload password (overrides CDE_PASSWORD env). cde_endpoint_url: CDE service URL (overrides CDE_ENDPOINT_URL env).
Inputs: cde_user, cde_password, cde_endpoint_url. Returns: result
generate_dag_from_contractA
Generate Airflow DAG Python from an ODCS data contract (no WAP).
Validates live table schema (required columns, PK uniqueness) and data quality rules using Airflow SQLCheckOperator against the CDW connection (default: CDW). Default schedule is once daily (0 3 * * *) unless overridden.
Args: contract_json: ODCS-compatible contract JSON (schema_objects + quality rules). schedule: Cron schedule (default 0 3 * * * — daily at 03:00 UTC). cdw_connection_id: Airflow connection id for Hive/Impala in CDW (default CDW). audit_table: Optional qualified audit table name for check results.
Inputs: contract_json (required), schedule, audit_table, cdw_connection_id. Returns: result
get_airflow_dag_run_statusA
Monitor latest CDE Airflow job run status (via CDE run API).
Returns the most recent run for the given CDE Airflow job, including status, start/end times, and run id for further inspection.
Inputs: job_name (required), cde_user, cde_password, cde_endpoint_url. Returns: result
get_cde_configA
Return current CDE/CDW configuration (passwords redacted).
Reads CDE_USER, CDE_PASSWORD, CDE_ENDPOINT_URL, CDE_BIN, CDW_CONNECTION_ID from environment or .env file.
Inputs: no named parameters. Returns: result
get_cde_job_run_statusB
Get status of a specific CDE job run.
Inputs: run_id (required), cde_user, cde_password, cde_endpoint_url. Returns: result
list_cde_job_runsA
List CDE runs for a job (monitor execution status).
Inputs: job_name (required), cde_user, cde_password, cde_endpoint_url. Returns: result
list_cde_jobsC
List all CDE jobs.
Inputs: cde_user, cde_password, cde_endpoint_url. Returns: result
list_cde_resourcesC
List CDE file resources.
Inputs: cde_user, cde_password, cde_endpoint_url. Returns: result
preview_contract_checksA
Preview schema and quality SQL checks that would be generated from a contract.
Inputs: contract_json (required). Returns: result
trigger_cde_jobC
Trigger a manual CDE job run.
Inputs: job_name (required), cde_user, cde_password, cde_endpoint_url. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 13 tools
Most tools are distinct, but there is notable overlap among run-monitoring tools (get_cde_job_run_status, get_airflow_dag_run_status, list_cde_job_runs) and between generate_dag_from_contract and deploy_contract_dq_job, which both generate DAGs. This creates moderate ambiguity for an agent.
All tools follow a consistent verb_noun pattern in lowercase snake_case (e.g., list_cde_jobs, trigger_cde_job, delete_cde_job). The verbs are distinct and the pattern is highly predictable.
13 tools is well-scoped for a CDE/CDW data-quality server, covering configuration, connectivity checks, DAG generation/deployment, job lifecycle, and monitoring without unnecessary bloat.
The tool surface covers the full workflow: configure and diagnose connectivity, generate/preview/deploy data contract DAGs, manage jobs, trigger runs, monitor status, and delete jobs. No critical operations are missing for the stated domain.
Maintenance
Related MCP Connectors
Auto-discover validation rules from data — scan, profile, health-score. No rules to write.
Deterministic validation for AI-generated artifacts: JSON Schema, OpenAPI response, SQL syntax.
Create and manage scheduled, guarded AI agent jobs with built-in quality control and 900+ connectors
Build and manage data pipelines, add custom connectors, and chat with your data.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceProvides governed MCP tools for validating, explaining, and remediating clinical supply chain data quality across Bronze, Silver, and Gold layers on Azure Databricks, with data contracts and an evaluation harness.-
- FlicenseNot gradedqualityDmaintenanceEnables governed, read-only EIM data-quality checks against Oracle On-Prem and ATP by loading ACTIVE rules and running approved aggregate SQL to produce Markdown reports.-
- AlicenseNot gradedqualityBmaintenanceEnables AI coding assistants to audit datasets against DAMA-DMBOK quality dimensions, sanitize PII, generate differential-privacy synthetic twins, compress prompt context, and export dbt schema tests and interactive HTML reports. It supports development against sensitive data with zero real records exposed to external LLMs.1MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI assistants to audit dbt project quality, profile warehouse data, detect schema drift, and auto-generate or update documentation and tests through natural language.-