dbt-investigator
Allows investigation of dbt test failures by reading the dbt manifest and run_results, enabling root cause analysis through lineage tracing and data profiling.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@dbt-investigatorWhy did not_null_fct_transactions_merchant fail?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Data Quality Agent
An agentic AI system that automatically investigates dbt test failures, traces the root cause through BigQuery lineage, and generates a plain-English incident report ā cutting investigation time from hours to minutes.
Architecture
Related MCP server: aegis-dq
What it does
When a dbt test fails you normally get a cryptic error message. This agent:
Fetches the failing rows from BigQuery ā sees the actual bad data
Reads the dbt manifest ā understands the full lineage graph
Traces upstream ā profiles columns in parent models and source tables
Identifies the root cause ā finds where the bad data entered the pipeline
Writes an incident report ā plain-English root cause, lineage trace, recommended fix, severity
not_null_fct_transactions_merchant failed (23 rows)
ā
ā¼
Agent fetches failing rows ā reads fct lineage ā traces to int_ ā traces to stg_ ā checks raw source
ā
ā¼
Root cause: 23 rows in raw.bank_transactions have NULL narration.
Merchant extraction returns NULL when narration is NULL.
Fix: Add COALESCE(narration, '') in stg_bank__transactions.
Severity: HIGHThree trigger modes
1 ā CLI
python agent.py \
--test not_null_fct_transactions_merchant \
--model fct_transactions \
--column merchant \
--verbose2 ā Webhook (Airflow or any HTTP caller)
python server.py # starts on port 5051
curl -X POST http://localhost:5051/investigate \
-H "Content-Type: application/json" \
-d '{"test_name": "not_null_fct_transactions_merchant", "model": "fct_transactions", "column": "merchant"}'Point your Airflow DAG's on_failure_callback at this endpoint.
3 ā MCP (any AI client)
The MCP server exposes three tools to any MCP-compatible client ā Claude Code, OpenClaw (ChatGPT / Gemini / any client), Cursor, Zed:
Tool | What it does |
| Full agentic investigation ā incident report |
| List failing tests from run_results.json |
| Read a saved incident report |
Claude Code:
claude mcp add -s user \
-e GCP_PROJECT=your-project \
-e BQ_LOCATION=asia-south1 \
-e DBT_MANIFEST_PATH=/path/to/dbt_bank/target/manifest.json \
-e DBT_RUN_RESULTS_PATH=/path/to/dbt_bank/target/run_results.json \
-e GEMINI_API_KEY=your-key \
dbt-investigator \
-- /path/to/venv/bin/python /path/to/mcp_server.pyOpenClaw (ChatGPT, Gemini, or any other client):
openclaw mcp set dbt-investigator '{
"command": "/path/to/venv/bin/python",
"args": ["/path/to/mcp_server.py"],
"cwd": "/path/to/data-quality-agent",
"env": {
"GCP_PROJECT": "your-project",
"GEMINI_API_KEY": "your-key",
"DBT_MANIFEST_PATH": "/path/to/manifest.json",
"DBT_RUN_RESULTS_PATH": "/path/to/run_results.json"
}
}'
openclaw mcp probe # ā dbt-investigator: 3 tools āAgent tools
Tool | What the agent calls |
| Queries BigQuery for actual bad rows |
| Reads manifest.json for upstream/downstream |
| Gets compiled SQL for any model |
| null count, distinct count, min, max |
| Custom read-only BQ investigation |
| Checks staleness of source tables |
| Writes the final incident report |
Safety wall: all BigQuery queries are read-only (SELECT/WITH only). DML/DDL rejected before execution.
Setup
git clone https://github.com/ARAVINDHRAJA123/data-quality-agent.git
cd data-quality-agent
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
# Auth
gcloud auth application-default login
# Set environment
export GCP_PROJECT=your-project
export BQ_LOCATION=asia-south1
export DBT_MANIFEST_PATH=/path/to/dbt_bank/target/manifest.json
export DBT_RUN_RESULTS_PATH=/path/to/dbt_bank/target/run_results.json
# LLM (pick one)
export GEMINI_API_KEY=your-key # free
export ANTHROPIC_API_KEY=your-key # paidGenerate the manifest first (from your dbt project):
cd /path/to/dbt_project && dbt compile
# manifest.json is now at target/manifest.jsonStack
Claude / Gemini ā LLM provider (auto-detected, free Gemini supported)
BigQuery ā data warehouse (GCP)
dbt manifest.json ā lineage graph and compiled SQL
FastMCP ā MCP server (any AI client)
Flask ā webhook server (Airflow integration)
pytest ā test suite
Project structure
data-quality-agent/
āāā agent.py ā agentic investigation loop (Claude + Gemini)
āāā server.py ā Flask webhook server
āāā mcp_server.py ā FastMCP server (any MCP client)
āāā report.py ā incident report formatter
āāā tools/
ā āāā bq_tools.py ā BigQuery: failing rows, queries, freshness
ā āāā dbt_tools.py ā manifest: lineage, SQL, test results
āāā tests/
ā āāā test_tools.py ā 11 unit tests (no BQ/LLM needed)
āāā reports/ ā saved incident reports (markdown)
āāā requirements.txtThis server cannot be deployed
Maintenance
Related MCP Connectors
Triage failing GitHub Actions jobs and see what self-heal repaired, in natural language.
AI agent run monitoring with incident replay and SLA receipts.
Synthetic checks, nightly regression replay and model-drift alerts for AI agents
Diagnose AI workflows for failure, security, and handoff risks ā RED/AMBER/GREEN per node.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceData observability for AI agents. Query alerts, monitor freshness, investigate schema drift, and trace lineage across your data warehouse via 53 MCP tools.1MIT
- AlicenseBqualityDmaintenanceAgentic data quality MCP server ā runs structured validation rules against warehouses (DuckDB, BigQuery, Athena, Databricks, Postgres), diagnoses failures with LLM root cause analysis, and proposes SQL remediations. Full audit trail of every AI decision.64Apache 2.0
- AlicenseNot gradedqualityCmaintenanceAI-driven MCP server that audits, profiles, detects schema drift, and auto-generates documentation for dbt projects, enabling natural language interaction with your dbt project's health.134MIT
- FlicenseNot gradedqualityBmaintenanceEnables AI to automatically perform Root Cause Analysis for app issues (e.g., sessions not recording, heatmap empty, replica drift) by querying logs, MongoDB, Shopify, and rendering rrweb replays.-