Data Engineering MCP Server
Data Engineering MCP Server lets an MCP host investigate synthetic data-engineering workflows through typed tools, read-only resources, and reusable prompts.
Check an ETL job’s latest status, recent history, and bounded structured logs.
Inspect database table schemas, columns, keys, and indexes without credentials.
Validate SQL as a single safe read-only query and execute bounded SELECT/WITH queries.
Search synthetic runbooks and historical incidents.
Prepare deterministic evidence for a failed ETL run.
Access read-only context via resources: jobs, incidents, database schema, and documentation.
Use prompts for ETL failure investigation and SQL analysis workflows.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Data Engineering MCP Server what went wrong with the orders ETL job this morning?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Data Engineering MCP Server
Status: Completed local portfolio implementation; synthetic data and stdio transport
Status: Completed local portfolio implementation; synthetic data and stdio transport
A small, runnable Model Context Protocol server for data-engineering investigation workflows. It uses the official MCP Python SDK, Pydantic validation, SQLite synthetic data, and a service/repository architecture that is easy to study and replace with production adapters.
What MCP Is
MCP is an open protocol that lets an AI host discover and use external context and capabilities through a standard server interface. This project demonstrates all three primitives:
Primitive | Meaning | Example |
Tool | An operation the model may invoke |
|
Resource | Read-only context addressed by a URI |
|
Prompt | A reusable interaction template |
|
The MCP client discovers capabilities, chooses a tool from its schema, sends structured arguments, and receives structured content. MCP is complementary to function calling: function calling is usually a model/API feature, while MCP standardizes how tools and context are exposed by an external server and reused by many hosts.
Related MCP server: Claude MCP Data Engineer Server
Problem And Solution
Data engineers repeatedly inspect ETL status, logs, schemas, incidents, runbooks, and read-only data. This server exposes those operations once so an MCP-compatible AI client can use them without a custom integration for every application.
flowchart TD
A[AI Client / MCP Host] -->|stdio MCP protocol| B[FastMCP Server]
B --> C[Tools]
B --> D[Resources]
B --> E[Prompts]
C --> F[Application Services]
F --> G[Repository]
G --> H[(Synthetic SQLite)]Tools
get_job_status(job_name): latest status, run time, duration, and records.get_job_logs(job_name, run_id, severity, limit): bounded structured logs.get_job_history(job_name, days): recent executions.get_database_schema(table_name): columns, types, nullability, keys, and indexes.validate_sql(sql): accepts oneSELECTorWITHstatement only.execute_readonly_sql(sql, limit): validates and executes a bounded read-only query.search_documentation(query, top_k): searches synthetic runbooks.search_incidents(query, limit): searches historical synthetic incidents.analyze_job_failure(job_name, run_id): deterministic evidence gathering, not an autonomous agent.
All tool errors are structured and internal stack traces are kept in server logs. Audit logs include request ID, tool, status, duration, and non-sensitive argument metadata.
Resources And Prompts
Resources: resource://jobs, resource://incidents, resource://database/schema, and resource://documentation. A resource provides contextual read-only data; it does not perform an action or decide how the data should be used.
Prompts: investigate_etl_failure(job_name, run_id) and analyze_sql(sql). A prompt is a reusable workflow instruction for an MCP host. It guides tool selection but does not itself execute the investigation.
Quick Start
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
python scripts/init_db.py
python -m data_engineering_mcp.serverThe server uses stdio, so it should be launched by an MCP host rather than opened in a browser. Run the real MCP client demo with:
python -m data_engineering_mcp.client_demoThe demo initializes an MCP session, lists tools/resources/prompts, and invokes three tools. The core project requires no LLM API key.
Docker
docker compose up --buildThe compose service is intentionally stdio-oriented. An MCP host that supports launching container commands can run the container as its MCP server process. An HTTP transport can be added later when a deployment target requires a remotely reachable server.
Configuration And Safety
Configuration can be supplied through environment variables: DATABASE_URL (SQLite URLs only), LOG_LEVEL, SQL_QUERY_TIMEOUT_SECONDS (1–300), MAX_QUERY_ROWS (1–100), and MAX_LOG_ROWS (1–100). A local .env file is also supported for development; .env and .env.* files are ignored by Git. Never put production credentials in a checked-in file. Only synthetic data is included. Credentials are never returned by schema tools.
SQL execution is defense-in-depth: it accepts only a single SELECT or WITH, rejects write and administrative keywords, runs the server connection in SQLite query-only mode, interrupts expensive SQL after the configured timeout, and enforces a row limit. Production deployments should additionally use a dedicated read-only database identity and database-enforced statement timeouts.
Data And Tests
The seed creates 10 ETL jobs, 120 executions, 360 logs, 35 incidents, 10 documents, and relational sample tables (customers, products, orders, order_items).
python -m pytest -qThe 20 evaluation scenarios are in evaluation/scenarios.json. They cover status, failure analysis, schema, searches, safe SQL, bounded queries, invalid input, and error cases. A full production evaluation would also measure tool-selection accuracy, argument accuracy, latency, and false-positive SQL rejection.
Architecture And Production Evolution
The MCP decorators are adapters only. The path is MCP tool -> service -> repository -> database, so replacing synthetic SQLite with PostgreSQL, a monitoring API, Confluence, Jira, or Azure adapters does not require rewriting the MCP layer. The current integrations are synthetic; no real vendor integration is claimed.
Production improvements would include authentication at the host boundary, per-tool authorization, a real read-only database role, rate limiting, query cancellation, distributed audit logs, metrics/traces, secret management, and contract tests against each adapter.
Interview Preparation
60-second explanation: This is an official-SDK MCP server for data-engineering support. An MCP host discovers nine typed tools, four read-only resources, and two reusable prompts over stdio. Tools call Pydantic-validated application services backed by synthetic SQLite. SQL is restricted to bounded read-only queries, errors are structured, and audit logs capture request metadata.
Five-minute architecture: The host starts the stdio server and initializes an MCP session. FastMCP publishes schemas and capability metadata. The host selects a tool and sends JSON arguments. The tool adapter calls a service, which validates policy and delegates to a repository. The repository uses SQLite. Results return as structured JSON content. Resources expose catalogs by URI, while prompts provide reusable workflows. In production, adapters can target monitoring, warehouse, documentation, and incident systems independently.
Common interview answers: MCP is a reusable protocol boundary, not an LLM; a server exposes capabilities and a client/host consumes them; tools are action-oriented, resources are contextual, and prompts are workflow templates. Security comes from narrow schemas, authorization, read-only identities, limits, timeouts, validation, and audit logs. If a tool fails, the client receives a safe structured error while detailed diagnostics stay server-side. Scale by making services stateless, moving state to managed stores, adding connection pooling and rate limits, and horizontally scaling transport workers. MCP is not needed for a single internal function call or when a stable ordinary API already fully solves the integration.
Difficult questions to practice: How is tool discovery different from REST documentation? Where should authorization run? How do you prevent prompt injection from tool output? How do you handle pagination? How do you version schemas? How do you cancel long queries? How do you test host compatibility? How do you trace one request across tools? How do you isolate tenants? How do you handle partial failure? How do you select a read replica? How do you prevent data exfiltration? How do you rotate secrets? How do you cache resources safely? How do you roll out a breaking tool change?
Scenario questions: diagnose a timeout, explain a schema mismatch, reject an unsafe query, handle an unknown run, select between a resource and a tool, investigate an incident with evidence, control a high-cost query, recover from a downstream outage, explain a failed prompt workflow, and migrate SQLite to PostgreSQL.
Resume Relevance
Demonstrates the official MCP Python SDK, typed tools/resources/prompts, Pydantic validation, read-only SQL controls, service/repository separation, synthetic ETL data, structured errors, audit metadata, pytest, and Docker packaging.
Author and Related Work
Sunil Javadi · GitHub · Portfolio · LinkedIn
Local Git Workflow
git add .
git commit -m "Build data engineering MCP server"
git push origin mainResume bullets: built an official MCP Python server exposing typed data-engineering tools, resources, and prompts; implemented read-only SQL policy, bounded execution, structured errors, and audit logging; generated deterministic synthetic ETL data and an MCP stdio client with pytest coverage and Docker packaging.
Limitations
This is a local portfolio and interview project. It does not provide real cloud, warehouse, Jira, Confluence, authentication, or LLM integrations. SQLite is suitable for study and deterministic tests, not a high-concurrency production control plane.
Available Tools
9 toolsanalyze_job_failureC
Prepare deterministic evidence for a failed ETL run.
Inputs: run_id (required), job_name (required). Returns: object
execute_readonly_sqlA
Execute a bounded SELECT or WITH query against the synthetic database.
Inputs: sql (required), limit. Returns: object
get_database_schemaB
Return table columns and indexes without credentials.
Inputs: table_name (required). Returns: object
get_job_historyC
Return recent executions for an ETL job.
Inputs: job_name (required), days. Returns: object
get_job_logsC
Return bounded structured logs for an ETL job execution.
Inputs: job_name (required), limit, run_id, severity. Returns: object
get_job_statusC
Return the latest execution status for an ETL job.
Inputs: job_name (required). Returns: object
search_documentationC
Search synthetic data engineering runbooks.
Inputs: query (required), top_k. Returns: object
search_incidentsC
Search historical synthetic incidents.
Inputs: query (required), limit. Returns: object
validate_sqlC
Check whether SQL is a single safe read-only query.
Inputs: sql (required). Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Most tools target distinct resources and actions: job status, logs, history, schema, SQL validation/execution, documentation, incidents, and failure analysis are clearly separated. The only mild overlap is get_job_status versus get_job_history, since both relate to recent execution state, but the descriptions clarify that one returns the latest status and the other returns a list of executions.
All tool names follow a consistent verb_noun pattern using lowercase snake_case, such as get_job_status, validate_sql, search_incidents, and analyze_job_failure. The verb varies based on the action, but the structure is uniform and predictable.
Nine tools is well-scoped for a data engineering support server. Each tool serves a clear purpose across job inspection, read-only database access, documentation and incident lookup, and failure analysis without unnecessary redundancy.
The tool surface covers the core diagnostic workflow: inspect job execution, analyze schema, validate and run read-only SQL, search runbooks and incidents, and assemble failure evidence. A minor gap is the lack of a way to list all available jobs or tables directly, but search and schema tools help compensate.
Maintenance
Related MCP Connectors
List datasets, schemas, run APL queries, and use prompts for exploration, anomalies, and monitoring.
Query your app's logs, metrics and incidents from your AI assistant.
Lets AI agents analyze API metrics, request logs, and traces, and manage apps and alerts.
AI data analyst for product analytics, billing and warehouse data. Investigations and watchers.
Related MCP Servers
- FlicenseAqualityDmaintenanceProvides tools for AI assistants to manage and diagnose MWAA Airflow environments, EMR Serverless jobs, S3 files, and Confluence documentation. It features automated pipeline failure diagnosis and comprehensive log retrieval across these integrated platforms.44-
- FlicenseNot gradedqualityDmaintenanceProvides specialized tools for data engineering tasks like SQL formatting, dbt model generation, and Snowflake table creation. It enables users to analyze CSV data, validate pipeline configurations, and summarize ETL lineage through natural language.-
- AlicenseNot gradedqualityDmaintenanceProvides AI agents with a toolset to query model inventories, trace dependencies, and analyze the impact of changes across machine learning models and data pipelines.99 PyPI17Apache 2.0
- FlicenseNot gradedqualityDmaintenanceEnables autonomous infrastructure health management by exposing tools for retrieving system logs, querying a knowledge base, executing SQL analytics, and simulating system commands, all integrated into an AI-driven incident response workflow.-