Skip to main content
Glama
Data Engineering MCP Server

Data Engineering MCP Server

by suniljavadi

Data Engineering MCP Server

Status: Completed local portfolio implementation; synthetic data and stdio transport

Status: Completed local portfolio implementation; synthetic data and stdio transport

A small, runnable Model Context Protocol server for data-engineering investigation workflows. It uses the official MCP Python SDK, Pydantic validation, SQLite synthetic data, and a service/repository architecture that is easy to study and replace with production adapters.

What MCP Is

MCP is an open protocol that lets an AI host discover and use external context and capabilities through a standard server interface. This project demonstrates all three primitives:

Primitive

Meaning

Example

Tool

An operation the model may invoke

get_job_status

Resource

Read-only context addressed by a URI

resource://jobs

Prompt

A reusable interaction template

investigate_etl_failure

The MCP client discovers capabilities, chooses a tool from its schema, sends structured arguments, and receives structured content. MCP is complementary to function calling: function calling is usually a model/API feature, while MCP standardizes how tools and context are exposed by an external server and reused by many hosts.

Related MCP server: Claude MCP Data Engineer Server

Problem And Solution

Data engineers repeatedly inspect ETL status, logs, schemas, incidents, runbooks, and read-only data. This server exposes those operations once so an MCP-compatible AI client can use them without a custom integration for every application.

flowchart TD
    A[AI Client / MCP Host] -->|stdio MCP protocol| B[FastMCP Server]
    B --> C[Tools]
    B --> D[Resources]
    B --> E[Prompts]
    C --> F[Application Services]
    F --> G[Repository]
    G --> H[(Synthetic SQLite)]

Tools

  • get_job_status(job_name): latest status, run time, duration, and records.

  • get_job_logs(job_name, run_id, severity, limit): bounded structured logs.

  • get_job_history(job_name, days): recent executions.

  • get_database_schema(table_name): columns, types, nullability, keys, and indexes.

  • validate_sql(sql): accepts one SELECT or WITH statement only.

  • execute_readonly_sql(sql, limit): validates and executes a bounded read-only query.

  • search_documentation(query, top_k): searches synthetic runbooks.

  • search_incidents(query, limit): searches historical synthetic incidents.

  • analyze_job_failure(job_name, run_id): deterministic evidence gathering, not an autonomous agent.

All tool errors are structured and internal stack traces are kept in server logs. Audit logs include request ID, tool, status, duration, and non-sensitive argument metadata.

Resources And Prompts

Resources: resource://jobs, resource://incidents, resource://database/schema, and resource://documentation. A resource provides contextual read-only data; it does not perform an action or decide how the data should be used.

Prompts: investigate_etl_failure(job_name, run_id) and analyze_sql(sql). A prompt is a reusable workflow instruction for an MCP host. It guides tool selection but does not itself execute the investigation.

Quick Start

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
python scripts/init_db.py
python -m data_engineering_mcp.server

The server uses stdio, so it should be launched by an MCP host rather than opened in a browser. Run the real MCP client demo with:

python -m data_engineering_mcp.client_demo

The demo initializes an MCP session, lists tools/resources/prompts, and invokes three tools. The core project requires no LLM API key.

Docker

docker compose up --build

The compose service is intentionally stdio-oriented. An MCP host that supports launching container commands can run the container as its MCP server process. An HTTP transport can be added later when a deployment target requires a remotely reachable server.

Configuration And Safety

Configuration can be supplied through environment variables: DATABASE_URL (SQLite URLs only), LOG_LEVEL, SQL_QUERY_TIMEOUT_SECONDS (1–300), MAX_QUERY_ROWS (1–100), and MAX_LOG_ROWS (1–100). A local .env file is also supported for development; .env and .env.* files are ignored by Git. Never put production credentials in a checked-in file. Only synthetic data is included. Credentials are never returned by schema tools.

SQL execution is defense-in-depth: it accepts only a single SELECT or WITH, rejects write and administrative keywords, runs the server connection in SQLite query-only mode, interrupts expensive SQL after the configured timeout, and enforces a row limit. Production deployments should additionally use a dedicated read-only database identity and database-enforced statement timeouts.

Data And Tests

The seed creates 10 ETL jobs, 120 executions, 360 logs, 35 incidents, 10 documents, and relational sample tables (customers, products, orders, order_items).

python -m pytest -q

The 20 evaluation scenarios are in evaluation/scenarios.json. They cover status, failure analysis, schema, searches, safe SQL, bounded queries, invalid input, and error cases. A full production evaluation would also measure tool-selection accuracy, argument accuracy, latency, and false-positive SQL rejection.

Architecture And Production Evolution

The MCP decorators are adapters only. The path is MCP tool -> service -> repository -> database, so replacing synthetic SQLite with PostgreSQL, a monitoring API, Confluence, Jira, or Azure adapters does not require rewriting the MCP layer. The current integrations are synthetic; no real vendor integration is claimed.

Production improvements would include authentication at the host boundary, per-tool authorization, a real read-only database role, rate limiting, query cancellation, distributed audit logs, metrics/traces, secret management, and contract tests against each adapter.

Interview Preparation

60-second explanation: This is an official-SDK MCP server for data-engineering support. An MCP host discovers nine typed tools, four read-only resources, and two reusable prompts over stdio. Tools call Pydantic-validated application services backed by synthetic SQLite. SQL is restricted to bounded read-only queries, errors are structured, and audit logs capture request metadata.

Five-minute architecture: The host starts the stdio server and initializes an MCP session. FastMCP publishes schemas and capability metadata. The host selects a tool and sends JSON arguments. The tool adapter calls a service, which validates policy and delegates to a repository. The repository uses SQLite. Results return as structured JSON content. Resources expose catalogs by URI, while prompts provide reusable workflows. In production, adapters can target monitoring, warehouse, documentation, and incident systems independently.

Common interview answers: MCP is a reusable protocol boundary, not an LLM; a server exposes capabilities and a client/host consumes them; tools are action-oriented, resources are contextual, and prompts are workflow templates. Security comes from narrow schemas, authorization, read-only identities, limits, timeouts, validation, and audit logs. If a tool fails, the client receives a safe structured error while detailed diagnostics stay server-side. Scale by making services stateless, moving state to managed stores, adding connection pooling and rate limits, and horizontally scaling transport workers. MCP is not needed for a single internal function call or when a stable ordinary API already fully solves the integration.

Difficult questions to practice: How is tool discovery different from REST documentation? Where should authorization run? How do you prevent prompt injection from tool output? How do you handle pagination? How do you version schemas? How do you cancel long queries? How do you test host compatibility? How do you trace one request across tools? How do you isolate tenants? How do you handle partial failure? How do you select a read replica? How do you prevent data exfiltration? How do you rotate secrets? How do you cache resources safely? How do you roll out a breaking tool change?

Scenario questions: diagnose a timeout, explain a schema mismatch, reject an unsafe query, handle an unknown run, select between a resource and a tool, investigate an incident with evidence, control a high-cost query, recover from a downstream outage, explain a failed prompt workflow, and migrate SQLite to PostgreSQL.

Resume Relevance

Demonstrates the official MCP Python SDK, typed tools/resources/prompts, Pydantic validation, read-only SQL controls, service/repository separation, synthetic ETL data, structured errors, audit metadata, pytest, and Docker packaging.

Sunil Javadi · GitHub · Portfolio · LinkedIn

Local Git Workflow

git add .
git commit -m "Build data engineering MCP server"
git push origin main

Resume bullets: built an official MCP Python server exposing typed data-engineering tools, resources, and prompts; implemented read-only SQL policy, bounded execution, structured errors, and audit logging; generated deterministic synthetic ETL data and an MCP stdio client with pytest coverage and Docker packaging.

Limitations

This is a local portfolio and interview project. It does not provide real cloud, warehouse, Jira, Confluence, authentication, or LLM integrations. SQLite is suitable for study and deterministic tests, not a high-concurrency production control plane.

Available Tools

9 tools
analyze_job_failureC

Prepare deterministic evidence for a failed ETL run.

Inputs: run_id (required), job_name (required). Returns: object

execute_readonly_sqlA

Execute a bounded SELECT or WITH query against the synthetic database.

Inputs: sql (required), limit. Returns: object

get_database_schemaB

Return table columns and indexes without credentials.

Inputs: table_name (required). Returns: object

get_job_historyC

Return recent executions for an ETL job.

Inputs: job_name (required), days. Returns: object

get_job_logsC

Return bounded structured logs for an ETL job execution.

Inputs: job_name (required), limit, run_id, severity. Returns: object

get_job_statusC

Return the latest execution status for an ETL job.

Inputs: job_name (required). Returns: object

search_documentationC

Search synthetic data engineering runbooks.

Inputs: query (required), top_k. Returns: object

search_incidentsC

Search historical synthetic incidents.

Inputs: query (required), limit. Returns: object

validate_sqlC

Check whether SQL is a single safe read-only query.

Inputs: sql (required). Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.1/5.0

    Scored across 9 tools

    Disambiguation4/5

    Most tools target distinct resources and actions: job status, logs, history, schema, SQL validation/execution, documentation, incidents, and failure analysis are clearly separated. The only mild overlap is get_job_status versus get_job_history, since both relate to recent execution state, but the descriptions clarify that one returns the latest status and the other returns a list of executions.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern using lowercase snake_case, such as get_job_status, validate_sql, search_incidents, and analyze_job_failure. The verb varies based on the action, but the structure is uniform and predictable.

    Tool Count5/5

    Nine tools is well-scoped for a data engineering support server. Each tool serves a clear purpose across job inspection, read-only database access, documentation and incident lookup, and failure analysis without unnecessary redundancy.

    Completeness4/5

    The tool surface covers the core diagnostic workflow: inspect job execution, analyze schema, validate and run read-only SQL, search runbooks and incidents, and assemble failure evidence. A minor gap is the lack of a way to list all available jobs or tables directly, but search and schema tools help compensate.

    Maintenance

    ActivityMaintained
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      A
      quality
      D
      maintenance
      Provides tools for AI assistants to manage and diagnose MWAA Airflow environments, EMR Serverless jobs, S3 files, and Confluence documentation. It features automated pipeline failure diagnosis and comprehensive log retrieval across these integrated platforms.
      44
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      Provides specialized tools for data engineering tasks like SQL formatting, dbt model generation, and Snowflake table creation. It enables users to analyze CSV data, validate pipeline configurations, and summarize ETL lineage through natural language.
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables autonomous infrastructure health management by exposing tools for retrieving system logs, querying a knowledge base, executing SQL analytics, and simulating system commands, all integrated into an AI-driven incident response workflow.
      -