Skip to main content
Glama
Databricks Spark Analyzer MCP

Databricks Spark Analyzer MCP

by ibenlems

Databricks Spark Analyzer MCP

Python 3.10+ Apache Spark License: MIT

A lightweight MCP server for Databricks that helps coding agents inspect Databricks Spark job runs, query plans, stage behavior, and cluster bottlenecks using real execution evidence.

This project is built as an MCP server first. It can be connected to any MCP-compatible coding agent, IDE, or local workflow. An example integration with OpenCode is provided below.

Table of contents

Related MCP server: Databricks MCP Server

What this project gives you

  • Databricks run-level Spark analysis via MCP tools

  • stage-level investigation and evidence collection

  • query-plan review for skew, spill, joins, filters, window logic, and shuffle pressure

  • structured output that can be consumed by any AI agent, not only OpenCode

  • optional, user-defined pattern rules and thresholds

Why integrate it

A Spark job often fails or slows down because the evidence is spread across many places: stage metrics, SQL plans, executor data, cluster config, and logs. This server brings that evidence into a single MCP interface that any agent can query consistently.

Quick start

1) Install

git clone https://github.com/ibenlems/databricks-spark-analyzer-mcp.git
cd databricks-spark-analyzer-mcp

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e '.[dev]'

If you prefer uv:

uv venv .venv
source .venv/bin/activate
uv pip install -e '.[dev]'

2) Configure environment

export DATABRICKS_HOST="https://<your-workspace-host>"
export DATABRICKS_TOKEN="<your-personal-access-token>"

3) Run the server

databricks-spark-analyzer-mcp

Or:

python -m databricks_mcp_server

MCP integration: use it with any coding agent

This project is designed to be generic. It is not tied to OpenCode. The same server can be plugged into any MCP client that supports local stdio or similar tool execution.

Typical MCP configuration pattern:

{
  "mcpServers": {
    "databricks-spark-analyzer": {
      "command": "/absolute/path/to/databricks-spark-analyzer-mcp/.venv/bin/databricks-spark-analyzer-mcp",
      "env": {
        "DATABRICKS_HOST": "https://<your-workspace-host>",
        "DATABRICKS_TOKEN": "<your-token>"
      }
    }
  }
}

The exact JSON shape depends on the agent, but the important point is this: the server is reusable across many environments. OpenCode is just a concrete example, not a requirement.

OpenCode example (reference only)

OpenCode is a convenient reference integration, but not the only supported use case.

Example file: examples/opencode.jsonc

{
  "$schema": "https://opencode.ai/config.json",
  "model": "github-copilot/claude-sonnet-5.5",
  "mcp": {
    "databricks-spark-analyzer": {
      "type": "local",
      "command": [
        "/absolute/path/to/databricks-spark-analyzer-mcp/.venv/bin/databricks-spark-analyzer-mcp"
      ],
      "enabled": true,
      "environment": {
        "DATABRICKS_HOST": "https://<your-workspace-host>",
        "DATABRICKS_TOKEN": "<your-token>"
      }
    }
  }
}

Typical OpenCode config path:

~/.config/opencode/opencode.jsonc

Then restart OpenCode and the tools should appear as MCP capabilities.

Tools exposed by the server

The server exposes tools such as:

  • analyze_run_by_id(run_id, job_id=None, stage_plans=None)

  • analyze_stage(run_id, stage_id, job_id=None)

  • fetch_job_run_context(run_id, job_id=None)

  • analyze_run(...)

These let an agent fetch Databricks run evidence and summarize Spark patterns, cluster characteristics, task skew, spill, joins, and slow stages.

Example prompt for an integrated agent

Use the Databricks Spark Analyzer MCP tool to inspect run 12345 in job 99.
Focus on spill, skew, shuffle pressure, missing broadcast joins, and cluster sizing.
Return only the highest-priority issues with evidence and remediation suggestions.

Example result format

This is the kind of output the tool is designed to generate after a run-level analysis:

Status was SUCCESS. The run had 102 stages and 3,996 tasks, with 77.5 min of total task time. Input was 64.6 GiB, shuffle write 4.67 GiB, memory spill 25.6 GiB and disk spill 1.47 GiB. GC was 6.2% of task time, and the cluster ran Spark 15.4 without Photon on 8 cores.

Limits of this analysis
- I used the MCP run-level analysis. I did not pull the single slowest SQL execution's final plan or node metrics.
- Task IDs are not available, so I cite stage IDs.
- 33 of 102 stages have no query plan, because 12 events exceeded the 2 MiB event-log limit.

Issues that meet the skill thresholds
1. High spill (Pattern 4), severity High
- Stage 157: 10 of 10 tasks spilled. Memory spill was 6.88 GiB and disk spill 371 MiB.
- Stage 161: 9 of 10 tasks spilled. Memory spill was 5.91 GiB and disk spill 382 MiB.
- Likely cause: only 10 shuffle partitions in the heavy stages and limited memory per core.
- Recommendation: raise shuffle partition count or use larger memory-optimized workers.

2. Task straggler (Pattern 14), severity Medium
- Stage 2: median 124 ms, max 3.7 s.
- Recommendation: check hot keys, enable AQE skew handling, and inspect data distribution.

3. Missing broadcast join (Pattern 1), probable
- Evidence: 3 SortMergeJoins and 0 broadcast joins in the plan.
- Recommendation: test a BROADCAST hint on the smaller join side or raise autoBroadcastJoinThreshold.

Additional observations
- Window with no PARTITION BY: this is the highest-priority fix.
- Repeated scans of the same large table are visible and should be cached or materialized once.

What patterns are available

This project includes an opinionated set of Spark anti-pattern checks, with example pattern definitions under:

These include patterns such as:

  • missing broadcast join

  • skewed partitions

  • high spill

  • full scans without partition/date filters

  • small files / too many tasks

  • excessive shuffle volume

  • missing partition keys in window logic

  • task stragglers

  • repeated scans and wide scans

You can add, remove, or tighten any of these rules depending on your workload.

Custom patterns and thresholds

The analyzer can be customized to match your team’s risk profile. In practice, the signal is usually built from three things:

  • pattern name

  • severity

  • threshold or condition

Example:

PATTERNS = [
    {
        "id": "high_spill",
        "name": "High spill",
        "severity": "high",
        "condition": lambda s: s.get("memory_spill_gib", 0) > 4.0 or s.get("disk_spill_gib", 0) > 0.5,
        "recommendation": "Increase shuffle partitions or use larger memory-optimized executors."
    },
    {
        "id": "window_without_partition",
        "name": "Window without PARTITION BY",
        "severity": "critical",
        "condition": lambda s: "RunningWindowFunctionExec" in (s.get("plan", "") or "") and "PARTITION BY" not in (s.get("plan", "") or ""),
        "recommendation": "Add a partition key or rewrite the window logic."
    }
]

That gives you a simple way to create a stronger or more specific analysis policy without changing the MCP server itself.

Development and tests

pytest -q

Or with uv:

uv run pytest -q

Contributing

Contributions are welcome. See CONTRIBUTING.md for local setup, testing, and PR guidance.

License

This project is licensed under the MIT License. See LICENSE for details.

Available Tools

5 tools
analyze_runC

Analyze a Databricks job run from supplied task metrics, Spark plans, executors and cluster specs: anti-patterns, numeric shuffle/skew/spill metrics, plan analysis and cluster sizing.

Inputs: run (required), plan, tasks, stages, +4 more. Returns: object

analyze_run_by_idB

Fetch one Databricks job run (Spark UI or event logs plus cluster config) and return a complete analysis: anti-patterns, shuffle/skew/spill/GC numbers per stage, detailed query-plan analysis for bad stages, and cluster sizing versus data volume, CPU and memory.

Inputs: run_id (required), job_id, stage_plans. Returns: object

analyze_stageB

Deep-dive one Spark stage of a run: shuffle/skew/spill/GC numbers, slowest tasks, anti-patterns and a detailed analysis of the stage's query plan.

Inputs: run_id (required), stage_id (required), job_id. Returns: object

diagnose_jobC

Alias for analyze_run intended for natural-language agent workflows.

Inputs: run (required), plan, tasks, stages, +4 more. Returns: object

fetch_job_run_contextB

Return the full Databricks run context: run metadata, task details, stage-level metrics, SQL text, and Spark plan evidence ready for AI analysis.

Inputs: run_id (required), job_id. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    C2.7/5.0

    Scored across 5 tools

    Disambiguation2/5

    diagnose_job is explicitly documented as an alias for analyze_run, so two tools do the same thing by design. analyze_run (metrics supplied by caller) vs analyze_run_by_id (fetches the run) and fetch_job_run_context (returns raw-ish context that analyze_run_by_id also fetches internally) create overlapping boundaries an agent must infer from subtle wording.

    Naming Consistency4/5

    All names are snake_case verb_noun (analyze_run, analyze_run_by_id, analyze_stage, fetch_job_run_context, diagnose_job), which is predictable. The only wrinkle is diagnose_job, whose verb implies a different action than the analyze_* family despite being an alias.

    Tool Count4/5

    Five tools is a tight, well-scoped set for Spark run analysis. The only waste is diagnose_job occupying a slot purely as a duplicate of analyze_run, leaving four effectively unique operations.

    Completeness3/5

    The analysis lifecycle is covered from raw metrics, run id, context fetch, and single-stage deep dive, but there is no way to discover run or job ids (no list_jobs/list_runs), which blocks analyze_run_by_id for an agent starting cold. Cross-run comparison or recommendation export is also absent.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      Not graded
      maintenance
      Exposes Spark History Server metrics and metadata as tools for LLM-based analysis of Spark applications. It enables deep optimization of Spark jobs by providing access to job summaries, stage details, SQL execution plans, and executor performance.
      -
    • A
      license
      Not graded
      quality
      Not graded
      maintenance
      Enables LLMs to manage Databricks clusters, jobs, and notebooks while providing schema references for gold and silver data layers. It allows agents to perform data discovery and execute SQL queries directly against Databricks environments.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Exposes Spark History Server data as tools for AI agents, enabling natural language querying of Spark applications, jobs, stages, and performance metrics.
      207
      Apache 2.0
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables AI agents to explore Unity Catalog metadata, execute SQL queries, and analyze data lineage including notebooks and jobs, empowering autonomous data discovery and query generation in Databricks.
      MIT