Databricks Spark Analyzer MCP
This MCP server lets coding agents inspect Databricks Spark job runs and diagnose performance issues using execution evidence.
Fetch Databricks run context, including run metadata, task details, stage metrics, SQL text, and Spark plan evidence.
Analyze a run by ID for anti-patterns, shuffle/skew/spill/GC metrics per stage, query-plan issues, and cluster sizing versus data volume, CPU, and memory.
Deep-dive individual Spark stages for shuffle/skew/spill/GC numbers, slowest tasks, anti-patterns, and detailed query-plan analysis.
Analyze supplied run data directly, including task metrics, Spark plans, executors, stages, Spark config, stage plans, and cluster specs.
Use
diagnose_jobas an alias foranalyze_runin natural-language agent workflows.Optionally customize pattern rules and thresholds to match team risk profiles.
Enables inspecting Databricks Spark job runs, query plans, stage behavior, and cluster bottlenecks. Provides tools for run-level and stage-level analysis, including evidence collection for skew, spill, joins, filters, shuffle pressure, task stragglers, and cluster sizing.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Databricks Spark Analyzer MCP Analyze Databricks run 12345 for spill, skew, and shuffle issues."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Databricks Spark Analyzer MCP
A lightweight MCP server for Databricks that helps coding agents inspect Databricks Spark job runs, query plans, stage behavior, and cluster bottlenecks using real execution evidence.
This project is built as an MCP server first. It can be connected to any MCP-compatible coding agent, IDE, or local workflow. An example integration with OpenCode is provided below.
Table of contents
Related MCP server: Databricks MCP Server
What this project gives you
Databricks run-level Spark analysis via MCP tools
stage-level investigation and evidence collection
query-plan review for skew, spill, joins, filters, window logic, and shuffle pressure
structured output that can be consumed by any AI agent, not only OpenCode
optional, user-defined pattern rules and thresholds
Why integrate it
A Spark job often fails or slows down because the evidence is spread across many places: stage metrics, SQL plans, executor data, cluster config, and logs. This server brings that evidence into a single MCP interface that any agent can query consistently.
Quick start
1) Install
git clone https://github.com/ibenlems/databricks-spark-analyzer-mcp.git
cd databricks-spark-analyzer-mcp
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e '.[dev]'If you prefer uv:
uv venv .venv
source .venv/bin/activate
uv pip install -e '.[dev]'2) Configure environment
export DATABRICKS_HOST="https://<your-workspace-host>"
export DATABRICKS_TOKEN="<your-personal-access-token>"3) Run the server
databricks-spark-analyzer-mcpOr:
python -m databricks_mcp_serverMCP integration: use it with any coding agent
This project is designed to be generic. It is not tied to OpenCode. The same server can be plugged into any MCP client that supports local stdio or similar tool execution.
Typical MCP configuration pattern:
{
"mcpServers": {
"databricks-spark-analyzer": {
"command": "/absolute/path/to/databricks-spark-analyzer-mcp/.venv/bin/databricks-spark-analyzer-mcp",
"env": {
"DATABRICKS_HOST": "https://<your-workspace-host>",
"DATABRICKS_TOKEN": "<your-token>"
}
}
}
}The exact JSON shape depends on the agent, but the important point is this: the server is reusable across many environments. OpenCode is just a concrete example, not a requirement.
OpenCode example (reference only)
OpenCode is a convenient reference integration, but not the only supported use case.
Example file: examples/opencode.jsonc
{
"$schema": "https://opencode.ai/config.json",
"model": "github-copilot/claude-sonnet-5.5",
"mcp": {
"databricks-spark-analyzer": {
"type": "local",
"command": [
"/absolute/path/to/databricks-spark-analyzer-mcp/.venv/bin/databricks-spark-analyzer-mcp"
],
"enabled": true,
"environment": {
"DATABRICKS_HOST": "https://<your-workspace-host>",
"DATABRICKS_TOKEN": "<your-token>"
}
}
}
}Typical OpenCode config path:
~/.config/opencode/opencode.jsoncThen restart OpenCode and the tools should appear as MCP capabilities.
Tools exposed by the server
The server exposes tools such as:
analyze_run_by_id(run_id, job_id=None, stage_plans=None)analyze_stage(run_id, stage_id, job_id=None)fetch_job_run_context(run_id, job_id=None)analyze_run(...)
These let an agent fetch Databricks run evidence and summarize Spark patterns, cluster characteristics, task skew, spill, joins, and slow stages.
Example prompt for an integrated agent
Use the Databricks Spark Analyzer MCP tool to inspect run 12345 in job 99.
Focus on spill, skew, shuffle pressure, missing broadcast joins, and cluster sizing.
Return only the highest-priority issues with evidence and remediation suggestions.Example result format
This is the kind of output the tool is designed to generate after a run-level analysis:
Status was SUCCESS. The run had 102 stages and 3,996 tasks, with 77.5 min of total task time. Input was 64.6 GiB, shuffle write 4.67 GiB, memory spill 25.6 GiB and disk spill 1.47 GiB. GC was 6.2% of task time, and the cluster ran Spark 15.4 without Photon on 8 cores.
Limits of this analysis
- I used the MCP run-level analysis. I did not pull the single slowest SQL execution's final plan or node metrics.
- Task IDs are not available, so I cite stage IDs.
- 33 of 102 stages have no query plan, because 12 events exceeded the 2 MiB event-log limit.
Issues that meet the skill thresholds
1. High spill (Pattern 4), severity High
- Stage 157: 10 of 10 tasks spilled. Memory spill was 6.88 GiB and disk spill 371 MiB.
- Stage 161: 9 of 10 tasks spilled. Memory spill was 5.91 GiB and disk spill 382 MiB.
- Likely cause: only 10 shuffle partitions in the heavy stages and limited memory per core.
- Recommendation: raise shuffle partition count or use larger memory-optimized workers.
2. Task straggler (Pattern 14), severity Medium
- Stage 2: median 124 ms, max 3.7 s.
- Recommendation: check hot keys, enable AQE skew handling, and inspect data distribution.
3. Missing broadcast join (Pattern 1), probable
- Evidence: 3 SortMergeJoins and 0 broadcast joins in the plan.
- Recommendation: test a BROADCAST hint on the smaller join side or raise autoBroadcastJoinThreshold.
Additional observations
- Window with no PARTITION BY: this is the highest-priority fix.
- Repeated scans of the same large table are visible and should be cached or materialized once.What patterns are available
This project includes an opinionated set of Spark anti-pattern checks, with example pattern definitions under:
These include patterns such as:
missing broadcast join
skewed partitions
high spill
full scans without partition/date filters
small files / too many tasks
excessive shuffle volume
missing partition keys in window logic
task stragglers
repeated scans and wide scans
You can add, remove, or tighten any of these rules depending on your workload.
Custom patterns and thresholds
The analyzer can be customized to match your team’s risk profile. In practice, the signal is usually built from three things:
pattern name
severity
threshold or condition
Example:
PATTERNS = [
{
"id": "high_spill",
"name": "High spill",
"severity": "high",
"condition": lambda s: s.get("memory_spill_gib", 0) > 4.0 or s.get("disk_spill_gib", 0) > 0.5,
"recommendation": "Increase shuffle partitions or use larger memory-optimized executors."
},
{
"id": "window_without_partition",
"name": "Window without PARTITION BY",
"severity": "critical",
"condition": lambda s: "RunningWindowFunctionExec" in (s.get("plan", "") or "") and "PARTITION BY" not in (s.get("plan", "") or ""),
"recommendation": "Add a partition key or rewrite the window logic."
}
]That gives you a simple way to create a stronger or more specific analysis policy without changing the MCP server itself.
Development and tests
pytest -qOr with uv:
uv run pytest -qContributing
Contributions are welcome. See CONTRIBUTING.md for local setup, testing, and PR guidance.
License
This project is licensed under the MIT License. See LICENSE for details.
Available Tools
5 toolsanalyze_runC
Analyze a Databricks job run from supplied task metrics, Spark plans, executors and cluster specs: anti-patterns, numeric shuffle/skew/spill metrics, plan analysis and cluster sizing.
Inputs: run (required), plan, tasks, stages, +4 more. Returns: object
analyze_run_by_idB
Fetch one Databricks job run (Spark UI or event logs plus cluster config) and return a complete analysis: anti-patterns, shuffle/skew/spill/GC numbers per stage, detailed query-plan analysis for bad stages, and cluster sizing versus data volume, CPU and memory.
Inputs: run_id (required), job_id, stage_plans. Returns: object
analyze_stageB
Deep-dive one Spark stage of a run: shuffle/skew/spill/GC numbers, slowest tasks, anti-patterns and a detailed analysis of the stage's query plan.
Inputs: run_id (required), stage_id (required), job_id. Returns: object
diagnose_jobC
Alias for analyze_run intended for natural-language agent workflows.
Inputs: run (required), plan, tasks, stages, +4 more. Returns: object
fetch_job_run_contextB
Return the full Databricks run context: run metadata, task details, stage-level metrics, SQL text, and Spark plan evidence ready for AI analysis.
Inputs: run_id (required), job_id. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
diagnose_job is explicitly documented as an alias for analyze_run, so two tools do the same thing by design. analyze_run (metrics supplied by caller) vs analyze_run_by_id (fetches the run) and fetch_job_run_context (returns raw-ish context that analyze_run_by_id also fetches internally) create overlapping boundaries an agent must infer from subtle wording.
All names are snake_case verb_noun (analyze_run, analyze_run_by_id, analyze_stage, fetch_job_run_context, diagnose_job), which is predictable. The only wrinkle is diagnose_job, whose verb implies a different action than the analyze_* family despite being an alias.
Five tools is a tight, well-scoped set for Spark run analysis. The only waste is diagnose_job occupying a slot purely as a duplicate of analyze_run, leaving four effectively unique operations.
The analysis lifecycle is covered from raw metrics, run id, context fetch, and single-stage deep dive, but there is no way to discover run or job ids (no list_jobs/list_runs), which blocks analyze_run_by_id for an agent starting cold. Cross-run comparison or recommendation export is also absent.
Maintenance
Related MCP Connectors
Your Databricks Lakehouse in natural language: run SQL on your SQL warehouses, track long-running qu
Query your AI agent traces, root-cause verdicts and issues, and run investigations, from your agent.
Lets AI agents analyze API metrics, request logs, and traces, and manage apps and alerts.
AI agent observability for production traces, natural-language insights, and improvement loops.
Related MCP Servers
- FlicenseNot gradedqualityNot gradedmaintenanceExposes Spark History Server metrics and metadata as tools for LLM-based analysis of Spark applications. It enables deep optimization of Spark jobs by providing access to job summaries, stage details, SQL execution plans, and executor performance.-
- AlicenseNot gradedqualityNot gradedmaintenanceEnables LLMs to manage Databricks clusters, jobs, and notebooks while providing schema references for gold and silver data layers. It allows agents to perform data discovery and execute SQL queries directly against Databricks environments.MIT
- AlicenseNot gradedqualityBmaintenanceExposes Spark History Server data as tools for AI agents, enabling natural language querying of Spark applications, jobs, stages, and performance metrics.207Apache 2.0
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to explore Unity Catalog metadata, execute SQL queries, and analyze data lineage including notebooks and jobs, empowering autonomous data discovery and query generation in Databricks.MIT