This MCP server lets coding agents inspect Databricks Spark job runs and diagnose performance issues using execution evidence.
Fetch Databricks run context, including run metadata, task details, stage metrics, SQL text, and Spark plan evidence.
Analyze a run by ID for anti-patterns, shuffle/skew/spill/GC metrics per stage, query-plan issues, and cluster sizing versus data volume, CPU, and memory.
Deep-dive individual Spark stages for shuffle/skew/spill/GC numbers, slowest tasks, anti-patterns, and detailed query-plan analysis.
Analyze supplied run data directly, including task metrics, Spark plans, executors, stages, Spark config, stage plans, and cluster specs.
Use diagnose_job as an alias for analyze_run in natural-language agent workflows.
Optionally customize pattern rules and thresholds to match team risk profiles.
Enables inspecting Databricks Spark job runs, query plans, stage behavior, and cluster bottlenecks. Provides tools for run-level and stage-level analysis, including evidence collection for skew, spill, joins, filters, shuffle pressure, task stragglers, and cluster sizing.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Databricks Spark Analyzer MCP Analyze Databricks run 12345 for spill, skew, and shuffle issues."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A lightweight MCP server for Databricks that helps coding agents inspect Databricks Spark job runs, query plans, stage behavior, and cluster bottlenecks using real execution evidence.
This project is built as an MCP server first. It can be connected to any MCP-compatible coding agent, IDE, or local workflow. An example integration with OpenCode is provided below.
Related MCP server: Databricks MCP Server
Databricks run-level Spark analysis via MCP tools
stage-level investigation and evidence collection
query-plan review for skew, spill, joins, filters, window logic, and shuffle pressure
structured output that can be consumed by any AI agent, not only OpenCode
optional, user-defined pattern rules and thresholds
A Spark job often fails or slows down because the evidence is spread across many places: stage metrics, SQL plans, executor data, cluster config, and logs. This server brings that evidence into a single MCP interface that any agent can query consistently.
git clone https://github.com/ibenlems/databricks-spark-analyzer-mcp.git
cd databricks-spark-analyzer-mcp
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e '.[dev]'If you prefer uv:
uv venv .venv
source .venv/bin/activate
uv pip install -e '.[dev]'export DATABRICKS_HOST="https://<your-workspace-host>"
export DATABRICKS_TOKEN="<your-personal-access-token>"databricks-spark-analyzer-mcpOr:
python -m databricks_mcp_serverThis project is designed to be generic. It is not tied to OpenCode. The same server can be plugged into any MCP client that supports local stdio or similar tool execution.
Typical MCP configuration pattern:
{
"mcpServers": {
"databricks-spark-analyzer": {
"command": "/absolute/path/to/databricks-spark-analyzer-mcp/.venv/bin/databricks-spark-analyzer-mcp",
"env": {
"DATABRICKS_HOST": "https://<your-workspace-host>",
"DATABRICKS_TOKEN": "<your-token>"
}
}
}
}The exact JSON shape depends on the agent, but the important point is this: the server is reusable across many environments. OpenCode is just a concrete example, not a requirement.
OpenCode is a convenient reference integration, but not the only supported use case.
Example file: examples/opencode.jsonc
{
"$schema": "https://opencode.ai/config.json",
"model": "github-copilot/claude-sonnet-5.5",
"mcp": {
"databricks-spark-analyzer": {
"type": "local",
"command": [
"/absolute/path/to/databricks-spark-analyzer-mcp/.venv/bin/databricks-spark-analyzer-mcp"
],
"enabled": true,
"environment": {
"DATABRICKS_HOST": "https://<your-workspace-host>",
"DATABRICKS_TOKEN": "<your-token>"
}
}
}
}Typical OpenCode config path:
~/.config/opencode/opencode.jsoncThen restart OpenCode and the tools should appear as MCP capabilities.
The server exposes tools such as:
analyze_run_by_id(run_id, job_id=None, stage_plans=None)
analyze_stage(run_id, stage_id, job_id=None)
fetch_job_run_context(run_id, job_id=None)
analyze_run(...)
These let an agent fetch Databricks run evidence and summarize Spark patterns, cluster characteristics, task skew, spill, joins, and slow stages.
Use the Databricks Spark Analyzer MCP tool to inspect run 12345 in job 99.
Focus on spill, skew, shuffle pressure, missing broadcast joins, and cluster sizing.
Return only the highest-priority issues with evidence and remediation suggestions.This is the kind of output the tool is designed to generate after a run-level analysis:
Status was SUCCESS. The run had 102 stages and 3,996 tasks, with 77.5 min of total task time. Input was 64.6 GiB, shuffle write 4.67 GiB, memory spill 25.6 GiB and disk spill 1.47 GiB. GC was 6.2% of task time, and the cluster ran Spark 15.4 without Photon on 8 cores.
Limits of this analysis
- I used the MCP run-level analysis. I did not pull the single slowest SQL execution's final plan or node metrics.
- Task IDs are not available, so I cite stage IDs.
- 33 of 102 stages have no query plan, because 12 events exceeded the 2 MiB event-log limit.
Issues that meet the skill thresholds
1. High spill (Pattern 4), severity High
- Stage 157: 10 of 10 tasks spilled. Memory spill was 6.88 GiB and disk spill 371 MiB.
- Stage 161: 9 of 10 tasks spilled. Memory spill was 5.91 GiB and disk spill 382 MiB.
- Likely cause: only 10 shuffle partitions in the heavy stages and limited memory per core.
- Recommendation: raise shuffle partition count or use larger memory-optimized workers.
2. Task straggler (Pattern 14), severity Medium
- Stage 2: median 124 ms, max 3.7 s.
- Recommendation: check hot keys, enable AQE skew handling, and inspect data distribution.
3. Missing broadcast join (Pattern 1), probable
- Evidence: 3 SortMergeJoins and 0 broadcast joins in the plan.
- Recommendation: test a BROADCAST hint on the smaller join side or raise autoBroadcastJoinThreshold.
Additional observations
- Window with no PARTITION BY: this is the highest-priority fix.
- Repeated scans of the same large table are visible and should be cached or materialized once.This project includes an opinionated set of Spark anti-pattern checks, with example pattern definitions under:
These include patterns such as:
missing broadcast join
skewed partitions
high spill
full scans without partition/date filters
small files / too many tasks
excessive shuffle volume
missing partition keys in window logic
task stragglers
repeated scans and wide scans
You can add, remove, or tighten any of these rules depending on your workload.
The analyzer can be customized to match your team’s risk profile. In practice, the signal is usually built from three things:
pattern name
severity
threshold or condition
Example:
PATTERNS = [
{
"id": "high_spill",
"name": "High spill",
"severity": "high",
"condition": lambda s: s.get("memory_spill_gib", 0) > 4.0 or s.get("disk_spill_gib", 0) > 0.5,
"recommendation": "Increase shuffle partitions or use larger memory-optimized executors."
},
{
"id": "window_without_partition",
"name": "Window without PARTITION BY",
"severity": "critical",
"condition": lambda s: "RunningWindowFunctionExec" in (s.get("plan", "") or "") and "PARTITION BY" not in (s.get("plan", "") or ""),
"recommendation": "Add a partition key or rewrite the window logic."
}
]That gives you a simple way to create a stronger or more specific analysis policy without changing the MCP server itself.
pytest -qOr with uv:
uv run pytest -qContributions are welcome. See CONTRIBUTING.md for local setup, testing, and PR guidance.
This project is licensed under the MIT License. See LICENSE for details.
Analyze a Databricks job run from supplied task metrics, Spark plans, executors and cluster specs: anti-patterns, numeric shuffle/skew/spill metrics, plan analysis and cluster sizing.
| Name | Required | Description | Default |
|---|---|---|---|
| run | Yes | ||
| plan | No | ||
| tasks | No | ||
| stages | No | ||
| executors | No | ||
| spark_conf | No | ||
| stage_plans | No | ||
| cluster_specs | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and does disclose what analyses are performed (anti-patterns, numeric metrics, plan analysis, sizing), which is useful behavioral context. However, it never states that the tool is read-only/non-mutating, what happens if inputs are partial or mismatched, or any prerequisite relationships between inputs — significant gaps for an 8-parameter tool with zero annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with a colon-delimited output list — no filler and no redundancy. It is dense but every clause earns its place; slight deduction for packing multiple analyses into one long sentence rather than structuring them.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex analysis tool with 8 inputs including nested objects, 0% schema description coverage and no annotations, the description is too thin. It doesn't explain the expected shape of the required 'run' object or how plan/tasks/stages/spark_conf/stage_plans/cluster_specs feed the analyses. The output schema covers return values, but input-side completeness is lacking.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, but it only loosely alludes to four inputs (task metrics, Spark plans, executors, cluster specs) without using parameter names or formats. It omits the required 'run' object entirely and never mentions stages, spark_conf, or stage_plans — a major gap for 8 parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Analyze) and resource (a Databricks job run), and enumerates the analyses produced: anti-patterns, shuffle/skew/spill metrics, plan analysis and cluster sizing. It implicitly differentiates from analyze_run_by_id via 'from supplied task metrics...' (i.e., input supplied rather than fetched), but never names any sibling explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to use this tool versus analyze_run_by_id, analyze_stage or diagnose_job, and no prerequisites or exclusion conditions. The only guidance is the implicit 'from supplied ... ' phrase, leaving the agent to infer that this is for pre-collected data.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch one Databricks job run (Spark UI or event logs plus cluster config) and return a complete analysis: anti-patterns, shuffle/skew/spill/GC numbers per stage, detailed query-plan analysis for bad stages, and cluster sizing versus data volume, CPU and memory.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | No | ||
| run_id | Yes | ||
| stage_plans | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It discloses that the tool reads Spark UI or event logs plus cluster config and returns detailed analysis, which is useful, but it does not state whether the operation is read-only, what permissions or access are required, how long it may take, or how errors are handled.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that efficiently lists the returned analysis dimensions without wasted preamble. It is dense but appropriate for the amount of information conveyed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is complex and has no annotations, but an output schema exists so return values need not be fully explained. The description covers the analysis scope reasonably well, yet it leaves key gaps around parameter meanings, sibling-tool selection, and operational constraints.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for undocumented parameters. It only indirectly implies the required run_id via the tool name and 'one Databricks job run'; it does not explain the optional job_id or stage_plans parameters at all.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: fetch one Databricks job run and return a complete analysis including anti-patterns, stage metrics, query plans, and sizing. It is clear what the tool does, but it does not distinguish itself from the sibling analyze_run tool or explain how it differs from analyze_stage/diagnose_job.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage by saying it fetches one run and returns a complete analysis, but it gives no explicit when-to-use guidance, no exclusions, and no alternatives among the sibling tools such as analyze_run or analyze_stage. An agent must infer when this tool is preferred over the others.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deep-dive one Spark stage of a run: shuffle/skew/spill/GC numbers, slowest tasks, anti-patterns and a detailed analysis of the stage's query plan.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | No | ||
| run_id | Yes | ||
| stage_id | Yes |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does disclose substantive content: the metric families analyzed, slowest-task inspection, anti-pattern detection, and query-plan analysis. However, it says nothing about execution cost/latency (a 'deep-dive' over a large stage is plausibly expensive), whether it works on incomplete or running runs, or required access scope.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence naming the verb, scope, and output content with no filler. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return shape needn't be described, and the description does characterize what the analysis surfaces. But for a 3-parameter tool with zero schema description coverage and no annotations, the unexplained job_id and missing access/cost guidance leave real gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate and largely fails. The phrase 'one Spark stage of a run' loosely implies run_id and stage_id, but the optional job_id is never mentioned and its disambiguation role is left entirely unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Deep-dive one Spark stage of a run') and enumerates what the analysis covers (shuffle/skew/spill/GC, slowest tasks, anti-patterns, query plan). The 'one Spark stage' scoping implicitly contrasts with analyze_run's whole-run scope, but no sibling is named explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer this is the stage-level drill-down tool to reach for after a run-level analysis. There is no explicit when-to-use guidance, no mention of when to prefer diagnose_job or analyze_run, and no prerequisites or cost warnings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Alias for analyze_run intended for natural-language agent workflows.
| Name | Required | Description | Default |
|---|---|---|---|
| run | Yes | ||
| plan | No | ||
| tasks | No | ||
| stages | No | ||
| executors | No | ||
| spark_conf | No | ||
| stage_plans | No | ||
| cluster_specs | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral disclosure burden. It says nothing about whether this is read-only, what it returns, side effects, permissions, or how it differs behaviorally from analyze_run. Calling it an alias does not disclose behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that is front-loaded and contains no filler. However, its brevity leaves the definition under-specified rather than fully efficient given the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 8 parameters, nested objects, no annotations, and only an output schema. The description does not compensate for the missing schema descriptions or annotation coverage. It is far too incomplete for an agent to call the tool correctly without external knowledge.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are 8 parameters with 0% schema description coverage, and the description adds no parameter-level meaning. Required parameter 'run' and optional nested objects like tasks, stages, executors, and cluster_specs are unexplained. This is a severe gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies the tool as an alias for analyze_run, which gives some relational purpose but does not state what the tool actually does or what analyze_run does. An agent must already know the sibling's semantics to understand diagnose_job. It weakly distinguishes itself by framing as an alias, but the core purpose remains vague.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It mentions 'natural-language agent workflows' as an intended context, but provides no explicit when-to-use rules, no when-not-to-use rules, and no comparison to analyze_run or other siblings. The agent is left to infer selection criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the full Databricks run context: run metadata, task details, stage-level metrics, SQL text, and Spark plan evidence ready for AI analysis.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | No | ||
| run_id | Yes |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does disclose what the response contains (metadata, metrics, SQL, Spark plan), which is useful behavioral context, but says nothing about permissions, payload size/latency, or whether the call is strictly read-only and side-effect free.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence that lists the returned artifacts without filler. Dense but every clause adds identifying detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values needn't be re-explained, but with zero parameter documentation and no annotation coverage the definition leaves an agent guessing about the two arguments and about when to prefer this over analyze_run_by_id or diagnose_job.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description never mentions run_id or job_id at all. With no syntax, meaning, or explanation of how optional job_id relates to required run_id, the description fails to compensate for the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Return') and resource ('full Databricks run context') and enumerates the payload: run metadata, task details, stage metrics, SQL text, Spark plan. This clearly separates it from the analyze_* siblings as a raw-context fetch, though it never names them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'ready for AI analysis' faintly implies this is the fetch step preceding analyze_run or diagnose_job, but there is no explicit when-to-use, when-not, or alternative named among four closely related siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0analyze_runanalyze_run_by_idanalyze_stagediagnose_jobfetch_job_run_contextScored across 5 tools
diagnose_job is explicitly documented as an alias for analyze_run, so two tools do the same thing by design. analyze_run (metrics supplied by caller) vs analyze_run_by_id (fetches the run) and fetch_job_run_context (returns raw-ish context that analyze_run_by_id also fetches internally) create overlapping boundaries an agent must infer from subtle wording.
All names are snake_case verb_noun (analyze_run, analyze_run_by_id, analyze_stage, fetch_job_run_context, diagnose_job), which is predictable. The only wrinkle is diagnose_job, whose verb implies a different action than the analyze_* family despite being an alias.
Five tools is a tight, well-scoped set for Spark run analysis. The only waste is diagnose_job occupying a slot purely as a duplicate of analyze_run, leaving four effectively unique operations.
The analysis lifecycle is covered from raw metrics, run id, context fetch, and single-stage deep dive, but there is no way to discover run or job ids (no list_jobs/list_runs), which blocks analyze_run_by_id for an agent starting cold. Cross-run comparison or recommendation export is also absent.
Your Databricks Lakehouse in natural language: run SQL on your SQL warehouses, track long-running qu
AI agent observability for production traces, natural-language insights, and improvement loops.
Data + AI observability — monitor and troubleshoot production-grade agents and the context they use.
Read-only AI coding tools for change verification, release readiness, capacity, and guidance.
