Ein MCP-Server (Model Context Protocol), der KI-Agenten – Claude Desktop, Claude Code, Devin oder jeden MCP-kompatiblen Client – zwei Fähigkeiten für die Arbeit mit Apache-Spark-Jobs bietet:
🔴 diagnose_spark_failure — ein Spark-Job ist fehlgeschlagen; du erhältst eine Ursachenanalyse und eine konkrete Lösung, basierend auf dem tatsächlichen Fehlerprotokoll und dem spezifischen Code, der fehlgeschlagen ist.
🟢 optimize_spark_performance — ein Spark-Job war erfolgreich, ist aber langsam oder teuer; du erhältst gezielte, evidenzbasierte Optimierungsempfehlungen.
Entwickelt von einem Data Engineer mit über 12 Jahren praktischer Apache-Spark-Erfahrung, um denselben Debugging-Instinkt – „Um welche Datei geht es bei diesem Fehler eigentlich und warum?“ – in einen KI-gestützten Workflow zu bringen.
Spark-Fehler lassen sich normalerweise allein aus dem Log diagnostizieren – aber einen 200-zeiligen Stacktrace zu lesen, ihn in einer großen Codebasis mit mehreren Jobs der richtigen Datei zuzuordnen und zu wissen, welche von einem Dutzend möglicher Ursachen tatsächlich vorliegt, erfordert echte Spark-Erfahrung. Dieses Tool automatisiert diesen ersten Schritt: Es findet den relevanten Code (nicht das gesamte Repository), übergibt ihn zusammen mit dem Log an ein LLM und liefert eine strukturierte Diagnose, die du überprüfen und umsetzen kannst.
Related MCP server: Spark MCP Optimizer
Designentscheidung | Warum das wichtig ist |
Quellenunabhängig – EMR (Cluster + Step-ID) oder ein lokaler Ordner | Funktioniert, ob dein Job auf AWS oder on-premises/lokal läuft |
Anbieterunabhängig – Bedrock, Anthropic, OpenAI oder kein Anbieter | Keine Herstellerbindung; |
Intelligente Dateiauswahl | Große Projekte führen viele Jobs aus – dieses Tool parst den Stacktrace des Fehlerprotokolls (Python und Scala/Java, einschließlich gemischter PySpark-Traces), um nur die spezifischen Dateien einzubeziehen, die an dem Fehler beteiligt sind, begrenzt auf 10 Dateien, anstatt eine gesamte Codebasis in den Prompt zu kippen |
Niemals gebündelte Anmeldedaten | Jeder Benutzer bringt seine eigenen AWS- und/oder LLM-Anmeldedaten mit. Hier wird keine Abrechnung oder Zugriff zwischen Benutzern geteilt |
Mit diesem Beispiel-Scala-Fehlerprotokoll und den zugehörigen Projektdateien liefert diagnose_spark_failure (mit konfiguriertem Anbieter) Folgendes zurück:
ROOT CAUSE:
CustomerHelper.validate() calls .trim() on the "email" field without
checking for null first. Records with a missing email cause a
NullPointerException, which aborts the job after 4 failed task retries.
EVIDENCE:
- Caused by: java.lang.NullPointerException: Cannot invoke "String.trim()"
because "email" is null
- at com.company.jobs.CustomerHelper$.validate(CustomerHelper.scala:22)
- Source shows: email.trim().nonEmpty with no null check beforehand
SUGGESTED FIX:
def validate(row: Row): Boolean = {
val email = Option(row.getAs[String]("email"))
email.exists(_.trim.nonEmpty)
}
CONFIDENCE: HighBeachte, dass das Tool automatisch CustomerHelper.scala einbezogen hat (die Datei, auf die das Caused by des Traces tatsächlich zeigt), nicht das gesamte Projekt oder sogar die Top-Level-Einstiegsdatei CustomerOrderJoin.scala – weil der Stacktrace-Parser den tiefsten relevanten Frame aufgelöst hat.
pip install spark-sense-aiInstalliere Extras nur für das, was du tatsächlich nutzt:
pip install spark-sense-ai[aws] # for EMR source or Bedrock provider
pip install spark-sense-ai[anthropic] # for provider="anthropic"
pip install spark-sense-ai[openai] # for provider="openai"
pip install spark-sense-ai[all] # everythingprovider="none" mit source_type="local" benötigt überhaupt keine Extras – nur die Basis-Abhängigkeit mcp.
# | Log-/Code-Quelle | LLM-Anbieter | Benötigte Extras | AWS-Anmeldedaten erforderlich? |
1 | EMR-Cluster + Step | Bedrock |
| Ja – zum Abrufen und für die Diagnose |
2 | EMR-Cluster + Step | Anthropic / OpenAI |
| Ja – nur zum Abrufen |
3 | Lokaler Ordner | Keiner (Agent analysiert, z. B. in Devin) | none | Nein |
4 | Lokaler Ordner | Anthropic / OpenAI |
| Nein |
AWS-Anmeldedaten werden bei Bedarf automatisch aus deiner bestehenden Umgebung übernommen (aws configure, eine zugewiesene IAM-Rolle oder Standard-Umgebungsvariablen AWS_*) – niemals als Tool-Parameter übergeben.
Bearbeite claude_desktop_config.json:
{
"mcpServers": {
"sparksense": {
"command": "sparksense-mcp",
"env": {
"SPARKSENSE_AWS_REGION": "ap-south-1"
}
}
}
}claude mcp add sparksense -- sparksense-mcpSiehe Devins MCP-Dokumentation für die aktuelle Konfigurationsmethode für deinen Devin-Agentenmodus (Cascade und Devin Local verwenden leicht unterschiedliche Konfigurationsspeicherorte). Richte es genauso wie oben auf den Befehl sparksense-mcp aus.
„Mein Spark-Job ist fehlgeschlagen – EMR-Cluster j-ABC123, Step s-XYZ789. Verwende sparksense, um ihn mit Bedrock zu diagnostizieren.“
„Hier sind das Log meines lokalen Jobs unter
./logs/error.logund der Code unter./src– diagnostiziere den Fehler.“
„Ich weiß, dass
jobs/customer_order_join.pyfehlgeschlagen ist – verwende sparksense mit diesem Einstiegspunkt.“
„Verwende sparksense, um das Log unter
./logs/job.logabzurufen – ich werde es selbst überprüfen.“ (provider="none"– das Tool ruft nur ab; der aufrufende Agent übernimmt die Analyse)
„Mein Job war erfolgreich, dauerte aber 40 Minuten. Verwende sparksense, um die Ausführungsstatistiken auf Optimierungsmöglichkeiten zu prüfen.“
diagnose_spark_failureParameter | Erforderlich | Hinweise |
| Ja |
|
| Wenn | |
| Wenn | |
| Nein | S3-URI zum Quellcode |
| Wenn | Datei oder Ordner |
| Nein | Lokaler Quellcode-Ordner |
| Nein | Bestimmter Dateiname/relativer Pfad zur direkten Verwendung, wobei die automatische Extraktion übersprungen wird – am besten, wenn du bereits weißt, welcher Job fehlgeschlagen ist |
| Nein (Standard |
|
| Nein | Für anthropic/openai; andernfalls liest es |
optimize_spark_performanceGleiche Parameter wie oben, zusätzlich:
Parameter | Erforderlich | Hinweise |
| Nein | Executor-Speicher, Cores, Shuffle-Partitionen usw. |
1. job_entry_point given?
→ use ONLY that file. No auto-extraction.
2. Else, parse the error log for:
→ Python: File "<path>", line <N>
→ Scala/Java: at <package>.<Class>.<method>(<Filename>:<N>)
(handles mixed PySpark traces — Python frames bottoming into JVM
frames — by scanning for both patterns in the same log)
→ filters out framework/library internals (site-packages, pyspark,
org.apache.spark, scala.*, java.*, etc.)
→ fetches up to 10 matched files
3. Else, fallback: broad scan of the project folder, capped at 10 filesVariable | Standard | Zweck |
|
| Region für EMR/S3/Bedrock-Aufrufe |
|
| Zu verwendendes Bedrock-Modell |
| — | Wird verwendet, wenn |
| — | Wird verwendet, wenn |
git clone https://github.com/YOUR_GITHUB_USERNAME/spark-sense-ai.git
cd spark-sense-ai
pip install -e ".[all]"
# Local source + Anthropic provider, includes Python and Scala samples
export ANTHROPIC_API_KEY="sk-ant-..."
python tests/test_local_anthropic.py
# EMR source + Bedrock provider (needs a real EMR cluster/step)
aws configure
python tests/test_emr_bedrock.py --cluster-id j-XXXXXXX --step-id s-XXXXXXXBeide Skripte führen zunächst einen kostenlosen Plausibilitätstest ohne API-Aufrufe durch (provider="none"), bevor kostenpflichtige LLM-Aufrufe getätigt werden.
Automatische Auslösung über Lambda/EventBridge beim Abschluss von EMR-/Glue-Jobs
Databricks als dritter source_type
Integration der strukturierten Spark-History-Server-API
Skew-Erkennung mit Statistiken auf Partitionsebene
MIT – siehe LICENSE.
Diagnose why an Apache Spark job failed.
Args: source_type: Where to fetch the log/code from - "emr" or "local". emr_cluster_id: EMR cluster ID (required if source_type="emr"). emr_step_id: EMR step ID (required if source_type="emr"). s3_project_location: Optional S3 URI to the job's source code (used only if source_type="emr"). local_log_path: Path to a local log file or folder (required if source_type="local"). local_project_path: Optional local folder containing the job's source code (used only if source_type="local"). job_entry_point: Optional filename/relative path of the specific job file that ran (e.g. "jobs/customer_order_join.py"). If given, ONLY this file is used as code context - skips auto-extraction entirely. Best used when the caller already knows which job failed. If omitted, the tool automatically parses the error log's stack trace (Python and/or Scala/Java patterns - handles mixed PySpark traces) to find the relevant file(s) in the project, up to 10 files, filtering out framework/library internals. This keeps large, multi-job projects from having their entire codebase sent to the model - only the code actually implicated by the failure is included. provider: Which LLM does the reasoning - "bedrock", "anthropic", "openai", or "none" (default). "none" returns the fetched log/code as-is, for the CALLING AGENT to diagnose itself - no LLM call is made by this tool in that case. api_key: API key for "anthropic" or "openai" providers. If omitted, reads from the ANTHROPIC_API_KEY / OPENAI_API_KEY environment variable. Not used for "bedrock" (uses locally configured AWS credentials) or "none".
Returns: A structured diagnosis (root cause, evidence, fix, confidence) if a provider is set, or the raw fetched log/code for the calling agent to analyze if provider="none".
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | ||
| provider | No | none | |
| emr_step_id | No | ||
| source_type | Yes | ||
| emr_cluster_id | No | ||
| local_log_path | No | ||
| job_entry_point | No | ||
| local_project_path | No | ||
| s3_project_location | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false) are consistent with the description, which implies fetch-and-analyze operations with no side effects. The description adds transparency about the optional LLM call and the 'none' provider returning raw data, which is valuable context beyond the annotations. No contradictions found.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured, with clear sections for parameters and returns. Each piece of information serves a purpose (e.g., explaining conditional requirements, provider options, and the detection algorithm). While slightly verbose, the complexity of the tool justifies the length. It is not padded with irrelevant details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (multiple providers, conditional parameters, return type variations), the description is exceptionally complete. It covers return values for both provider and 'none' cases, explains the code extraction logic, and provides examples. No missing information that an agent would need to call this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite the schema having zero description coverage, the tool description explains every parameter in detail, including conditional requirements (e.g., 'required if source_type="emr"'), defaults, environment variable fallbacks, and the behavior of job_entry_point. It even includes an example and explains the auto-extraction logic. This fully compensates for the schema's lack of per-parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and object: 'Diagnose why an Apache Spark job failed.' This clearly distinguishes the tool from the sibling 'optimize_spark_performance' by focusing on failure diagnosis rather than performance tuning. The purpose is unambiguous and immediately actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides thorough guidance on when to use the tool and how to configure it for different scenarios (source_type, provider, conditional parameters). It explains fallback behavior (env vars, 'none' provider) and the logic for auto-extraction. However, it does not explicitly contrast with the sibling tool, relying instead on the name difference. This is a minor gap, keeping it from a perfect score.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze a successful Spark job's execution log/stats and suggest performance optimizations.
Args: source_type: Where to fetch the log/code from - "emr" or "local". emr_cluster_id: EMR cluster ID (required if source_type="emr"). emr_step_id: EMR step ID (required if source_type="emr"). s3_project_location: Optional S3 URI to the job's source code. local_log_path: Path to a local log file/folder (required if source_type="local"). local_project_path: Optional local folder with source code. job_entry_point: Optional filename/relative path of the specific job file to focus on. If given, ONLY this file is used - skips auto-extraction. If omitted, the tool parses the execution log for stack-trace-like file references (Python and/or Scala/Java) to narrow down which of the project's files are relevant, up to 10 files. Useful for large, multi-job repositories. current_spark_config: Optional text describing the current Spark configuration (executor memory, cores, shuffle partitions, etc.), to ground recommendations in what's configurable. provider: "bedrock", "anthropic", "openai", or "none" (default - returns fetched content for the calling agent to analyze). api_key: API key for "anthropic"/"openai". Reads from env var if omitted. Unused for "bedrock" or "none".
Returns: A structured analysis (observations, recommendations, estimated impact) if a provider is set, or the raw fetched content for the calling agent to analyze if provider="none".
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | ||
| provider | No | none | |
| emr_step_id | No | ||
| source_type | Yes | ||
| emr_cluster_id | No | ||
| local_log_path | No | ||
| job_entry_point | No | ||
| local_project_path | No | ||
| s3_project_location | No | ||
| current_spark_config | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, and non-destructive. The description adds clarity by explaining that with provider='none' it returns raw fetched content, and otherwise returns a structured analysis. There is no contradiction with annotations, and the description provides useful context on what the tool does without describing side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized with a main purpose statement, an Args section listing each parameter with explanations, and a Returns section. It is thorough but not redundant, providing all necessary information without unnecessary verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description explains the full behavior of the tool: what it analyzes, how it uses the parameters, and what it returns under different provider settings. Although an explicit output schema is not shown, the description clearly describes the two possible return types (structured analysis or raw content), making it complete for the agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage for its 10 parameters, but the tool description compensates fully by explaining each parameter: source_type, emr_cluster_id, emr_step_id, s3_project_location, local_log_path, local_project_path, job_entry_point, current_spark_config, provider, and api_key. It clarifies requirements (e.g., 'required if source_type=...') and optionality, making the semantics complete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyze successful Spark job logs/stats and suggest performance optimizations. It names the specific action (analyze, suggest) and resource (execution log/stats), and differentiates from the sibling tool by emphasizing 'successful' jobs, implying a contrast with failure diagnosis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (for successful jobs to get optimization suggestions) by contrasting with 'successful' and the sibling 'diagnose_spark_failure'. It also explains provider options (including 'none' for raw content) and argument dependencies. However, it does not explicitly state 'use this instead of diagnose_spark_failure when the job succeeded,' leaving the distinction implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.3diagnose_spark_failureoptimize_spark_performanceScored across 2 tools
The two tools have clearly distinct purposes: one diagnoses failures and the other optimizes performance. Despite sharing similar arguments, their actions and return types are unambiguously different.
Both tool names follow the consistent verb_noun pattern (diagnose_spark_failure, optimize_spark_performance), which is clear and predictable.
With only two tools, the server is slightly below the typical well-scoped range, but it is focused on a narrow domain (Spark job analysis) where these two core operations are appropriate.
The server covers the primary needs of diagnosing failures and optimizing performance. However, there may be gaps such as retrieving job metrics or listing available jobs, though these are not strictly required for the stated purpose.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for AI dialogue using various LLM models via AceDataCloud
