MCP（Model Context Protocol）サーバー — AIエージェント（Claude Desktop、Claude Code、Devin、またはMCP互換クライアント）に、Apache Sparkジョブを扱うための2つの機能を提供します:
🔴 diagnose_spark_failure — Sparkジョブが失敗しました。実際のエラーログと失敗した特定のコードに基づいて、根本原因と具体的な修正方法を取得します。
🟢 optimize_spark_performance — Sparkジョブは成功したが、遅い、またはコストがかかる場合。的を絞った、エビデンスに基づくチューニングの推奨事項を取得します。
Apache Sparkの実務経験12年以上を持つデータエンジニアによって構築され、「このエラーは実際にはどのファイルに関するもので、なぜなのか」というデバッグの勘所をAI支援ワークフローに取り込みます。
Sparkの障害は通常、ログだけでも診断可能です。しかし、200行のスタックトレースを読み、大規模で複数のジョブを含むコードベース内の正しいファイルに対応付け、十数ある可能性のある原因のうち実際にどれなのかを判断するには、本物のSpark経験が必要です。このツールはその最初のパスを自動化します。関連するコード（リポジトリ全体ではなく）を特定し、ログと一緒にLLMに渡し、検証して対処できる構造化された診断結果を返します。
Related MCP server: Spark MCP Optimizer
設計上の選択 | 重要な理由 |
ソース非依存 — EMR（クラスター + ステップID）またはローカルフォルダ | ジョブがAWS上でもオンプレミス/ローカルでも動作します |
プロバイダー非依存 — Bedrock、Anthropic、OpenAI、またはなし | ベンダーロックインなし。 |
スマートなファイル選択 | 大規模プロジェクトでは多数のジョブが実行されます。このツールはエラーログのスタックトレース（Python および Scala/Java、混合PySparkトレースを含む）を解析し、失敗に関係する特定のファイルのみを最大10ファイルまで取り込みます。コードベース全体をプロンプトにダンプすることはありません |
認証情報を一切バンドルしない | すべてのユーザーが自身のAWSおよび/またはLLM認証情報を持ち込みます。ここではユーザー間で課金やアクセスを共有することは一切ありません |
このサンプルScalaエラーログ とその対応するプロジェクトファイル を指定すると、diagnose_spark_failure（プロバイダー設定済み）は次を返します:
ROOT CAUSE:
CustomerHelper.validate() calls .trim() on the "email" field without
checking for null first. Records with a missing email cause a
NullPointerException, which aborts the job after 4 failed task retries.
EVIDENCE:
- Caused by: java.lang.NullPointerException: Cannot invoke "String.trim()"
because "email" is null
- at com.company.jobs.CustomerHelper$.validate(CustomerHelper.scala:22)
- Source shows: email.trim().nonEmpty with no null check beforehand
SUGGESTED FIX:
def validate(row: Row): Boolean = {
val email = Option(row.getAs[String]("email"))
email.exists(_.trim.nonEmpty)
}
CONFIDENCE: Highこのツールが、トレースの Caused by が実際に指すファイル CustomerHelper.scala を自動的に取り込んでいることに注目してください。プロジェクト全体や、最上位のエントリファイル CustomerOrderJoin.scala ではなく。スタックトレースパーサーが最も深い関連フレームを解決したためです。
pip install spark-sense-ai実際に使用する分のextrasだけをインストールしてください:
pip install spark-sense-ai[aws] # for EMR source or Bedrock provider
pip install spark-sense-ai[anthropic] # for provider="anthropic"
pip install spark-sense-ai[openai] # for provider="openai"
pip install spark-sense-ai[all] # everythingprovider="none" と source_type="local" の組み合わせは、extrasを一切必要としません。ベースの mcp 依存関係のみで十分です。
# | ログ/コードソース | LLMプロバイダー | 必要なextras | AWS認証情報が必要？ |
1 | EMRクラスター + ステップ | Bedrock |
| はい — 取得および診断のため |
2 | EMRクラスター + ステップ | Anthropic / OpenAI |
| はい — 取得のみ |
3 | ローカルフォルダ | なし（エージェントが推論、例：Devin内） | なし | いいえ |
4 | ローカルフォルダ | Anthropic / OpenAI |
| いいえ |
AWS認証情報は、必要な場合、既存の設定（aws configure、アタッチされたIAMロール、標準のAWS_*環境変数など）から自動的に取得されます。ツールのパラメータとして渡されることは決してありません。
claude_desktop_config.json を編集:
{
"mcpServers": {
"sparksense": {
"command": "sparksense-mcp",
"env": {
"SPARKSENSE_AWS_REGION": "ap-south-1"
}
}
}
}claude mcp add sparksense -- sparksense-mcpお使いのDevinエージェントモード（CascadeとDevin Localでは構成ファイルの場所が少し異なります）の現在の構成方法については、DevinのMCPドキュメント を参照してください。上記と同じ方法で sparksense-mcp コマンドを指定します。
「Sparkジョブが失敗しました — EMRクラスター j-ABC123、ステップ s-XYZ789。sparksenseを使ってBedrockで診断してください。」
「ローカルジョブのログは
./logs/error.log、コードは./srcにあります — 失敗を診断してください。」
「失敗したのは
jobs/customer_order_join.pyだとわかっています — それをエントリポイントとしてsparksenseを使ってください。」
「sparksenseを使って
./logs/job.logのログを取得してください — 自分で確認します。」 （provider="none"— ツールは取得するだけで、推論は呼び出し元エージェントが行います）
「ジョブは成功しましたが、40分かかりました。最適化の機会がないか、sparksenseで実行統計を確認してください。」
diagnose_spark_failureパラメータ | 必須 | 備考 |
| はい |
|
|
| |
|
| |
| いいえ | ソースコードへのS3 URI |
|
| ファイルまたはフォルダ |
| いいえ | ローカルのソースコードフォルダ |
| いいえ | 自動抽出をスキップして直接使用する特定のファイル名/相対パス — 失敗したジョブがすでにわかっている場合に最適 |
| いいえ（デフォルト |
|
| いいえ | anthropic/openai用。それ以外の場合は |
optimize_spark_performance上記と同じパラメータに加えて:
パラメータ | 必須 | 備考 |
| いいえ | エグゼキューターのメモリ、コア数、シャッフルパーティション数など |
1. job_entry_point given?
→ use ONLY that file. No auto-extraction.
2. Else, parse the error log for:
→ Python: File "<path>", line <N>
→ Scala/Java: at <package>.<Class>.<method>(<Filename>:<N>)
(handles mixed PySpark traces — Python frames bottoming into JVM
frames — by scanning for both patterns in the same log)
→ filters out framework/library internals (site-packages, pyspark,
org.apache.spark, scala.*, java.*, etc.)
→ fetches up to 10 matched files
3. Else, fallback: broad scan of the project folder, capped at 10 files変数 | デフォルト | 目的 |
|
| EMR/S3/Bedrock呼び出しのリージョン |
|
| 使用するBedrockモデル |
| — |
|
| — |
|
git clone https://github.com/YOUR_GITHUB_USERNAME/spark-sense-ai.git
cd spark-sense-ai
pip install -e ".[all]"
# Local source + Anthropic provider, includes Python and Scala samples
export ANTHROPIC_API_KEY="sk-ant-..."
python tests/test_local_anthropic.py
# EMR source + Bedrock provider (needs a real EMR cluster/step)
aws configure
python tests/test_emr_bedrock.py --cluster-id j-XXXXXXX --step-id s-XXXXXXXどちらのスクリプトも、有料のLLM呼び出しを行う前に、無料のAPI呼び出しなしの健全性チェック（provider="none"）を最初に実行します。
EMR/Glueジョブ完了時のLambda/EventBridgeによる自動トリガー
3つ目の source_type としてのDatabricks
構造化されたSpark History Server APIの統合
パーティションレベルの統計によるスキュー検出
MIT — LICENSE を参照してください。
Diagnose why an Apache Spark job failed.
Args: source_type: Where to fetch the log/code from - "emr" or "local". emr_cluster_id: EMR cluster ID (required if source_type="emr"). emr_step_id: EMR step ID (required if source_type="emr"). s3_project_location: Optional S3 URI to the job's source code (used only if source_type="emr"). local_log_path: Path to a local log file or folder (required if source_type="local"). local_project_path: Optional local folder containing the job's source code (used only if source_type="local"). job_entry_point: Optional filename/relative path of the specific job file that ran (e.g. "jobs/customer_order_join.py"). If given, ONLY this file is used as code context - skips auto-extraction entirely. Best used when the caller already knows which job failed. If omitted, the tool automatically parses the error log's stack trace (Python and/or Scala/Java patterns - handles mixed PySpark traces) to find the relevant file(s) in the project, up to 10 files, filtering out framework/library internals. This keeps large, multi-job projects from having their entire codebase sent to the model - only the code actually implicated by the failure is included. provider: Which LLM does the reasoning - "bedrock", "anthropic", "openai", or "none" (default). "none" returns the fetched log/code as-is, for the CALLING AGENT to diagnose itself - no LLM call is made by this tool in that case. api_key: API key for "anthropic" or "openai" providers. If omitted, reads from the ANTHROPIC_API_KEY / OPENAI_API_KEY environment variable. Not used for "bedrock" (uses locally configured AWS credentials) or "none".
Returns: A structured diagnosis (root cause, evidence, fix, confidence) if a provider is set, or the raw fetched log/code for the calling agent to analyze if provider="none".
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | ||
| provider | No | none | |
| emr_step_id | No | ||
| source_type | Yes | ||
| emr_cluster_id | No | ||
| local_log_path | No | ||
| job_entry_point | No | ||
| local_project_path | No | ||
| s3_project_location | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false) are consistent with the description, which implies fetch-and-analyze operations with no side effects. The description adds transparency about the optional LLM call and the 'none' provider returning raw data, which is valuable context beyond the annotations. No contradictions found.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured, with clear sections for parameters and returns. Each piece of information serves a purpose (e.g., explaining conditional requirements, provider options, and the detection algorithm). While slightly verbose, the complexity of the tool justifies the length. It is not padded with irrelevant details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (multiple providers, conditional parameters, return type variations), the description is exceptionally complete. It covers return values for both provider and 'none' cases, explains the code extraction logic, and provides examples. No missing information that an agent would need to call this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite the schema having zero description coverage, the tool description explains every parameter in detail, including conditional requirements (e.g., 'required if source_type="emr"'), defaults, environment variable fallbacks, and the behavior of job_entry_point. It even includes an example and explains the auto-extraction logic. This fully compensates for the schema's lack of per-parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and object: 'Diagnose why an Apache Spark job failed.' This clearly distinguishes the tool from the sibling 'optimize_spark_performance' by focusing on failure diagnosis rather than performance tuning. The purpose is unambiguous and immediately actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides thorough guidance on when to use the tool and how to configure it for different scenarios (source_type, provider, conditional parameters). It explains fallback behavior (env vars, 'none' provider) and the logic for auto-extraction. However, it does not explicitly contrast with the sibling tool, relying instead on the name difference. This is a minor gap, keeping it from a perfect score.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze a successful Spark job's execution log/stats and suggest performance optimizations.
Args: source_type: Where to fetch the log/code from - "emr" or "local". emr_cluster_id: EMR cluster ID (required if source_type="emr"). emr_step_id: EMR step ID (required if source_type="emr"). s3_project_location: Optional S3 URI to the job's source code. local_log_path: Path to a local log file/folder (required if source_type="local"). local_project_path: Optional local folder with source code. job_entry_point: Optional filename/relative path of the specific job file to focus on. If given, ONLY this file is used - skips auto-extraction. If omitted, the tool parses the execution log for stack-trace-like file references (Python and/or Scala/Java) to narrow down which of the project's files are relevant, up to 10 files. Useful for large, multi-job repositories. current_spark_config: Optional text describing the current Spark configuration (executor memory, cores, shuffle partitions, etc.), to ground recommendations in what's configurable. provider: "bedrock", "anthropic", "openai", or "none" (default - returns fetched content for the calling agent to analyze). api_key: API key for "anthropic"/"openai". Reads from env var if omitted. Unused for "bedrock" or "none".
Returns: A structured analysis (observations, recommendations, estimated impact) if a provider is set, or the raw fetched content for the calling agent to analyze if provider="none".
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | ||
| provider | No | none | |
| emr_step_id | No | ||
| source_type | Yes | ||
| emr_cluster_id | No | ||
| local_log_path | No | ||
| job_entry_point | No | ||
| local_project_path | No | ||
| s3_project_location | No | ||
| current_spark_config | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, and non-destructive. The description adds clarity by explaining that with provider='none' it returns raw fetched content, and otherwise returns a structured analysis. There is no contradiction with annotations, and the description provides useful context on what the tool does without describing side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized with a main purpose statement, an Args section listing each parameter with explanations, and a Returns section. It is thorough but not redundant, providing all necessary information without unnecessary verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description explains the full behavior of the tool: what it analyzes, how it uses the parameters, and what it returns under different provider settings. Although an explicit output schema is not shown, the description clearly describes the two possible return types (structured analysis or raw content), making it complete for the agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage for its 10 parameters, but the tool description compensates fully by explaining each parameter: source_type, emr_cluster_id, emr_step_id, s3_project_location, local_log_path, local_project_path, job_entry_point, current_spark_config, provider, and api_key. It clarifies requirements (e.g., 'required if source_type=...') and optionality, making the semantics complete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyze successful Spark job logs/stats and suggest performance optimizations. It names the specific action (analyze, suggest) and resource (execution log/stats), and differentiates from the sibling tool by emphasizing 'successful' jobs, implying a contrast with failure diagnosis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (for successful jobs to get optimization suggestions) by contrasting with 'successful' and the sibling 'diagnose_spark_failure'. It also explains provider options (including 'none' for raw content) and argument dependencies. However, it does not explicitly state 'use this instead of diagnose_spark_failure when the job succeeded,' leaving the distinction implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.3diagnose_spark_failureoptimize_spark_performanceScored across 2 tools
The two tools have clearly distinct purposes: one diagnoses failures and the other optimizes performance. Despite sharing similar arguments, their actions and return types are unambiguously different.
Both tool names follow the consistent verb_noun pattern (diagnose_spark_failure, optimize_spark_performance), which is clear and predictable.
With only two tools, the server is slightly below the typical well-scoped range, but it is focused on a narrow domain (Spark job analysis) where these two core operations are appropriate.
The server covers the primary needs of diagnosing failures and optimizing performance. However, there may be gaps such as retrieving job metrics or listing available jobs, though these are not strictly required for the stated purpose.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for AI dialogue using various LLM models via AceDataCloud
