AI 에이전트(Claude Desktop, Claude Code, Devin 또는 MCP 호환 클라이언트)에게 Apache Spark 작업을 처리하는 두 가지 기능을 제공하는 MCP(Model Context Protocol) 서버입니다:
🔴 diagnose_spark_failure — Spark 작업이 실패한 경우, 실제 오류 로그와 실패한 특정 코드를 바탕으로 근본 원인과 구체적인 해결 방법을 제공합니다.
🟢 optimize_spark_performance — Spark 작업이 성공했지만 느리거나 비용이 많이 드는 경우, 근거 기반의 맞춤형 튜닝 권장 사항을 제공합니다.
12년 이상의 실무 Apache Spark 경험을 가진 데이터 엔지니어가, 동일한 디버깅 직관 — "이 오류가 실제로 어떤 파일에 관한 것이며, 왜 그런가" — 을 AI 지원 워크플로우에 적용하기 위해 만들었습니다.
Spark 실패는 대개 로그만으로 진단할 수 있습니다. 그러나 200줄짜리 스택 트레이스를 읽고, 대규모 다중 작업 코드베이스에서 해당 오류가 실제로 가리키는 올바른 파일을 찾아내며, 수십 가지 가능한 원인 중 실제 원인이 무엇인지 파악하는 데는 진짜 Spark 경험이 필요합니다. 이 도구는 그 첫 번째 단계를 자동화합니다. 관련된 코드(전체 저장소가 아닌)를 찾아서 로그와 함께 LLM에 전달하고, 검증하고 조치를 취할 수 있는 구조화된 진단 결과를 돌려받습니다.
Related MCP server: Spark MCP Optimizer
설계 선택 | 중요한 이유 |
소스 독립적 — EMR(클러스터 + 단계 ID) 또는 로컬 폴더 | 작업이 AWS나 온프레미스/로컬에서 실행되는지에 관계없이 동작합니다. |
공급자 독립적 — Bedrock, Anthropic, OpenAI, 또는 없음 | 특정 벤더에 종속되지 않습니다. |
스마트 파일 선택 | 대규모 프로젝트는 여러 작업을 실행합니다. 이 도구는 오류 로그의 스택 트레이스(Python 및 Scala/Java, 혼합 PySpark 트레이스 포함)를 파싱하여 전체 코드베이스를 프롬프트에 던져 넣는 대신, 실패와 관련된 특정 파일만(최대 10개 파일) 가져옵니다. |
번들 자격 증명은 절대 없음 | 모든 사용자가 자신의 AWS 및/또는 LLM 자격 증명을 가져옵니다. 여기에는 사용자 간에 결제 또는 접근 권한을 공유하는 기능이 전혀 없습니다. |
이 샘플 Scala 오류 로그와 해당 프로젝트 파일이 주어지면 diagnose_spark_failure(공급자 구성 시)는 다음을 반환합니다:
ROOT CAUSE:
CustomerHelper.validate() calls .trim() on the "email" field without
checking for null first. Records with a missing email cause a
NullPointerException, which aborts the job after 4 failed task retries.
EVIDENCE:
- Caused by: java.lang.NullPointerException: Cannot invoke "String.trim()"
because "email" is null
- at com.company.jobs.CustomerHelper$.validate(CustomerHelper.scala:22)
- Source shows: email.trim().nonEmpty with no null check beforehand
SUGGESTED FIX:
def validate(row: Row): Boolean = {
val email = Option(row.getAs[String]("email"))
email.exists(_.trim.nonEmpty)
}
CONFIDENCE: High도구가 자동으로 CustomerHelper.scala(트레이스의 Caused by가 실제로 가리키는 파일)를 가져온 것을 확인할 수 있습니다. 전체 프로젝트나 최상위 엔트리 파일인 CustomerOrderJoin.scala조차 가져오지 않았습니다. 스택 트레이스 파서가 가장 깊은 관련 프레임을 해석했기 때문입니다.
pip install spark-sense-ai실제로 사용할 기능에 대해서만 추가 패키지를 설치하세요:
pip install spark-sense-ai[aws] # for EMR source or Bedrock provider
pip install spark-sense-ai[anthropic] # for provider="anthropic"
pip install spark-sense-ai[openai] # for provider="openai"
pip install spark-sense-ai[all] # everythingprovider="none" 및 source_type="local" 조합은 추가 패키지가 전혀 필요 없습니다 — 기본 mcp 의존성만 있으면 됩니다.
# | 로그/코드 소스 | LLM 공급자 | 필요한 추가 패키지 | AWS 자격 증명 필요? |
1 | EMR 클러스터 + 단계 | Bedrock |
| 예 — 로그 가져오기 및 진단에 필요 |
2 | EMR 클러스터 + 단계 | Anthropic / OpenAI |
| 예 — 로그 가져오기에만 필요 |
3 | 로컬 폴더 | 없음 (에이전트가 추론, 예: Devin 내부) | 없음 | 아니요 |
4 | 로컬 폴더 | Anthropic / OpenAI |
| 아니요 |
AWS 자격 증명은 필요할 때 기존 설정(aws configure, 연결된 IAM 역할, 표준 AWS_* 환경 변수)에서 자동으로 가져옵니다 — 도구 파라미터로 절대 전달되지 않습니다.
claude_desktop_config.json을 편집하세요:
{
"mcpServers": {
"sparksense": {
"command": "sparksense-mcp",
"env": {
"SPARKSENSE_AWS_REGION": "ap-south-1"
}
}
}
}claude mcp add sparksense -- sparksense-mcp현재 Devin 에이전트 모드(캐스케이드와 Devin Local은 설정 위치가 약간 다릅니다)의 구성 방법은 Devin의 MCP 문서를 참조하세요. 위와 같은 방식으로 sparksense-mcp 명령을 가리키면 됩니다.
"제 Spark 작업이 실패했습니다 — EMR 클러스터 j-ABC123, 단계 s-XYZ789. sparksense로 Bedrock을 사용해 진단해 주세요."
"제 로컬 작업의 로그는
./logs/error.log, 코드는./src에 있습니다 — 실패를 진단해 주세요."
"실패한 것이
jobs/customer_order_join.py라는 것을 알고 있습니다 — sparksense에서 이를 엔트리 포인트로 사용해 주세요."
"sparksense를 사용해
./logs/job.log의 로그를 가져와 주세요 — 제가 직접 검토하겠습니다." (provider="none"— 도구는 로그만 가져오고, 호출하는 에이전트가 추론을 수행합니다)
"제 작업이 성공했지만 40분이 걸렸습니다. sparksense를 사용해 최적화 기회가 있는지 실행 통계를 확인해 주세요."
diagnose_spark_failure파라미터 | 필수 여부 | 설명 |
| 예 |
|
|
| |
|
| |
| 아니요 | 소스 코드의 S3 URI |
|
| 파일 또는 폴더 |
| 아니요 | 로컬 소스 코드 폴더 |
| 아니요 | 자동 추출을 건너뛰고 직접 사용할 특정 파일 이름/상대 경로 — 이미 어떤 작업이 실패했는지 알고 있을 때 가장 유용합니다 |
| 아니요 (기본값 |
|
| 아니요 | anthropic/openai용; 그 외에는 |
optimize_spark_performance위와 동일한 파라미터에 다음이 추가됩니다:
파라미터 | 필수 여부 | 설명 |
| 아니요 | 실행자 메모리, 코어, 셔플 파티션 등 |
1. job_entry_point given?
→ use ONLY that file. No auto-extraction.
2. Else, parse the error log for:
→ Python: File "<path>", line <N>
→ Scala/Java: at <package>.<Class>.<method>(<Filename>:<N>)
(handles mixed PySpark traces — Python frames bottoming into JVM
frames — by scanning for both patterns in the same log)
→ filters out framework/library internals (site-packages, pyspark,
org.apache.spark, scala.*, java.*, etc.)
→ fetches up to 10 matched files
3. Else, fallback: broad scan of the project folder, capped at 10 files변수 | 기본값 | 용도 |
|
| EMR/S3/Bedrock 호출에 사용할 리전 |
|
| 사용할 Bedrock 모델 |
| — |
|
| — |
|
git clone https://github.com/YOUR_GITHUB_USERNAME/spark-sense-ai.git
cd spark-sense-ai
pip install -e ".[all]"
# Local source + Anthropic provider, includes Python and Scala samples
export ANTHROPIC_API_KEY="sk-ant-..."
python tests/test_local_anthropic.py
# EMR source + Bedrock provider (needs a real EMR cluster/step)
aws configure
python tests/test_emr_bedrock.py --cluster-id j-XXXXXXX --step-id s-XXXXXXX두 스크립트 모두 유료 LLM 호출을 하기 전에 먼저 무료이며 API 호출이 없는 기본 점검(provider="none")을 실행합니다.
EMR/Glue 작업 완료 시 Lambda/EventBridge를 통한 자동 트리거
세 번째 source_type으로 Databricks 지원
구조화된 Spark History Server API 통합
파티션 수준 통계를 활용한 데이터 쏠림 감지
MIT — LICENSE 참조.
Diagnose why an Apache Spark job failed.
Args: source_type: Where to fetch the log/code from - "emr" or "local". emr_cluster_id: EMR cluster ID (required if source_type="emr"). emr_step_id: EMR step ID (required if source_type="emr"). s3_project_location: Optional S3 URI to the job's source code (used only if source_type="emr"). local_log_path: Path to a local log file or folder (required if source_type="local"). local_project_path: Optional local folder containing the job's source code (used only if source_type="local"). job_entry_point: Optional filename/relative path of the specific job file that ran (e.g. "jobs/customer_order_join.py"). If given, ONLY this file is used as code context - skips auto-extraction entirely. Best used when the caller already knows which job failed. If omitted, the tool automatically parses the error log's stack trace (Python and/or Scala/Java patterns - handles mixed PySpark traces) to find the relevant file(s) in the project, up to 10 files, filtering out framework/library internals. This keeps large, multi-job projects from having their entire codebase sent to the model - only the code actually implicated by the failure is included. provider: Which LLM does the reasoning - "bedrock", "anthropic", "openai", or "none" (default). "none" returns the fetched log/code as-is, for the CALLING AGENT to diagnose itself - no LLM call is made by this tool in that case. api_key: API key for "anthropic" or "openai" providers. If omitted, reads from the ANTHROPIC_API_KEY / OPENAI_API_KEY environment variable. Not used for "bedrock" (uses locally configured AWS credentials) or "none".
Returns: A structured diagnosis (root cause, evidence, fix, confidence) if a provider is set, or the raw fetched log/code for the calling agent to analyze if provider="none".
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | ||
| provider | No | none | |
| emr_step_id | No | ||
| source_type | Yes | ||
| emr_cluster_id | No | ||
| local_log_path | No | ||
| job_entry_point | No | ||
| local_project_path | No | ||
| s3_project_location | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false) are consistent with the description, which implies fetch-and-analyze operations with no side effects. The description adds transparency about the optional LLM call and the 'none' provider returning raw data, which is valuable context beyond the annotations. No contradictions found.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured, with clear sections for parameters and returns. Each piece of information serves a purpose (e.g., explaining conditional requirements, provider options, and the detection algorithm). While slightly verbose, the complexity of the tool justifies the length. It is not padded with irrelevant details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (multiple providers, conditional parameters, return type variations), the description is exceptionally complete. It covers return values for both provider and 'none' cases, explains the code extraction logic, and provides examples. No missing information that an agent would need to call this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite the schema having zero description coverage, the tool description explains every parameter in detail, including conditional requirements (e.g., 'required if source_type="emr"'), defaults, environment variable fallbacks, and the behavior of job_entry_point. It even includes an example and explains the auto-extraction logic. This fully compensates for the schema's lack of per-parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and object: 'Diagnose why an Apache Spark job failed.' This clearly distinguishes the tool from the sibling 'optimize_spark_performance' by focusing on failure diagnosis rather than performance tuning. The purpose is unambiguous and immediately actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides thorough guidance on when to use the tool and how to configure it for different scenarios (source_type, provider, conditional parameters). It explains fallback behavior (env vars, 'none' provider) and the logic for auto-extraction. However, it does not explicitly contrast with the sibling tool, relying instead on the name difference. This is a minor gap, keeping it from a perfect score.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze a successful Spark job's execution log/stats and suggest performance optimizations.
Args: source_type: Where to fetch the log/code from - "emr" or "local". emr_cluster_id: EMR cluster ID (required if source_type="emr"). emr_step_id: EMR step ID (required if source_type="emr"). s3_project_location: Optional S3 URI to the job's source code. local_log_path: Path to a local log file/folder (required if source_type="local"). local_project_path: Optional local folder with source code. job_entry_point: Optional filename/relative path of the specific job file to focus on. If given, ONLY this file is used - skips auto-extraction. If omitted, the tool parses the execution log for stack-trace-like file references (Python and/or Scala/Java) to narrow down which of the project's files are relevant, up to 10 files. Useful for large, multi-job repositories. current_spark_config: Optional text describing the current Spark configuration (executor memory, cores, shuffle partitions, etc.), to ground recommendations in what's configurable. provider: "bedrock", "anthropic", "openai", or "none" (default - returns fetched content for the calling agent to analyze). api_key: API key for "anthropic"/"openai". Reads from env var if omitted. Unused for "bedrock" or "none".
Returns: A structured analysis (observations, recommendations, estimated impact) if a provider is set, or the raw fetched content for the calling agent to analyze if provider="none".
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | ||
| provider | No | none | |
| emr_step_id | No | ||
| source_type | Yes | ||
| emr_cluster_id | No | ||
| local_log_path | No | ||
| job_entry_point | No | ||
| local_project_path | No | ||
| s3_project_location | No | ||
| current_spark_config | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, and non-destructive. The description adds clarity by explaining that with provider='none' it returns raw fetched content, and otherwise returns a structured analysis. There is no contradiction with annotations, and the description provides useful context on what the tool does without describing side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized with a main purpose statement, an Args section listing each parameter with explanations, and a Returns section. It is thorough but not redundant, providing all necessary information without unnecessary verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description explains the full behavior of the tool: what it analyzes, how it uses the parameters, and what it returns under different provider settings. Although an explicit output schema is not shown, the description clearly describes the two possible return types (structured analysis or raw content), making it complete for the agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage for its 10 parameters, but the tool description compensates fully by explaining each parameter: source_type, emr_cluster_id, emr_step_id, s3_project_location, local_log_path, local_project_path, job_entry_point, current_spark_config, provider, and api_key. It clarifies requirements (e.g., 'required if source_type=...') and optionality, making the semantics complete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyze successful Spark job logs/stats and suggest performance optimizations. It names the specific action (analyze, suggest) and resource (execution log/stats), and differentiates from the sibling tool by emphasizing 'successful' jobs, implying a contrast with failure diagnosis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (for successful jobs to get optimization suggestions) by contrasting with 'successful' and the sibling 'diagnose_spark_failure'. It also explains provider options (including 'none' for raw content) and argument dependencies. However, it does not explicitly state 'use this instead of diagnose_spark_failure when the job succeeded,' leaving the distinction implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.3diagnose_spark_failureoptimize_spark_performanceScored across 2 tools
The two tools have clearly distinct purposes: one diagnoses failures and the other optimizes performance. Despite sharing similar arguments, their actions and return types are unambiguously different.
Both tool names follow the consistent verb_noun pattern (diagnose_spark_failure, optimize_spark_performance), which is clear and predictable.
With only two tools, the server is slightly below the typical well-scoped range, but it is focused on a narrow domain (Spark job analysis) where these two core operations are appropriate.
The server covers the primary needs of diagnosing failures and optimizing performance. However, there may be gaps such as retrieving job metrics or listing available jobs, though these are not strictly required for the stated purpose.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for AI dialogue using various LLM models via AceDataCloud
