Spark History Server MCP (TypeScript)

Give an LLM read access to your Spark History Server so it can do the tedious part of Spark work: finding why a job failed, and finding where a slow job spends its time.

It is a TypeScript port of kubeflow/mcp-apache-spark-history-server, verified response-for-response against the Python original — see PARITY.md. On top of the port it ships two agent skills that turn the raw tools into an expert workflow for root-cause analysis and performance tuning.

                    ┌──────────────────┐
  data engineer ──▶ │  LLM client      │   Claude Code / Claude Desktop / any MCP client
                    │  + skills        │   ← skills/ supply the method
                    └────────┬─────────┘
                             │ MCP (stdio or streamable-http)
                    ┌────────▼─────────┐
                    │  this server     │   17 tools, 2 prompts
                    └────────┬─────────┘
                             │ HTTP  GET /api/v1/...
                    ┌────────▼─────────┐
                    │ Spark History    │   your existing one, or the bundled demo
                    │ Server           │
                    └────────┬─────────┘
                             │ reads
                    ┌────────▼─────────┐
                    │ event logs       │   s3://…, hdfs://…, file://…
                    └──────────────────┘

The server only ever issues GET requests to the History Server's REST API. It cannot modify anything.

Contents

1. Quick start

2. Pointing it at your Spark History Server

3. Connecting your LLM client

4. Installing the skills

5. The tools

6. How it works

7. Deployment

8. Troubleshooting

9. Development

Related MCP server: Spark EventLog MCP Server

1. Quick start

Option A — Docker (nothing to install but Docker)

Starts a Spark History Server loaded with sample event logs and this MCP:

git clone https://github.com/ukonduru91/spark-history-mcp.git
cd spark-history-mcp
docker compose up --build

The bundled logs include a healthy pipeline and a deliberately failed job, so the tools have something real to show before you point them at your own cluster.

To run only the History Server:

./start_local_spark_history.sh          # macOS / Linux / Git Bash
.\start_local_spark_history.ps1         # Windows PowerShell

Option B — from source

Requires Node.js 20+ (22 recommended).

git clone https://github.com/ukonduru91/spark-history-mcp.git
cd spark-history-mcp
npm install
npm run build
npm start

Verify it works

node scripts/mcp-cli.mjs list-tools
node scripts/mcp-cli.mjs call list_applications '{"limit": 5}'

If applications come back, you are connected.

2. Pointing it at your Spark History Server

This is the one thing you must configure. Three ways, highest precedence first — environment variables win over the .env file, which wins over YAML.

a. Environment variables (best for containers and CI)

Nesting uses a double underscore. LOCAL below is just a name you choose for the server:

export SHS_SERVERS__LOCAL__URL=http://spark-history.internal:18080
export SHS_SERVERS__LOCAL__DEFAULT=true

b. A YAML config file

The server looks for one in this order:

1. the path given to --config, or $SHS_MCP_CONFIG

2. ./config.yaml in the working directory

3. ~/.config/spark-mcp/config.yaml

servers:
  prod:
    url: "https://spark-history.company.com:18080"
    default: true          # used when a tool call omits `server`
    verify_ssl: true
    ssl_ca_cert: "/etc/ssl/custom-ca/ca-bundle.pem"   # private CA
    timeout: 30            # seconds
    auth:
      username: admin
      password: ${SPARK_PASSWORD}   # see the note below
      # token: <bearer token>       # or a bearer token instead

  staging:
    url: "https://spark-history-staging.company.com:18080"

On secrets: values in YAML are literal — ${SPARK_PASSWORD} is not expanded. Keep credentials in environment variables (SHS_SERVERS__PROD__AUTH__PASSWORD), which override the file. This matches the upstream project's behaviour.

c. A .env file

Same variable names as (a), read from .env in the working directory.

Multiple servers

Configure as many as you like. Tools take an optional server argument; when it is omitted the server discovers which configured History Server has that application and uses it (cached for 5 minutes). An engineer can therefore ask about an application id without knowing which cluster ran it.

Every setting

Single-underscore variables (SHS_MCP_PORT) still work but log a deprecation warning, exactly as upstream.

Reaching a History Server you cannot route to

An SSH tunnel plus use_proxy: true covers the common locked-down-cluster case:

ssh -D 8157 -N user@bastion    # SOCKS5 proxy on :8157

3. Connecting your LLM client

stdio (Claude Code, Claude Desktop, most clients)

{
  "mcpServers": {
    "spark-history": {
      "command": "node",
      "args": ["/absolute/path/to/spark-history-mcp/dist/index.js"],
      "env": {
        "SHS_MCP__TRANSPORT": "stdio",
        "SHS_SERVERS__PROD__URL": "https://spark-history.company.com:18080",
        "SHS_SERVERS__PROD__DEFAULT": "true"
      }
    }
  }
}

Claude Code users can do the same in one line:

claude mcp add spark-history \
  --env SHS_MCP__TRANSPORT=stdio \
  --env SHS_SERVERS__PROD__URL=https://spark-history.company.com:18080 \
  --env SHS_SERVERS__PROD__DEFAULT=true \
  -- node /absolute/path/to/spark-history-mcp/dist/index.js

streamable-http (one shared server for a team)

Run it once, point everyone at it:

SHS_MCP__TRANSPORT=streamable-http SHS_MCP__ADDRESS=0.0.0.0 npm start

Clients connect to http://<host>:18888/mcp. The server is read-only, but it is also unauthenticated — put it behind your normal internal ingress, and enable DNS rebinding protection if it is reachable from a browser:

mcp:
  transport_security:
    enable_dns_rebinding_protection: true
    allowed_hosts: ["spark-mcp.internal:*"]
    allowed_origins: ["https://spark-mcp.internal"]

4. Installing the skills

The tools give the model access to the data. The skills give it the method — the order to gather evidence in, the thresholds that separate a finding from noise, and the rule that it must not name a cause it has not seen in the data.

# per project
mkdir -p .claude/skills
cp -r skills/spark-rca skills/spark-optimization .claude/skills/

# or for every project
mkdir -p ~/.claude/skills
cp -r skills/spark-rca skills/spark-optimization ~/.claude/skills/

They trigger on their own from a normal question — nobody has to remember a command:

"the 2am load failed again, app_1724… — can you look?"

See skills/README.md for what is inside each one and how to extend them with your team's own knowledge.

5. The tools

All 17 live in src/tools/tools.ts; their JSON schemas are in src/schemas/generated.ts. Run node scripts/mcp-cli.mjs list-tools to see them with their arguments.

Finding things

Going deep

Diagnosing

Comparing two runs

Prompts

investigate_failure(app_id, server?) and compare_applications(app_a, app_b, server?, context?) — interactive walkthroughs from the upstream project, for when the engineer wants to drive instead of handing the analysis over.

6. How it works

A tool call becomes one or more GETs against /api/v1/..., and the JSON comes back shaped exactly as the Python original shaped it.

src/
  index.ts                 CLI entry, transport selection (stdio | streamable-http)
  config/config.ts         YAML + .env + SHS_* resolution and precedence
  core/
    app.ts                 MCP request handlers; maps results to content blocks
    validation.ts          pydantic-compatible argument validation and messages
    json.ts                Python-compatible JSON rendering
    pyfloat.ts             int/float fidelity across the JSON round-trip
    pyrepr.ts              Python repr() for validation messages
    errors.ts              error text shaping
  api/
    httpClient.ts          HTTP transport, ApiException taxonomy, auth, TLS, SOCKS
    sparkClient.ts         Spark REST facade: pagination, attempts, status filters
  models/
    generated.ts           model shapes, generated from the upstream OpenAPI models
    deserialize.ts         from_dict / model_dump equivalents
    mcpTypes.ts            curated LLM-facing output models
  tools/tools.ts           the 17 tools
  prompts/prompts.ts       the 2 prompts
  schemas/generated.ts     tool + prompt catalogue (names, descriptions, schemas)

Three details worth knowing if you plan to modify it:

• models/generated.ts and schemas/generated.ts are generated, by tools/gen_models.py and tools/gen_schemas.py, from the upstream Python project. Regenerate rather than hand-edit — that is what keeps the catalogue and the response shapes identical to the original.

• The low-level Server API is used, not McpServer, because the result shape has to match FastMCP's: one text block per list element, and structuredContent only for the tools whose Python signature declared a concrete return type.

• Application discovery lets tools omit server. ApplicationDiscovery probes each configured server for the application id and caches the answer for 5 minutes.

7. Deployment

Docker

docker build -t spark-history-mcp .
docker run -p 18888:18888 \
  -e SHS_SERVERS__PROD__URL=https://spark-history.company.com:18080 \
  -e SHS_SERVERS__PROD__DEFAULT=true \
  -e SHS_MCP__ADDRESS=0.0.0.0 \
  spark-history-mcp

Kubernetes

Run it as a normal Deployment with the URL in the env and credentials from a Secret:

env:
  - name: SHS_MCP__TRANSPORT
    value: streamable-http
  - name: SHS_MCP__ADDRESS
    value: "0.0.0.0"
  - name: SHS_SERVERS__PROD__URL
    value: http://spark-history-server.spark.svc.cluster.local:18080
  - name: SHS_SERVERS__PROD__DEFAULT
    value: "true"
  - name: SHS_SERVERS__PROD__AUTH__TOKEN
    valueFrom:
      secretKeyRef: { name: spark-history-auth, key: token }

The process is stateless apart from the 5-minute discovery cache, so it scales horizontally without coordination.

8. Troubleshooting

Set SHS_MCP__DEBUG=true for verbose logs.

9. Development

npm install
npm run build        # compile to dist/
npm run dev          # run from source, no build step
npm test             # unit tests
npm run typecheck    # tsc --noEmit

Cross-implementation parity testing lives in parity/ — it runs the same MCP calls against this server and the Python original and diffs every response. PARITY.md records the results and the exact differences that remain.

Not ported from upstream

License

Apache-2.0, as with the upstream project.