Spark History Server MCP (TypeScript)

Give an LLM read access to your Spark History Server so it can do the tedious part of Spark work: finding why a job failed, and finding where a slow job spends its time.

It is a TypeScript port of kubeflow/mcp-apache-spark-history-server, verified response-for-response against the Python original — see PARITY.md. On top of the port it ships two agent skills that turn the raw tools into an expert workflow for root-cause analysis and performance tuning.

                    ┌──────────────────┐
  data engineer ──▶ │  LLM client      │   Claude Code / Claude Desktop / any MCP client
                    │  + skills        │   ← skills/ supply the method
                    └────────┬─────────┘
                             │ MCP (stdio or streamable-http)
                    ┌────────▼─────────┐
                    │  this server     │   17 tools, 2 prompts
                    └────────┬─────────┘
                             │ HTTP  GET /api/v1/...
                    ┌────────▼─────────┐
                    │ Spark History    │   your existing one, or the bundled demo
                    │ Server           │
                    └────────┬─────────┘
                             │ reads
                    ┌────────▼─────────┐
                    │ event logs       │   s3://…, hdfs://…, file://…
                    └──────────────────┘

The server only ever issues GET requests to the History Server's REST API. It cannot modify anything.

Contents

1. Prerequisites

2. Quick start

3. Pointing it at your Spark History Server

4. Connecting your LLM client

5. Installing the skills

6. The tools

7. How it works

8. Deployment

9. Troubleshooting

10. Development

Related MCP server: Spark History MCP Server

0. Prerequisites

Pick a route below and you only need what that route lists — you don't need both.

You do not need a Java/Spark install yourself in either route — Option A's Docker image bundles a real Spark distribution to serve the sample event logs; Option B assumes you already have a Spark History Server running somewhere (yours, or the one from Option A) and just points at its URL.

1. Quick start

Option A — Docker (nothing to install but Docker)

Starts a Spark History Server loaded with sample event logs and this MCP, in one command — this is the fastest path and needs nothing installed except Git and Docker (see Prerequisites):

git clone https://github.com/ukonduru91/spark-history-mcp.git
cd spark-history-mcp
docker compose up --build -d      # -d = detached; drop it to watch the logs

That single command does three things: builds this MCP server's Docker image (installing its npm dependencies and compiling TypeScript to dist/ inside the image), starts the bundled Spark History Server container, and starts this MCP server container in streamable-http mode, already configured (via docker-compose.yml's env vars) to point at that History Server.

Check both containers came up healthy with docker compose ps. The bundled logs include a healthy pipeline and a deliberately failed job, so the tools have something real to show before you point them at your own cluster.

To run only the History Server (no MCP container):

./start_local_spark_history.sh          # macOS / Linux / Git Bash
.\start_local_spark_history.ps1         # Windows PowerShell

Option B — from source

Requires Node.js 20+ (22 recommended) — see Prerequisites. Use this when you want the MCP server itself running as a local process (e.g. to register it with an LLM client over stdio — see §3) rather than as a Docker container, or when you're developing against it.

git clone https://github.com/ukonduru91/spark-history-mcp.git
cd spark-history-mcp
npm install                 # installs the packages from package.json into node_modules/
npm run build                # compiles src/ → dist/index.js

Before starting it, point it at your Spark History Server — edit config.yaml (see §2 for every option). The file ships pointing at http://localhost:18080, which matches Option A's Docker demo server if you have that running.

Then start it:

npm start                    # runs `node dist/index.js`, transport from config.yaml/env (default streamable-http, port 18888)

or, to have an LLM client (Claude Code, Claude Desktop, ...) launch it itself over stdio instead of running it standalone, skip npm start and register it as in §3.

Verify it works

node scripts/mcp-cli.mjs list-tools
node scripts/mcp-cli.mjs call list_applications '{"limit": 5}'

If applications come back, you are connected.

2. Pointing it at your Spark History Server

This is the one thing you must configure. Three ways, highest precedence first — environment variables win over the .env file, which wins over YAML.

a. Environment variables (best for containers and CI)

Nesting uses a double underscore. LOCAL below is just a name you choose for the server:

export SHS_SERVERS__LOCAL__URL=http://spark-history.internal:18080
export SHS_SERVERS__LOCAL__DEFAULT=true

b. A YAML config file

This is config.yaml at the repo root — it ships pre-filled with a local server pointing at http://localhost:18080 (the Docker demo History Server from Option A), set as the default. Edit the url: under servers: to point at your own Spark History Server, or add a new named server block (e.g. production:) as shown below and set its default: true.

The server looks for a config file in this order:

1. the path given to --config, or $SHS_MCP_CONFIG

2. ./config.yaml in the working directory

3. ~/.config/spark-mcp/config.yaml

servers:
  prod:
    url: "https://spark-history.company.com:18080"
    default: true          # used when a tool call omits `server`
    verify_ssl: true
    ssl_ca_cert: "/etc/ssl/custom-ca/ca-bundle.pem"   # private CA
    timeout: 30            # seconds
    auth:
      username: admin
      password: ${SPARK_PASSWORD}   # see the note below
      # token: <bearer token>       # or a bearer token instead

  staging:
    url: "https://spark-history-staging.company.com:18080"

On secrets: values in YAML are literal — ${SPARK_PASSWORD} is not expanded. Keep credentials in environment variables (SHS_SERVERS__PROD__AUTH__PASSWORD), which override the file. This matches the upstream project's behaviour.

c. A .env file

Same variable names as (a), read from .env in the working directory.

Multiple servers

Configure as many as you like. Tools take an optional server argument; when it is omitted the server discovers which configured History Server has that application and uses it (cached for 5 minutes). An engineer can therefore ask about an application id without knowing which cluster ran it.

Every setting

Single-underscore variables (SHS_MCP_PORT) still work but log a deprecation warning, exactly as upstream.

Reaching a History Server you cannot route to

An SSH tunnel plus use_proxy: true covers the common locked-down-cluster case:

ssh -D 8157 -N user@bastion    # SOCKS5 proxy on :8157

3. Connecting your LLM client

stdio (Claude Code, Claude Desktop, most clients)

{
  "mcpServers": {
    "spark-history": {
      "command": "node",
      "args": ["/absolute/path/to/spark-history-mcp/dist/index.js"],
      "env": {
        "SHS_MCP__TRANSPORT": "stdio",
        "SHS_SERVERS__PROD__URL": "https://spark-history.company.com:18080",
        "SHS_SERVERS__PROD__DEFAULT": "true"
      }
    }
  }
}

Claude Code users can do the same in one line:

claude mcp add spark-history \
  --env SHS_MCP__TRANSPORT=stdio \
  --env SHS_SERVERS__PROD__URL=https://spark-history.company.com:18080 \
  --env SHS_SERVERS__PROD__DEFAULT=true \
  -- node /absolute/path/to/spark-history-mcp/dist/index.js

streamable-http (one shared server for a team)

Run it once, point everyone at it:

SHS_MCP__TRANSPORT=streamable-http SHS_MCP__ADDRESS=0.0.0.0 npm start

Clients connect to http://<host>:18888/mcp. The server is read-only, but it is also unauthenticated — put it behind your normal internal ingress, and enable DNS rebinding protection if it is reachable from a browser:

mcp:
  transport_security:
    enable_dns_rebinding_protection: true
    allowed_hosts: ["spark-mcp.internal:*"]
    allowed_origins: ["https://spark-mcp.internal"]

4. Installing the skills

The tools give the model access to the data. The skills give it the method — the order to gather evidence in, the thresholds that separate a finding from noise, and the rule that it must not name a cause it has not seen in the data.

# per project
mkdir -p .claude/skills
cp -r skills/spark-rca skills/spark-optimization .claude/skills/

# or for every project
mkdir -p ~/.claude/skills
cp -r skills/spark-rca skills/spark-optimization ~/.claude/skills/

They trigger on their own from a normal question — nobody has to remember a command:

"the 2am load failed again, app_1724… — can you look?"

See skills/README.md for what is inside each one and how to extend them with your team's own knowledge.

5. The tools

All 17 live in src/tools/tools.ts; their JSON schemas are in src/schemas/generated.ts. Run node scripts/mcp-cli.mjs list-tools to see them with their arguments.

Finding things

Going deep

Diagnosing

Comparing two runs

Prompts

investigate_failure(app_id, server?) and compare_applications(app_a, app_b, server?, context?) — interactive walkthroughs from the upstream project, for when the engineer wants to drive instead of handing the analysis over.

6. How it works

A tool call becomes one or more GETs against /api/v1/..., and the JSON comes back shaped exactly as the Python original shaped it.

src/
  index.ts                 CLI entry, transport selection (stdio | streamable-http)
  config/config.ts         YAML + .env + SHS_* resolution and precedence
  core/
    app.ts                 MCP request handlers; maps results to content blocks
    validation.ts          pydantic-compatible argument validation and messages
    json.ts                Python-compatible JSON rendering
    pyfloat.ts             int/float fidelity across the JSON round-trip
    pyrepr.ts              Python repr() for validation messages
    errors.ts              error text shaping
  api/
    httpClient.ts          HTTP transport, ApiException taxonomy, auth, TLS, SOCKS
    sparkClient.ts         Spark REST facade: pagination, attempts, status filters
  models/
    generated.ts           model shapes, generated from the upstream OpenAPI models
    deserialize.ts         from_dict / model_dump equivalents
    mcpTypes.ts            curated LLM-facing output models
  tools/tools.ts           the 17 tools
  prompts/prompts.ts       the 2 prompts
  schemas/generated.ts     tool + prompt catalogue (names, descriptions, schemas)

Three details worth knowing if you plan to modify it:

• models/generated.ts and schemas/generated.ts are generated, by tools/gen_models.py and tools/gen_schemas.py, from the upstream Python project. Regenerate rather than hand-edit — that is what keeps the catalogue and the response shapes identical to the original.

• The low-level Server API is used, not McpServer, because the result shape has to match FastMCP's: one text block per list element, and structuredContent only for the tools whose Python signature declared a concrete return type.

• Application discovery lets tools omit server. ApplicationDiscovery probes each configured server for the application id and caches the answer for 5 minutes.

7. Deployment

Docker

docker build -t spark-history-mcp .
docker run -p 18888:18888 \
  -e SHS_SERVERS__PROD__URL=https://spark-history.company.com:18080 \
  -e SHS_SERVERS__PROD__DEFAULT=true \
  -e SHS_MCP__ADDRESS=0.0.0.0 \
  spark-history-mcp

Kubernetes

Run it as a normal Deployment with the URL in the env and credentials from a Secret:

env:
  - name: SHS_MCP__TRANSPORT
    value: streamable-http
  - name: SHS_MCP__ADDRESS
    value: "0.0.0.0"
  - name: SHS_SERVERS__PROD__URL
    value: http://spark-history-server.spark.svc.cluster.local:18080
  - name: SHS_SERVERS__PROD__DEFAULT
    value: "true"
  - name: SHS_SERVERS__PROD__AUTH__TOKEN
    valueFrom:
      secretKeyRef: { name: spark-history-auth, key: token }

The process is stateless apart from the 5-minute discovery cache, so it scales horizontally without coordination.

8. Troubleshooting

Set SHS_MCP__DEBUG=true for verbose logs.

9. Development

npm install
npm run build        # compile to dist/
npm run dev          # run from source, no build step
npm test             # unit tests
npm run typecheck    # tsc --noEmit

Cross-implementation parity testing lives in parity/ — it runs the same MCP calls against this server and the Python original and diffs every response. PARITY.md records the results and the exact differences that remain.

Not ported from upstream

License

Apache-2.0, as with the upstream project.