local-spark-mcp

An MCP server that gives an agent a stateful local Spark session to work in — a Jupyter-notebook-shaped surface with the UI stripped away. The agent runs PySpark "cells" against a long-lived session (state persists across calls), runs SQL and gets rows back, and manages the runtime through tools.

The purpose is local exploration in service of authoring PySpark notebooks that will run on Microsoft Fabric: figure things out locally against the same OneLake Delta data, then hand the honed code to the user as a notebook to run on Fabric with a reasonably similar outcome — no cloud compute burned while exploring.

Status

Version 0.2.x: the core session plus notebook parity — a Fabric notebook from the Git export runs unmodified against real OneLake data, in a sandbox by default. Validated live on Linux/WSL and Windows. See CLAUDE.md for the architecture and the locked design decisions.

Related MCP server: Fabric Data Engineering MCP Server

Running it (via uvx, from GitHub)

No clone or build needed — uvx installs and runs it in an ephemeral environment. Pick a runtime profile (the extra) that matches the Fabric runtime your workspace uses, and register it as an MCP server in Claude Code (.mcp.json):

{
  "mcpServers": {
    "local-spark": {
      "command": "uvx",
      "args": ["--python", "3.13", "--from", "local-spark-mcp[fabric-2.0] @ git+https://github.com/methodify/local-spark-mcp@v0.3.0", "local-spark-mcp"],
      "env": { "LOCAL_SPARK_WORKSPACE_NAME": "Data Warehouse", "LOCAL_SPARK_PROFILE": "fabric-2.0" }
    }
  }
}

Runtime profiles

A profile is a version set that matches one Fabric Spark runtime. The extra pins pyspark and delta-spark; everything else version-specific (the catalog jar's Scala line, hadoop-azure, accepted Java majors) follows from what is installed. One install holds exactly one profile; a workspace on the other runtime gets a second server entry.

Runtime 2.0 writes deletion vectors by default (Delta reader 3 / writer 7), so tables written by 2.0 notebooks need the fabric-2.0 profile to be writable locally. Set LOCAL_SPARK_PROFILE (or [runtime] profile) to declare the profile you intend; startup then fails with a clear message when the installed stack is a different one, and warns on parity drift (a different patch version, or a Python other than the runtime's). Without a declaration the installed stack decides. Install lines:

uvx --python 3.11 --from "local-spark-mcp[fabric-1.3] @ git+https://github.com/methodify/local-spark-mcp@v0.3.0" local-spark-mcp
uvx --python 3.13 --from "local-spark-mcp[fabric-2.0] @ git+https://github.com/methodify/local-spark-mcp@v0.3.0" local-spark-mcp

The base package pins no Spark on purpose: an install without a profile extra fails at startup naming both extras.

Windows: use --python 3.11 for both profiles. PySpark's Python workers crash on Windows under Python 3.12 and 3.13 unless pyspark is 3.5.9+ or 4.1.2+ (SPARK-53759), and neither profile can use those (Delta 3.2 needs Spark 3.5.5 or older; delta-spark 4.2.0 pins pyspark 4.1.1). Startup refuses the crashing combination with the same advice. Validated on Windows with Python 3.11.

Runs on Linux/WSL and Windows (both validated end to end against live OneLake). Prerequisites on the host:

• Java: 17 for fabric-1.3 (Spark 3.5 also accepts 8 and 11; 21 will not work), 21 or 17 for fabric-2.0. The server prefers a vfox-managed JDK in the profile's order, then JAVA_HOME, then java on PATH; or set runtime.java_home / LOCAL_SPARK_JAVA_HOME.

• az login — OneLake/Fabric auth is ambient via DefaultAzureCredential.

• Windows: nothing extra. Hadoop's winutils.exe/hadoop.dll (required for Spark to start at all on Windows) ship inside the package; point runtime.hadoop_home at your own Hadoop if you prefer.

The prebuilt OneLake token-provider jar ships inside the package, so Fabric mode works out of the box (no sbt needed). First run downloads PySpark/Delta jars and is slow; subsequent runs reuse the cached environment. Use --refresh to pick up a new commit: uvx --refresh --from git+https://github.com/methodify/local-spark-mcp local-spark-mcp.

Native / 3rd-party Python libs in distributed code

Spark Python workers run the same interpreter as the driver, so a library installed into the server's environment is importable in both run_code and in distributed code (mapPartitions / UDFs). Install such libs into that env — e.g. uvx --with jageocoder --with postal --from git+…/local-spark-mcp local-spark-mcp — and set any data-dir env vars (and/or PYTHONPATH) under [spark.env] in local-spark.toml; those are applied to both the driver and the workers. (A runtime sys.path.append only affects the driver — workers won't see it.)

Working like a Fabric notebook

• Tables by name. Lakehouses are Spark databases; spark.table("dataverse.custtable"), spark.sql, saveAsTable, INSERT, MERGE, and DeltaTable.forName resolve <lakehouse>.<table> on first touch, with no mount step. Set [lakehouses] default (env LOCAL_SPARK_DEFAULT_LAKEHOUSE) and unqualified names resolve against it, as on Fabric.

• Write policy ([runtime] write_mode, env LOCAL_SPARK_WRITE_MODE). sandbox (default): nothing reaches OneLake — a table you write becomes a local Delta shallow clone (metadata only, so reads stay live and the first write is quick) and new tables land locally; later reads in the session see them. readonly: sandbox plus refusal of table creation and DeltaTable.forName. writethrough: writes go to OneLake. shadow_status lists the local shadows with a state: read (materialized by a read, unchanged) or written (has local writes); discard_shadow resets them, or only the read or written ones with only=. Shadows are session-scoped unless persist_shadow = true.

• run_notebook runs a notebook from its Git .py source cell by cell in the persistent namespace, with cell selection ("0-4,7"), parameters (applied after the PARAMETERS CELL), and the notebook's own default lakehouse. %pip / !pip / %run lines are reported, not run — bring libraries in with uvx --with. [notebooks] root (env LOCAL_SPARK_NOTEBOOKS_ROOT) lets you address notebooks by Fabric display name.

• notebookutils / mssparkutils are importable: credentials.getSecret (Key Vault), variableLibrary.getLibrary (Fabric REST), fs.ls / fs.exists / fs.mount, notebook.run / runMultiple / exit, session.stop, runtime.context. Other members raise NotImplementedError naming the member.

• /lakehouse/default/Files is a real directory — a link to a local mirror of the default lakehouse's Files/, so open, os.listdir, subprocesses, and native libraries reading a data directory all work. List the subtrees to pull under [files] sync (env LOCAL_SPARK_FILES_SYNC) — subtrees like lib/ or single files like _dwlib_hydrate_options.txt; unchanged files are skipped, so a 2 GB tree downloads once. Writes land in the mirror; sync_files pulls more on demand and pushes only in writethrough. Tables/ is never mirrored. The mirror lives under ~/.local-spark/lakehouses/<workspace-id>/<lakehouse-id>/Files (env LOCAL_SPARK_MIRROR_ROOT), shared by every project that uses that lakehouse, and LOCAL_SPARK_FILES_ROOT names it for code that avoids the global path.

• Relative paths. A relative Files/... in Spark (spark.read.csv("Files/x"), df.write.csv("Files/out")) resolves against the default lakehouse's mirror, as it resolves against the default lakehouse on Fabric; session_info shows the working directory. Python's own cwd is unchanged: use /lakehouse/default/Files/... for open(). Spark file readers skip files whose names start with _ or . (Hadoop treats them as hidden, on Fabric too), so spark.read.text("Files/_dwlib_hydrate_options.txt") returns no rows by either path; read marker files with plain Python open().

• Table names are case-insensitive, as on Fabric: dataverse.chtMotifTable resolves however you spell it and reaches the OneLake-cased path.

• restore_shadow(table, version=0) rewinds one shadow to a Delta version of its local log (default: the snapshot as first cloned) without touching OneLake, for same-snapshot A/Bs. RESTORE TABLE cannot do this on a shallow clone.

• Session confs follow the runtime. Each profile applies the confs where a real Fabric session differs from Spark's default (taken from production session event logs): session time zone UTC, Kryo serializer, 25 MB broadcast threshold, cost-based optimizer on, Arrow for toPandas, Parquet timestamps as TIMESTAMP_MICROS, sources.default=delta (so an untyped CREATE TABLE and df.write.save mean Delta), and under fabric-2.0 ANSI off and deletion vectors on for new tables. Fabric-only engines and committers are not copied. [spark.extra_configs] still overrides. Python's own time zone stays the machine's.

• Snapshot semantics. A shallow clone is frozen at the OneLake table version current when the session first touched the table, so every read of it in the session sees the same rows, and two sessions started at different times can see different versions. discard_shadow re-clones at the current version; persist_shadow = true keeps the frozen version across sessions. Deletion-vector views (fabric-1.3) and writethrough tables are live.

• Robustness. session_info never waits on a busy kernel (it reports what is running and the last known state). Cells, SQL, and notebooks have no server-side timeout, and there is no idle timeout: a session lives until reset_runtime, the server exits, or the process dies. A dead driver (an OOM, a kill) is detected on the failing call and replaced by a fresh session; a worker that died between calls is reported at the top of the next result (notice: runtime restarted: … exited after Ns idle (killed by SIGKILL, the OS out-of-memory killer's signal) …), and session_info shows kernel started: <time> (Ns ago). Raise [spark] driver_memory for wide joins.

• Concurrent first touch is safe. Threads that touch an untouched table at the same time share one clone: the first materializes, the rest wait.

• Repeated partitioned overwrites work. mode("overwrite").saveAsTable on a partitioned shadow can be repeated (upstream Delta 4.x / Spark in-memory catalog would otherwise fail the second one after committing).

• Eager catalog population. [lakehouses] preload = ["dataverse"] (or ["all"]; env LOCAL_SPARK_PRELOAD) materializes every table of those lakehouses in the background right after the session starts, many at a time (preload_workers, default 32), so no read pays a first-touch mount later. Startup returns immediately; session_info shows progress, the next result after completion carries a notice, and preload_lakehouses starts one on demand. Each table is one shallow clone (a Spark job that reads the table's Delta log), so expect roughly 1.5 to 2 s per table of throughput on a 20-core laptop (mostly OneLake I/O waits): a 350-table lakehouse takes about 10 minutes.

• First-touch cost is visible. When a cell or query materializes a table for the first time, the result leads with notice: mounted <lakehouse>.<table> in N s (shallow clone), so the mount never hides inside your own timing.

The /lakehouse path

The link is one global path per machine, so a lockfile records which session owns it and the server refuses to repoint it while another live session holds it for a different lakehouse (session_info reports this, and the mirror path still works).

• Linux and WSL: /lakehouse must exist and be writable by you. One-time setup: sudo mkdir /lakehouse && sudo chown $USER /lakehouse. If /lakehouse is already a symlink into a directory you own, the server uses it as is. The server never creates /lakehouse itself; it reports the command and runs without the link.

• Windows: C:\lakehouse\default is a directory junction, created without elevation. A path beginning with / resolves against the current drive, so /lakehouse/default/Files works when the session runs from C:.

Interrupt, streaming, Arrow (0.4.0)

interrupt stops the cell or query that is running (cancels Spark jobs, raises KeyboardInterrupt in the kernel) and keeps the session's state. display(df) in a cell renders a DataFrame as a table (hosts embedding the worker receive it as Arrow). Hosts can also stream a cell's stdout as it is written and fetch SQL results as Arrow IPC; see docs/PROTOCOL.md. 0.4.1 makes run_sql interruptible, lets a host capture a bare trailing DataFrame as Arrow (capture_result), and reports the running cell's elapsed time and active Spark job count on the control socket's status.

Schema-enabled lakehouses and host tokens (0.4.1)

A lakehouse with schemas (Tables/<schema>/<table> on OneLake) appears with the Fabric spelling: test.dbo.holidays, SHOW NAMESPACES IN test, and USE test selecting its default schema. Each schema is also a plain session database, test__dbo. The server detects schemas from the OneLake listing at start; a host can declare them instead. Table discovery for preload and list_tables reads OneLake storage rather than the Fabric REST endpoint, which refuses schema-enabled lakehouses.

The worker takes every Azure token from the token endpoint it is started with (?scope= selects Fabric REST, OneLake, or Key Vault), so a host that embeds it needs no az login inside the worker, and preload failures appear in preload_status, never in a cell's output.

Lazy Files (0.6.0)

With [files] mode = "lazy" (env LOCAL_SPARK_FILES_MODE), Spark's relative Files/… is the default lakehouse's OneLake Files/, read directly through the host token with nothing mirrored, and each context's Files/ follows its own default lakehouse. In sandbox and readonly mode a Spark write under Files/ is refused with a message that names the alternatives; writethrough passes it through. For Python IO, /lakehouse/default/Files/... (and /lakehouse/<name>/...) works without syncing: a file is fetched into the mirror on first open, listings and exists/isdir come from OneLake, writes stay local (pushed only in writethrough), and /lakehouse/default means the active context's default lakehouse. Native readers still need the file on disk first (sync_files); mirror_status and clear_mirror manage the mirror. The default remains "mirror".

Contexts (0.5.0)

A host can run several notebooks against one Spark runtime without crosstalk: each context is its own Python namespace and its own Spark session (temp views, SQL conf, current database, UDFs), sharing the JVM, the catalog, clones, and cached data, the way Fabric's high-concurrency sessions work. The MCP server uses the default context; hosts create more through the worker protocol.

Embedding the worker

Hosts other than Claude Code can drive the Spark worker directly over its localhost socket protocol (Cobalt SQL Works does): see docs/PROTOCOL.md. profiles.json at the repo root is the machine-readable runtime manifest (python -m local_spark_mcp.profiles --json prints the installed package's), python -m local_spark_mcp.healthcheck checks an environment without starting Spark, and python -m local_spark_mcp.warm pre-resolves the Spark packages. [spark] jars and [spark] packages (env LOCAL_SPARK_JARS, LOCAL_SPARK_PACKAGES) add your own jars and Maven coordinates to the session.

Configuration

Configuration lives in a local-spark.toml file in the working directory (see local-spark.example.toml), discovered by walking up from where the server is launched. Environment variables (LOCAL_SPARK_*) override individual settings — convenient in the MCP env block above when you don't want a file. With no workspace configured the server runs local-only (no Fabric). Auth is ambient via az login, so nothing in the config is secret.

To control which file is read, set LOCAL_SPARK_CONFIG to a path, or to none to read no file at all (environment only). The console script accepts the same as --config PATH / --no-config. At startup the server logs to stderr which file it read (or why none), every LOCAL_SPARK_* override that applied, and the origin of java_home / token_jar_path / hadoop_home; a value that fails validation is reported with its origin, and the jar is checked for the classes this version needs before Spark starts.

Java discovery, when java_home is not set: a vfox-managed JDK 17/11, then JAVA_HOME, then java on PATH. Every candidate is resolved through symlinks and junctions, a path to bin/java is normalized to its home, and a JDK whose release file says anything other than 8, 11, or 17 is skipped. The error lists each candidate and why it was rejected.

License

Apache License 2.0. See LICENSE, and NOTICE for the bundled third-party components (Apache Hadoop winutils).