Summary: An MCP server that gives an agent a persistent local Spark session for exploring real OneLake/Fabric Delta data and honing PySpark notebook code locally before running it on Fabric.
Run PySpark cells (run_code): execute Python against a long-lived session where variables, imports, and state persist across calls (spark, sc, F, T, Window pre-imported).
Run Spark SQL (run_sql): query tables by Fabric name (lakehouse.table), auto-mounted on first use, with a configurable row cap and truncation flag.
Inspect or reset the session: session_info reports Spark version, master, current database, catalog databases, and registered lakehouses; reset_runtime restarts Spark and wipes all state.
Discover data: list_lakehouses and list_tables enumerate available Fabric lakehouses and their Delta tables.
Mount explicitly (mount_table, mount_lakehouse): pre-register one table or an entire lakehouse as a Spark database for use in code (usually unnecessary thanks to auto-mount).
Work like a Fabric notebook: unqualified names resolve against a default lakehouse, names are case-insensitive, session confs mirror the Fabric runtime, and run_notebook replays Git .py notebook sources cell by cell with parameters and cell selection.
Control writes: sandbox (default, local shallow clones), readonly, or writethrough to OneLake, plus shadow listing/discarding/restore.
Use Files and libraries: /lakehouse/default/Files as a real (mirrored or lazy) directory, relative Files/... Spark paths, notebookutils/mssparkutils, and third-party Python libs installable via uvx --with.
Match the Fabric runtime: fabric-1.3 or fabric-2.0 profiles pin pyspark/delta-spark, Python, and Java; works on Linux/WSL and Windows against live OneLake via az login.
Interrupt and stream: interrupt stops a running cell or query without losing state; display(df) renders tables and hosts can stream stdout or fetch SQL results as Arrow.
Host flexibility: multiple isolated contexts, a localhost worker protocol for other hosts, and TOML/env configuration with a healthcheck and warm-up tool.
Provides a stateful local Spark session for exploring and authoring code against Delta tables, including mounting Fabric lakehouse tables, running SQL and PySpark, creating local Delta shallow clones, and performing operations such as saveAsTable, INSERT, MERGE, and DeltaTable.forName.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@local-spark-mcp Run a SQL query to get count of orders by status"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An MCP server that gives an agent a stateful local Spark session to work in — a Jupyter-notebook-shaped surface with the UI stripped away. The agent runs PySpark "cells" against a long-lived session (state persists across calls), runs SQL and gets rows back, and manages the runtime through tools.
The purpose is local exploration in service of authoring PySpark notebooks that will run on Microsoft Fabric: figure things out locally against the same OneLake Delta data, then hand the honed code to the user as a notebook to run on Fabric with a reasonably similar outcome — no cloud compute burned while exploring.
Version 0.2.x: the core session plus notebook parity — a Fabric notebook
from the Git export runs unmodified against real OneLake data, in a sandbox by
default. Validated live on Linux/WSL and Windows. See CLAUDE.md for the
architecture and the locked design decisions.
Related MCP server: Fabric Data Engineering MCP Server
uvx, from GitHub)No clone or build needed — uvx installs and runs it in an ephemeral
environment. Pick a runtime profile (the extra) that matches the Fabric
runtime your workspace uses, and register it as an MCP server in Claude Code
(.mcp.json):
{
"mcpServers": {
"local-spark": {
"command": "uvx",
"args": ["--python", "3.13", "--from", "local-spark-mcp[fabric-2.0] @ git+https://github.com/methodify/local-spark-mcp@v0.3.0", "local-spark-mcp"],
"env": { "LOCAL_SPARK_WORKSPACE_NAME": "Data Warehouse", "LOCAL_SPARK_PROFILE": "fabric-2.0" }
}
}
}A profile is a version set that matches one Fabric Spark runtime. The extra pins pyspark and delta-spark; everything else version-specific (the catalog jar's Scala line, hadoop-azure, accepted Java majors) follows from what is installed. One install holds exactly one profile; a workspace on the other runtime gets a second server entry.
Profile | Fabric runtime | pyspark | delta-spark | Python | Java | Deletion-vector tables |
| 1.3 (Spark 3.5.5, Delta 3.2.1) | 3.5.5 | 3.2.0 | 3.11 | 8 / 11 / 17 | read-only live views |
| 2.0 (Spark 4.1.1, Delta 4.2.0) | 4.1.1 | 4.2.0 | 3.13 | 17 / 21 | full sandbox (clone) |
Runtime 2.0 writes deletion vectors by default (Delta reader 3 / writer 7), so
tables written by 2.0 notebooks need the fabric-2.0 profile to be writable
locally. Set LOCAL_SPARK_PROFILE (or [runtime] profile) to declare the
profile you intend; startup then fails with a clear message when the installed
stack is a different one, and warns on parity drift (a different patch version,
or a Python other than the runtime's). Without a declaration the installed
stack decides. Install lines:
uvx --python 3.11 --from "local-spark-mcp[fabric-1.3] @ git+https://github.com/methodify/local-spark-mcp@v0.3.0" local-spark-mcp
uvx --python 3.13 --from "local-spark-mcp[fabric-2.0] @ git+https://github.com/methodify/local-spark-mcp@v0.3.0" local-spark-mcpThe base package pins no Spark on purpose: an install without a profile extra fails at startup naming both extras.
Windows: use --python 3.11 for both profiles. PySpark's Python workers
crash on Windows under Python 3.12 and 3.13 unless pyspark is 3.5.9+ or 4.1.2+
(SPARK-53759), and neither profile can use those (Delta 3.2 needs Spark 3.5.5
or older; delta-spark 4.2.0 pins pyspark 4.1.1). Startup refuses the crashing
combination with the same advice. Validated on Windows with Python 3.11.
Runs on Linux/WSL and Windows (both validated end to end against live OneLake). Prerequisites on the host:
Java: 17 for fabric-1.3 (Spark 3.5 also accepts 8 and 11; 21 will not
work), 21 or 17 for fabric-2.0. The server prefers a vfox-managed JDK in the
profile's order, then JAVA_HOME, then java on PATH; or set
runtime.java_home / LOCAL_SPARK_JAVA_HOME.
az login — OneLake/Fabric auth is ambient via DefaultAzureCredential.
Windows: nothing extra. Hadoop's winutils.exe/hadoop.dll (required for
Spark to start at all on Windows) ship inside the package; point
runtime.hadoop_home at your own Hadoop if you prefer.
The prebuilt OneLake token-provider jar ships inside the package, so Fabric mode
works out of the box (no sbt needed). First run downloads PySpark/Delta jars and
is slow; subsequent runs reuse the cached environment. Use --refresh to pick up
a new commit: uvx --refresh --from git+https://github.com/methodify/local-spark-mcp local-spark-mcp.
Spark Python workers run the same interpreter as the driver, so a library
installed into the server's environment is importable in both run_code and in
distributed code (mapPartitions / UDFs). Install such libs into that env — e.g.
uvx --with jageocoder --with postal --from git+…/local-spark-mcp local-spark-mcp
— and set any data-dir env vars (and/or PYTHONPATH) under [spark.env] in
local-spark.toml; those are applied to both the driver and the workers. (A
runtime sys.path.append only affects the driver — workers won't see it.)
Tables by name. Lakehouses are Spark databases; spark.table("dataverse.custtable"),
spark.sql, saveAsTable, INSERT, MERGE, and DeltaTable.forName resolve
<lakehouse>.<table> on first touch, with no mount step. Set
[lakehouses] default (env LOCAL_SPARK_DEFAULT_LAKEHOUSE) and unqualified
names resolve against it, as on Fabric.
Write policy ([runtime] write_mode, env LOCAL_SPARK_WRITE_MODE).
sandbox (default): nothing reaches OneLake — a table you write becomes a
local Delta shallow clone (metadata only, so reads stay live and the first
write is quick) and new tables land locally; later reads in the session see
them. readonly: sandbox plus refusal of table creation and DeltaTable.forName.
writethrough: writes go to OneLake. shadow_status lists the local shadows
with a state: read (materialized by a read, unchanged) or written (has
local writes); discard_shadow resets them, or only the read or written
ones with only=. Shadows are session-scoped unless persist_shadow = true.
run_notebook runs a notebook from its Git .py source cell by cell in
the persistent namespace, with cell selection ("0-4,7"), parameters
(applied after the PARAMETERS CELL), and the notebook's own default lakehouse.
%pip / !pip / %run lines are reported, not run — bring libraries in with
uvx --with. [notebooks] root (env LOCAL_SPARK_NOTEBOOKS_ROOT) lets you
address notebooks by Fabric display name.
notebookutils / mssparkutils are importable: credentials.getSecret
(Key Vault), variableLibrary.getLibrary (Fabric REST), fs.ls / fs.exists
/ fs.mount, notebook.run / runMultiple / exit, session.stop,
runtime.context. Other members raise NotImplementedError naming the member.
/lakehouse/default/Files is a real directory — a link to a local mirror
of the default lakehouse's Files/, so open, os.listdir, subprocesses,
and native libraries reading a data directory all work. List the subtrees to
pull under [files] sync (env LOCAL_SPARK_FILES_SYNC) — subtrees like
lib/ or single files like _dwlib_hydrate_options.txt; unchanged files are
skipped, so a 2 GB tree downloads once. Writes land in the mirror; sync_files
pulls more on demand and pushes only in writethrough. Tables/ is never
mirrored. The mirror lives under ~/.local-spark/lakehouses/<workspace-id>/<lakehouse-id>/Files
(env LOCAL_SPARK_MIRROR_ROOT), shared by every project that uses that
lakehouse, and LOCAL_SPARK_FILES_ROOT names it for code that avoids the
global path.
Relative paths. A relative Files/... in Spark (spark.read.csv("Files/x"),
df.write.csv("Files/out")) resolves against the default lakehouse's mirror,
as it resolves against the default lakehouse on Fabric; session_info shows
the working directory. Python's own cwd is unchanged: use
/lakehouse/default/Files/... for open(). Spark file readers skip files
whose names start with _ or . (Hadoop treats them as hidden, on Fabric
too), so spark.read.text("Files/_dwlib_hydrate_options.txt") returns no
rows by either path; read marker files with plain Python open().
Table names are case-insensitive, as on Fabric: dataverse.chtMotifTable
resolves however you spell it and reaches the OneLake-cased path.
restore_shadow(table, version=0) rewinds one shadow to a Delta version
of its local log (default: the snapshot as first cloned) without touching
OneLake, for same-snapshot A/Bs. RESTORE TABLE cannot do this on a shallow
clone.
Session confs follow the runtime. Each profile applies the confs where a
real Fabric session differs from Spark's default (taken from production
session event logs): session time zone UTC, Kryo serializer, 25 MB
broadcast threshold, cost-based optimizer on, Arrow for toPandas,
Parquet timestamps as TIMESTAMP_MICROS, sources.default=delta (so an
untyped CREATE TABLE and df.write.save mean Delta), and under
fabric-2.0 ANSI off and deletion vectors on for new tables. Fabric-only engines and committers are
not copied. [spark.extra_configs] still overrides. Python's own time zone
stays the machine's.
Snapshot semantics. A shallow clone is frozen at the OneLake table
version current when the session first touched the table, so every read of it
in the session sees the same rows, and two sessions started at different times
can see different versions. discard_shadow re-clones at the current version;
persist_shadow = true keeps the frozen version across sessions.
Deletion-vector views (fabric-1.3) and writethrough tables are live.
Robustness. session_info never waits on a busy kernel (it reports what
is running and the last known state). Cells, SQL, and notebooks have no
server-side timeout, and there is no idle timeout: a session lives until
reset_runtime, the server exits, or the process dies. A dead driver (an
OOM, a kill) is detected on the failing call and replaced by a fresh session;
a worker that died between calls is reported at the top of the next result
(notice: runtime restarted: … exited after Ns idle (killed by SIGKILL, the OS out-of-memory killer's signal) …), and session_info shows kernel started: <time> (Ns ago). Raise [spark] driver_memory for wide joins.
Concurrent first touch is safe. Threads that touch an untouched table at the same time share one clone: the first materializes, the rest wait.
Repeated partitioned overwrites work. mode("overwrite").saveAsTable on
a partitioned shadow can be repeated (upstream Delta 4.x / Spark in-memory
catalog would otherwise fail the second one after committing).
Eager catalog population. [lakehouses] preload = ["dataverse"] (or
["all"]; env LOCAL_SPARK_PRELOAD) materializes every table of those
lakehouses in the background right after the session starts, many at a time
(preload_workers, default 32), so no read pays a first-touch mount later.
Startup returns immediately; session_info shows progress, the next result
after completion carries a notice, and preload_lakehouses starts one on
demand. Each table is one shallow clone (a Spark job that reads the table's
Delta log), so expect roughly 1.5 to 2 s per table of throughput on a
20-core laptop (mostly OneLake I/O waits): a 350-table lakehouse takes
about 10 minutes.
First-touch cost is visible. When a cell or query materializes a table
for the first time, the result leads with notice: mounted <lakehouse>.<table> in N s (shallow clone), so the mount never hides inside your own timing.
/lakehouse pathThe link is one global path per machine, so a lockfile records which session owns it and the server refuses to repoint it while another live session holds it for a different lakehouse (session_info reports this, and the mirror path still works).
Linux and WSL: /lakehouse must exist and be writable by you. One-time
setup: sudo mkdir /lakehouse && sudo chown $USER /lakehouse. If /lakehouse
is already a symlink into a directory you own, the server uses it as is. The
server never creates /lakehouse itself; it reports the command and runs
without the link.
Windows: C:\lakehouse\default is a directory junction, created without
elevation. A path beginning with / resolves against the current drive, so
/lakehouse/default/Files works when the session runs from C:.
interrupt stops the cell or query that is running (cancels Spark jobs, raises
KeyboardInterrupt in the kernel) and keeps the session's state. display(df)
in a cell renders a DataFrame as a table (hosts embedding the worker receive it
as Arrow). Hosts can also stream a cell's stdout as it is written and fetch SQL
results as Arrow IPC; see docs/PROTOCOL.md. 0.4.1 makes run_sql
interruptible, lets a host capture a bare trailing DataFrame as Arrow
(capture_result), and reports the running cell's elapsed time and active Spark
job count on the control socket's status.
A lakehouse with schemas (Tables/<schema>/<table> on OneLake) appears with
the Fabric spelling: test.dbo.holidays, SHOW NAMESPACES IN test, and USE test selecting its default schema. Each schema is also a plain session
database, test__dbo. The server detects schemas from the OneLake listing at
start; a host can declare them instead. Table discovery for preload and
list_tables reads OneLake storage rather than the Fabric REST endpoint, which
refuses schema-enabled lakehouses.
The worker takes every Azure token from the token endpoint it is started with
(?scope= selects Fabric REST, OneLake, or Key Vault), so a host that embeds
it needs no az login inside the worker, and preload failures appear in
preload_status, never in a cell's output.
With [files] mode = "lazy" (env LOCAL_SPARK_FILES_MODE), Spark's relative
Files/… is the default lakehouse's OneLake Files/, read directly through the
host token with nothing mirrored, and each context's Files/ follows its own
default lakehouse. In sandbox and readonly mode a Spark write under Files/ is
refused with a message that names the alternatives; writethrough passes it
through. For Python IO, /lakehouse/default/Files/... (and /lakehouse/<name>/...)
works without syncing: a file is fetched into the mirror on first open,
listings and exists/isdir come from OneLake, writes stay local (pushed only in
writethrough), and /lakehouse/default means the active context's default
lakehouse. Native readers still need the file on disk first (sync_files);
mirror_status and clear_mirror manage the mirror. The default remains
"mirror".
A host can run several notebooks against one Spark runtime without crosstalk: each context is its own Python namespace and its own Spark session (temp views, SQL conf, current database, UDFs), sharing the JVM, the catalog, clones, and cached data, the way Fabric's high-concurrency sessions work. The MCP server uses the default context; hosts create more through the worker protocol.
Hosts other than Claude Code can drive the Spark worker directly over its
localhost socket protocol (Cobalt SQL Works does): see docs/PROTOCOL.md.
profiles.json at the repo root is the machine-readable runtime manifest
(python -m local_spark_mcp.profiles --json prints the installed package's),
python -m local_spark_mcp.healthcheck checks an environment without starting
Spark, and python -m local_spark_mcp.warm pre-resolves the Spark packages.
[spark] jars and [spark] packages (env LOCAL_SPARK_JARS,
LOCAL_SPARK_PACKAGES) add your own jars and Maven coordinates to the session.
Configuration lives in a local-spark.toml file in the working directory (see
local-spark.example.toml), discovered by walking up from where the server is
launched. Environment variables (LOCAL_SPARK_*) override individual settings —
convenient in the MCP env block above when you don't want a file. With no
workspace configured the server runs local-only (no Fabric). Auth is ambient via
az login, so nothing in the config is secret.
To control which file is read, set LOCAL_SPARK_CONFIG to a path, or to none
to read no file at all (environment only). The console script accepts the same
as --config PATH / --no-config. At startup the server logs to stderr which
file it read (or why none), every LOCAL_SPARK_* override that applied, and the
origin of java_home / token_jar_path / hadoop_home; a value that fails
validation is reported with its origin, and the jar is checked for the classes
this version needs before Spark starts.
Java discovery, when java_home is not set: a vfox-managed JDK 17/11, then
JAVA_HOME, then java on PATH. Every candidate is resolved through
symlinks and junctions, a path to bin/java is normalized to its home, and a
JDK whose release file says anything other than 8, 11, or 17 is skipped. The
error lists each candidate and why it was rejected.
Apache License 2.0. See LICENSE, and NOTICE for the bundled third-party
components (Apache Hadoop winutils).
List the Fabric lakehouses available in this session. Each is registered as a Spark database; its tables are mounted on demand.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses that lakehouses are registered as Spark databases and tables are mounted on demand, implying listing does not mount tables. However, it does not state read-only behavior, permissions, or effect of no lakehouses, so moderate transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with the action, no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the zero-parameter interface and presence of an output schema, the description sufficiently explains purpose and key behavioral context. It covers what is listed and the on-demand mounting relationship, making it complete for a simple list tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has 0 parameters, so baseline is 4. The description adds no parameter-specific detail but explains the domain concept (lakehouses as databases), which is sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists Fabric lakehouses available in the session, using the specific verb 'List' and resource 'Fabric lakehouses'. The additional detail about Spark database registration distinguishes it from list_tables and other sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool over alternatives, but implies usage for viewing session-available lakehouses. It mentions they are registered as Spark databases and tables mounted on demand, providing context but no explicit exclusions or alternative tool recommendations.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the Delta tables in a Fabric lakehouse. Tables are not queryable via SQL until you mount them with mount_table or mount_lakehouse.
| Name | Required | Description | Default |
|---|---|---|---|
| lakehouse | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses a key behavior ('Tables are not queryable via SQL until you mount them'), which is useful. However, it does not explicitly state that listing itself is read-only, or mention behavior on missing lakehouses or return format details. Score 3 reflects adequate but not rich transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core purpose. The second sentence adds essential context about SQL queryability and mounting without any redundancy. It is precise and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool with an output schema, the description covers the basic purpose and a workflow note, but it lacks parameter semantics and any mention of prerequisites or error cases. It is minimally viable but not fully complete for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description adds only the vague phrase 'in a Fabric lakehouse.' It does not specify whether the 'lakehouse' parameter expects a name, ID, or path, nor any format constraints. The agent is left guessing about how to fill in the required parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'List' and identifies the exact resource ('Delta tables in a Fabric lakehouse'). It also distinguishes the tool from siblings by focusing on tables rather than lakehouses and by mentioning the mounting workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context: tables are not queryable via SQL until mounted, which tells the agent when this tool is useful (for discovery before mounting). It does not explicitly name alternatives for listing lakehouses, but the mount tools are mentioned, giving workflow guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Mount ALL tables in a lakehouse as <lakehouse>.<table>. Convenient, but can register many tables at once.
| Name | Required | Description | Default |
|---|---|---|---|
| lakehouse | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that this mounts all tables and can register many at once, which is a key behavioral trait and potential risk. It does not mention reversibility or permissions, but the main caveat is covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the core action, no redundant information. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present and a single simple parameter, the description sufficiently covers purpose, scope, and a side effect. It doesn't address edge cases like existing mounts, but for a bulk mount tool this is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has no descriptions (0% coverage), but the description uses the `<lakehouse>` placeholder to explain the parameter's role in the naming convention. This adds meaningful context to an otherwise bare parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Mount') and resource ('ALL tables in a lakehouse'), and clearly defines the naming convention (`<lakehouse>`.`<table>`). This distinguishes it from sibling tools like mount_table, which mounts a single table.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use this tool (when you want all tables) and warns about the side effect of registering many tables at once. It does not explicitly name alternatives like mount_table, but the contrast is clear from the phrase 'Mount ALL tables'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Explicitly register one Fabric Delta table as <lakehouse>.<table>. Usually unnecessary — run_sql auto-mounts referenced tables — but useful to pre-register a table for use in run_code.
| Name | Required | Description | Default |
|---|---|---|---|
| table | Yes | ||
| lakehouse | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It explains the registration action and the naming pattern, but does not disclose idempotency, persistence, error conditions, or side effects. This is minimal but sufficient for a conceptually simple registration operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the action and immediately followed by practical guidance. Every word contributes; no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description fully covers the tool's purpose, typical use case, and relationship to sibling tools. Given the low complexity (2 required string params, no nested objects) and existing output schema, nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and there are only 2 string parameters. The description adds meaning by showing the pattern `<lakehouse>`.`<table>`, directly mapping to the parameter names and clarifying that 'table' is a Fabric Delta table and 'lakehouse' is the lakehouse context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'register' with a clear resource 'one Fabric Delta table' as `<lakehouse>`.`<table>`. It distinguishes itself from run_sql and mount_lakehouse by explaining it pre-registers a table for run_code, contrasting with run_sql's auto-mounting.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use ('useful to pre-register a table for use in run_code') and when not needed ('Usually unnecessary — run_sql auto-mounts referenced tables'). The alternative tool is named and contrasted clearly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Reset the runtime: restart the Spark session and wipe all state — variables, imports, and mounted tables. Use to start from a clean slate.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses the destructive behavior ('wipe all state'), specifies what is affected ('variables, imports, and mounted tables'), and mentions restarting the session. It could have added more about irreversibility or impact on running work, but the core behavioral transparency is strong.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: the first states the action and its effects, the second gives the usage. It is front-loaded with the key verb 'Reset', avoids fluff, and every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity of the tool (no parameters) and the presence of an output schema, the description adequately covers the operation's purpose, effects, and use case. It is complete enough for an agent to select and invoke the tool correctly without additional context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so schema coverage is trivially 100%. The description does not need to explain parameters since there are none. Baseline for 0-parameter tools is 4, and no additional parameter semantics are necessary.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the action ('Reset the runtime'), the specific resource ('Spark session'), and the scope ('wipe all state — variables, imports, and mounted tables'). This clearly differentiates it from sibling tools like run_code, run_sql, or mount_table.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear guidance on when to use ('Use to start from a clean slate'), which implies a scenario where the user needs to clear all state. It does not explicitly name alternatives or exclusions, but the usage context is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a cell of Python/PySpark against the persistent session. State persists across calls; spark, sc, F, T, Window are pre-imported. Returns captured stdout and the last-expression echo, or the traceback if the cell raised.
| Name | Required | Description | Default |
|---|---|---|---|
| code | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and discloses key traits: persistent session, state preservation, pre-imported symbols, and return behavior (stdout, last-expression echo, or traceback). This is substantial coverage for a code execution tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the primary purpose. Every element earns its place: what it runs, persistence, pre-imports, and return behavior. No redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple (1 required param) and the description covers purpose, statefulness, available symbols, and return values. The presence of an output schema further clarifies the return structure, making this description adequate on its own.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single 'code' parameter is explained as a cell of Python/PySpark, and the pre-imports listed clarify what is available in scope. This adds meaning beyond the raw schema ('code' string) and compensates for the 0% schema description coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Run') and resource ('a cell of Python/PySpark against the persistent session'), clearly distinguishing it from sibling tools like run_sql. It states exactly what the tool does without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context: it is for Python/PySpark cells, state persists across calls, and pre-imports are listed. It implicitly distinguishes from SQL execution (run_sql) but does not explicitly state when not to use or mention alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a Spark SQL statement and return rows as a text table. Reference Fabric tables by name (lakehouse.table) — they auto-mount on first use and stay available for the session. limit caps returned rows (default from config, ~100) and the result flags truncation.
| Name | Required | Description | Default |
|---|---|---|---|
| sql | Yes | ||
| limit | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It reveals the text-table return format, truncation flag, auto-mounting, and session availability. It does not explicitly state whether statements are read-only, but the implied use case focuses on querying.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three concise sentences, front-loaded with the primary action. Every sentence provides essential information without redundancy, making it efficient and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers usage, parameter semantics, and key behaviors like auto-mounting and truncation. Since an output schema exists, return values need not be detailed. The main gap is the lack of explicit read-only or write clarification, which is minor for this query-oriented tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has no descriptions for parameters, but the description compensates fully: 'sql' is explained as the Spark SQL statement, and 'limit' is described as capping returned rows with a default from config (~100). This adds significant meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool runs a Spark SQL statement and returns rows as a text table, using a specific verb and resource. It distinguishes from sibling tools like run_code by emphasizing SQL execution and the text-table output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool, such as referencing Fabric tables and auto-mounting behavior. However, it does not explicitly mention when not to use it or name alternative tools, like run_code.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Show the live Spark session: version, master, current database, the catalog databases, and any Fabric lakehouses registered this session.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. The verb 'Show' strongly implies a non-destructive, read-only operation, and the term 'live' adds context that it reflects the current state. However, it does not explicitly state that it has no side effects or what happens if no session exists, so it is not a perfect 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that front-loads the core action ('Show the live Spark session') and then lists the details. Every word adds value, and there is no redundancy or irrelevant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a simple zero-parameter tool with an output schema. The description fully conveys what information is available, and the output schema provides the return structure. No additional context is needed for the agent to invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is no schema detail to explain. The description still adds value by enumerating the specific information returned (version, master, current database, catalog databases, lakehouses), which gives the agent a clear picture of the tool's output even without parameter definitions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Show') and clearly identifies the resource (live Spark session) and its key contents (version, master, current database, catalog databases, lakehouses). This makes the tool's purpose unambiguous and distinguishes it from siblings like run_code and run_sql, which execute code rather than introspect the session.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for inspecting current session state, which provides clear context for when to use it. It does not explicitly mention alternatives or when not to use it, but the purpose is obvious enough that an agent can select it appropriately among the sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0list_lakehouseslist_tablesmount_lakehousemount_tablereset_runtimerun_coderun_sqlsession_infoScored across 8 tools
Each tool has a clearly distinct role: running Python vs SQL, inspecting session state, resetting, and managing lakehouse/table mounting. The potential overlap between mount_table and mount_lakehouse is clearly scoped.
Most tools follow a verb_noun snake_case pattern (run_code, run_sql, list_lakehouses, mount_table), but session_info is a noun_noun exception. Still consistent style overall.
8 tools is well-scoped for a Spark session server, covering execution, SQL, state management, and catalog operations without redundancy.
The surface covers the full workflow: execute code, query SQL, inspect session, reset state, list and mount data sources. Auto-mounting in run_sql and explicit mounting in mount_* cover the data access lifecycle.
Your Databricks Lakehouse in natural language: run SQL on your SQL warehouses, track long-running qu
The all-in-one data stack for agents. Upload files, run SQL, evolve tables, and render charts.
Individual-analyst context layer: recall, remember, and reconcile your data and engineering context.
The grounded data layer for any LLM: governed SQL, metrics, lineage and catalog over your data.
