# Spark Iceberg MCP Server

A [Model Context Protocol](https://modelcontextprotocol.io/) server that provides **read-only** access to Apache Iceberg tables via **Spark SQL**.

Use it from Cursor, Claude Desktop, or any MCP client when you have Iceberg data on HDFS/S3 and want an LLM to inspect schemas and run `SELECT` queries — without Hive or Impala.

Inspired by [cloudera/iceberg-mcp-server](https://github.com/cloudera/iceberg-mcp-server) (Impala-based).

## Tools

| Tool | Description |
|------|-------------|
| `execute_query(query)` | Run read-only Spark SQL (`SELECT`, `SHOW`, `DESCRIBE`, `WITH`, `EXPLAIN`) |
| `list_namespaces()` | `SHOW NAMESPACES IN <catalog>` |
| `list_tables(namespace)` | `SHOW TABLES IN <catalog>.<namespace>` |
| `describe_table(table)` | `DESCRIBE TABLE` for a table |

Results are returned as JSON with `columns`, `rows`, and `truncated` flags.

## Requirements

- Python 3.9+
- Spark 3.x with Iceberg runtime JAR
- Network/filesystem access to the Iceberg warehouse (e.g. HDFS on a CDP node)

Two execution modes:

| Mode | When to use |
|------|-------------|
| **subprocess** (recommended on CDP) | `spark3-submit` available; set `SPARK_SUBMIT_BIN` |
| **pyspark** | Local dev with `uv sync --extra pyspark` |

## Quick start (CDP lab cluster)

Run the MCP server **on the cluster node** (or any host with `spark3-submit` + HDFS access):

```bash
git clone https://github.com/will-sh/spark-iceberg-mcp-server.git
cd spark-iceberg-mcp-server
uv sync
```

Set environment variables (example for wxiao-732 lab):

```bash
export SPARK_EXECUTION_MODE=subprocess
export SPARK_SUBMIT_BIN=spark3-submit
export SPARK_MASTER="local[2]"
export SPARK_ICEBERG_CATALOG=local
export SPARK_ICEBERG_CATALOG_TYPE=hadoop
export SPARK_ICEBERG_WAREHOUSE="hdfs://ccycloud-1.wxiao-732.root.comops.site:8020/user/systest/iceberg-warehouse"
export SPARK_ICEBERG_RUNTIME_JAR="/opt/cloudera/parcels/CDH-7.3.2-1.cdh7.3.2.p30000.83076434/lib/iceberg/iceberg-spark-runtime-3.5_2.12-*.jar"
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_MAX_ROWS=1000
```

Test a query directly:

```bash
uv run spark-iceberg-query --query "SELECT count(*) AS n FROM local.datapulse.events"
```

Start the MCP server:

```bash
uv run run-server
```

## Cursor configuration

Add to Cursor MCP settings (`~/.cursor/mcp.json` or project MCP config):

```json
{
  "mcpServers": {
    "spark-iceberg": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/spark-iceberg-mcp-server",
        "run",
        "run-server"
      ],
      "env": {
        "SPARK_EXECUTION_MODE": "subprocess",
        "SPARK_SUBMIT_BIN": "spark3-submit",
        "SPARK_MASTER": "local[2]",
        "SPARK_ICEBERG_CATALOG": "local",
        "SPARK_ICEBERG_CATALOG_TYPE": "hadoop",
        "SPARK_ICEBERG_WAREHOUSE": "hdfs://ccycloud-1.wxiao-732.root.comops.site:8020/user/systest/iceberg-warehouse",
        "SPARK_ICEBERG_RUNTIME_JAR": "/opt/cloudera/parcels/CDH-7.3.2-1.cdh7.3.2.p30000.83076434/lib/iceberg/iceberg-spark-runtime-3.5_2.12-*.jar",
        "HADOOP_CONF_DIR": "/etc/hadoop/conf",
        "SPARK_MAX_ROWS": "100"
      }
    }
  }
}
```

Or install from GitHub with `uvx`:

```json
{
  "mcpServers": {
    "spark-iceberg": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/will-sh/spark-iceberg-mcp-server@main",
        "run-server"
      ],
      "env": {
        "SPARK_EXECUTION_MODE": "subprocess",
        "SPARK_SUBMIT_BIN": "spark3-submit",
        "SPARK_ICEBERG_WAREHOUSE": "hdfs://your-nn:8020/user/systest/iceberg-warehouse",
        "SPARK_ICEBERG_RUNTIME_JAR": "/opt/cloudera/parcels/CDH/lib/iceberg/iceberg-spark-runtime-3.5_2.12-*.jar"
      }
    }
  }
}
```

> **Note:** `uvx` still needs `spark3-submit` and HDFS access on the host where Cursor launches the MCP process. For remote CDP clusters, run Cursor's MCP on the cluster node via SSH Remote, or use a bastion with Hadoop client configs.

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `SPARK_EXECUTION_MODE` | `auto` | `subprocess`, `pyspark`, or `auto` (subprocess if `SPARK_SUBMIT_BIN` set) |
| `SPARK_SUBMIT_BIN` | `spark3-submit` | Spark submit binary for subprocess mode |
| `SPARK_MASTER` | `local[2]` | Spark master URL (`local[2]`, `yarn`, etc.) |
| `SPARK_ICEBERG_CATALOG` | `local` | Iceberg catalog name |
| `SPARK_ICEBERG_CATALOG_TYPE` | `hadoop` | Catalog type (`hadoop`, `hive`, `rest`, …) |
| `SPARK_ICEBERG_WAREHOUSE` | — | Warehouse path (required for Hadoop catalog) |
| `SPARK_ICEBERG_RUNTIME_JAR` | — | Path to `iceberg-spark-runtime` JAR (glob supported) |
| `SPARK_HOME` | — | Optional `SPARK_HOME` for subprocess |
| `HADOOP_CONF_DIR` | — | Hadoop config directory |
| `SPARK_MAX_ROWS` | `1000` | Max rows returned per query |
| `MCP_TRANSPORT` | `stdio` | MCP transport: `stdio`, `http`, or `sse` |

## Security

- Only read-only SQL prefixes are allowed (`SELECT`, `SHOW`, `DESCRIBE`, `WITH`, `EXPLAIN`).
- This is a basic guardrail; do not expose the MCP server to untrusted networks without additional controls.
- Prefer a dedicated read-only HDFS/Spark principal in production.

## Development

```bash
uv sync --group dev
uv run pytest
uv run ruff check src tests
```

## License

Apache License 2.0 — see [LICENSE](LICENSE).