# databricks_ops_mcp

A governed MCP server exposing Databricks operations as tools — jobs orchestration,
SQL execution, notebook creation, Unity Catalog governance, lineage, clusters, and
DLT pipelines. Built as a reuse template: same pattern as a SharePoint/Jira MCP
server, pointed at the Databricks REST API.

## Tools (17)

| Category | Read-only | Write-gated |
|---|---|---|
| Jobs | list_jobs, get_run_status, list_failed_runs | run_job, repair_run |
| SQL | preview_table | execute_sql (DML/DDL only; SELECT is free) |
| Workspace | export_notebook, list_workspace | create_notebook |
| Unity Catalog | list_tables, get_table_grants, get_table_lineage | — |
| Compute/DLT | list_clusters, list_pipelines | restart_cluster, start_pipeline_update |

## Safety model

- **Dry-run by default**: every write tool returns a validated dry-run message until
  `DBX_WRITE_ENABLED=true`. Promote per environment (DEV=true, PROD=false + approval flow).
- **Audit log**: every tool call appends a JSONL record (timestamp, tool, payload).
- **Context safety**: results capped at `DBX_MAX_RESULT_ROWS` / `DBX_MAX_RESULT_CHARS`.
- **Read-only SQL enforcement**: statements must start with SELECT/SHOW/DESCRIBE/EXPLAIN/WITH
  unless writes are enabled; multi-statement submissions are rejected.
- **Placeholders only**: no real hostnames, principals, or credentials anywhere in code.
  Everything comes from environment variables.

## Setup

```bash
pip install -r requirements.txt
cp .env.example .env      # fill in host, token (PAT or OAuth M2M), warehouse ID
```

## Run

```bash
# Local (stdio) — for Claude Desktop / Claude Code
python server.py

# Remote (streamable HTTP) — for Databricks Apps / ECS / Lambda-backed deployments
MCP_TRANSPORT=http python server.py
```

### Claude Desktop config

```json
{
  "mcpServers": {
    "databricks_ops": {
      "command": "python",
      "args": ["/path/to/databricks_ops_mcp/server.py"],
      "env": {
        "DATABRICKS_HOST": "https://workspace_placeholder.cloud.databricks.com",
        "DATABRICKS_TOKEN": "token_placeholder",
        "DATABRICKS_WAREHOUSE_ID": "warehouse_id_placeholder",
        "DBX_WRITE_ENABLED": "false"
      }
    }
  }
}
```

### Claude Code

```bash
claude mcp add databricks_ops -- python /path/to/databricks_ops_mcp/server.py
```

### Bedrock / custom agents

Run with `MCP_TRANSPORT=http` behind your FastAPI gateway or as a Databricks App;
point the agent's MCP client at the streamable HTTP endpoint. For Databricks Apps
hosting, replace the bearer token with app OAuth (on-behalf-of user) so Unity
Catalog enforces per-user permissions.

## Example flows

- "List failed runs from today, get the status of the worst one, and repair it"
  → `list_failed_runs` → `get_run_status` → `repair_run` (approval-gated in PROD)
- "Create a notebook that deduplicates the claims feed and preview the target table"
  → `create_notebook` → `preview_table`
- "What feeds this table and who can modify it?"
  → `get_table_lineage` + `get_table_grants` (Conversational Data Steward core)

## Extension roadmap

1. `create_job_from_spec` — NL → metadata-driven pipeline config → Jobs API
2. Column-level lineage + cross-platform trace into MSTR cubes (second MCP server)
3. `deploy_bundle` via Asset Bundles for Git-first production deployment
4. Approval elicitation (`ctx.elicit`) on write tools instead of a global flag
5. Cost tools over `/api/2.0/usage` (billing usage export)

## Project layout

```
databricks_ops_mcp/
├── server.py            # FastMCP/MCPServer registration (SDK 1.x & 2.x compatible)
├── client.py            # Shared async REST client, errors, audit, truncation
├── config.py            # Env-driven settings and guardrails
├── tools/
│   ├── jobs.py          # /api/2.2/jobs/*
│   ├── sql.py           # /api/2.0/sql/statements (submit → poll → fetch)
│   ├── workspace.py     # /api/2.0/workspace/* (notebook import/export)
│   ├── unity_catalog.py # /api/2.1/unity-catalog/* + lineage-tracking
│   └── compute.py       # /api/2.1/clusters/*, /api/2.0/pipelines/*
├── requirements.txt
└── .env.example
```