Databricks MCP Server

# Databricks MCP Server A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with Databricks workspaces programmatically. This server provides tools for cluster management, notebook operations, job orchestration, Unity Catalog data governance, user management, permissions control, and FinOps cost analytics. ## Features ### 🖥️ Cluster Management - List, create, start, stop, and terminate clusters - Edit cluster configurations and resize clusters - Get cluster events and logs - Pin/unpin clusters for organization ### 📔 Notebook Management - List, create, read, and update notebooks - Execute notebooks with parameters - Import/export notebooks in multiple formats - Move and organize notebooks in workspace ### ⚙️ Job Management - Create and manage jobs with complex workflows - Schedule jobs with cron expressions - Trigger manual job runs - Monitor job execution and retrieve outputs - Configure notifications and alerts ### 👥 User & Group Management - List and manage workspace users - Create and manage groups - Add/remove users from groups - Manage service principals ### 🗄️ Unity Catalog - Manage catalogs, schemas, and tables - Query table metadata and statistics - List volumes and external locations - Execute SQL queries on catalog tables ### 🔒 Permissions & Access Control - Get and set permissions on workspace objects - Manage Unity Catalog grants and privileges - View effective permissions - Audit access control ### 💰 FinOps & Cost Analytics - Analyze cluster costs and DBU consumption - Track spending trends over time - Forecast future costs - Get cost optimization recommendations - Analyze costs by custom tags ## Installation ### Prerequisites - Node.js 18 or higher - npm or yarn - A Databricks workspace - Databricks personal access token or service principal credentials ### Install from npm ```bash npm install -g databricks-mcp-server ``` ### Install from source ```bash git clone https://github.com/yourusername/databricks-mcp-server.git cd databricks-mcp-server npm install npm run build ``` ## Configuration ### Environment Variables Create a `.env` file in your project directory: ```env DATABRICKS_HOST=https://your-workspace.cloud.databricks.com DATABRICKS_TOKEN=your-personal-access-token ``` For more configuration options, see [CONFIGURATION.md](./CONFIGURATION.md). ### Claude Desktop Configuration Add this to your Claude Desktop configuration file: **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` ```json { "mcpServers": { "databricks": { "command": "databricks-mcp-server", "env": { "DATABRICKS_HOST": "https://your-workspace.cloud.databricks.com", "DATABRICKS_TOKEN": "your-personal-access-token" } } } } ``` ## Quick Start Once configured, you can interact with your Databricks workspace through your AI assistant: ### Example Queries **Cluster Management:** - "List all running clusters in my workspace" - "Create a new cluster with 4 workers using the latest Spark version" - "Terminate the idle clusters to save costs" **Notebook Operations:** - "Show me all notebooks in the /Users/me/projects directory" - "Create a new Python notebook for data analysis" - "Run the ETL notebook with these parameters" **Job Management:** - "List all scheduled jobs" - "Create a daily job to run my data pipeline notebook" - "What's the status of job run 12345?" **Unity Catalog:** - "List all catalogs in my metastore" - "Show me the schema for the sales.customers table" - "Query the last 10 records from production.analytics.metrics" **Cost Analytics:** - "What are my cluster costs for the last 30 days?" - "Show me spending trends by week" - "Give me cost optimization recommendations" - "Forecast costs for the next month" ## Available Tools The server provides 60+ tools organized into 7 categories: ### Cluster Tools (11 tools) - `list_clusters` - List all clusters - `get_cluster_details` - Get detailed cluster info - `create_cluster` - Create a new cluster - `start_cluster` - Start a terminated cluster - `restart_cluster` - Restart a cluster - `terminate_cluster` - Terminate a cluster - `edit_cluster` - Edit cluster configuration - `resize_cluster` - Resize cluster workers - `get_cluster_events` - Get cluster event history - `pin_cluster` - Pin a cluster - `unpin_cluster` - Unpin a cluster ### Notebook Tools (8 tools) - `list_notebooks` - List notebooks in a path - `read_notebook` - Read notebook content - `create_notebook` - Create a new notebook - `update_notebook` - Update notebook content - `delete_notebook` - Delete a notebook - `move_notebook` - Move/rename a notebook - `create_directory` - Create workspace directory - `run_notebook` - Execute a notebook ### Job Tools (10 tools) - `list_jobs` - List all jobs - `get_job_details` - Get job configuration - `create_job` - Create a new job - `update_job` - Update job settings - `delete_job` - Delete a job - `run_job` - Trigger a job run - `get_job_run_status` - Check run status - `list_job_runs` - List job run history - `cancel_job_run` - Cancel a running job - `get_job_run_output` - Get job output ### User Tools (11 tools) - `list_users` - List all users - `get_user_details` - Get user information - `create_user` - Add a new user - `delete_user` - Remove a user - `list_groups` - List all groups - `get_group_details` - Get group info - `create_group` - Create a new group - `delete_group` - Delete a group - `add_user_to_group` - Add user to group - `remove_user_from_group` - Remove user from group - `list_service_principals` - List service principals ### Unity Catalog Tools (12 tools) - `list_catalogs` - List all catalogs - `get_catalog_details` - Get catalog info - `create_catalog` - Create a catalog - `delete_catalog` - Delete a catalog - `list_schemas` - List schemas in catalog - `get_schema_details` - Get schema info - `create_schema` - Create a schema - `delete_schema` - Delete a schema - `list_tables` - List tables in schema - `get_table_details` - Get table metadata - `delete_table` - Delete a table - `list_volumes` - List volumes - `list_external_locations` - List external locations - `query_table` - Execute SQL query ### Permission Tools (7 tools) - `get_permissions` - Get object permissions - `set_permissions` - Set object permissions - `update_permissions` - Update object permissions - `get_permission_levels` - Get available permission levels - `grant_unity_catalog_permissions` - Grant UC privileges - `revoke_unity_catalog_permissions` - Revoke UC privileges - `get_effective_permissions` - Get effective UC permissions ### FinOps Tools (5 tools) - `get_cluster_costs` - Analyze cluster costs - `analyze_spending_trends` - View spending trends - `forecast_costs` - Forecast future costs - `get_optimization_recommendations` - Get cost-saving recommendations - `analyze_cost_by_tag` - Analyze costs by tags For detailed documentation of all tools, see [API.md](./API.md). ## Development ### Building ```bash npm run build ``` ### Running in Development ```bash npm run dev ``` ### Project Structure ``` databricks-mcp-server/ ├── src/ │ ├── index.ts # MCP server entry point │ ├── databricks-client.ts # Databricks API client │ ├── tools/ │ │ ├── clusters.ts # Cluster management tools │ │ ├── notebooks.ts # Notebook management tools │ │ ├── jobs.ts # Job management tools │ │ ├── users.ts # User/group management tools │ │ ├── unity-catalog.ts # Unity Catalog tools │ │ ├── permissions.ts # Permission tools │ │ └── finops.ts # FinOps tools │ └── types/ │ └── databricks.ts # TypeScript type definitions ├── dist/ # Compiled output ├── package.json ├── tsconfig.json └── README.md ``` ## Examples See [EXAMPLES.md](./EXAMPLES.md) for detailed usage examples and common workflows. ## Authentication The server supports multiple authentication methods: 1. **Personal Access Token** (recommended for individual use) 2. **Service Principal OAuth** (recommended for production) 3. **Azure AD Token** (for Azure Databricks) See [CONFIGURATION.md](./CONFIGURATION.md) for detailed authentication setup. ## Security Best Practices - Store credentials in environment variables, never in code - Use service principals for production deployments - Implement least-privilege access for service accounts - Rotate tokens regularly - Use Unity Catalog for fine-grained data access control ## Limitations - Cost estimates in FinOps tools are approximations. For accurate billing, use the Databricks billing API or account console - Some operations require specific Databricks editions (Unity Catalog requires Premium/Enterprise) - Rate limiting is handled automatically with exponential backoff - SQL execution requires a running SQL warehouse ## Troubleshooting ### Common Issues **Authentication Failed** - Verify your DATABRICKS_HOST is correct and includes https:// - Check that your DATABRICKS_TOKEN is valid and not expired - Ensure your token has sufficient permissions **Tool Not Found** - Make sure the server is properly installed and configured - Check Claude Desktop configuration file syntax - Restart Claude Desktop after configuration changes **Permission Denied** - Verify your token has the required permissions - Check workspace access controls - Review Unity Catalog grants for data access For more troubleshooting help, see [CONFIGURATION.md](./CONFIGURATION.md). ## Contributing Contributions are welcome! Please feel free to submit a Pull Request. ## License MIT License - see LICENSE file for details ## Support For issues, questions, or contributions: - GitHub Issues: [Report an issue](https://github.com/yourusername/databricks-mcp-server/issues) - Documentation: [Full API Docs](./API.md) - Examples: [Usage Examples](./EXAMPLES.md) ## Related Projects - [Model Context Protocol](https://modelcontextprotocol.io) - [Databricks SDK](https://docs.databricks.com/dev-tools/sdk-python.html) - [Claude Desktop](https://claude.ai/desktop) ## Acknowledgments Built with the Model Context Protocol SDK by Anthropic.