Dataproc MCP Server

A Model Context Protocol (MCP) server that provides tools for managing Google Cloud Dataproc clusters and jobs. This server enables AI assistants to interact with Dataproc resources through a standardized interface.

Features

Cluster Management

List Clusters: View all clusters in a project and region
Create Cluster: Provision new Dataproc clusters with custom configurations
Delete Cluster: Remove existing clusters
Get Cluster: Retrieve detailed information about specific clusters

Job Management

Submit Jobs: Run Spark, PySpark, Spark SQL, Hive, Pig, and Hadoop jobs
List Jobs: View jobs across clusters with filtering options
Get Job: Retrieve detailed job information and status
Cancel Job: Stop running jobs

Batch Operations

Create Batch Jobs: Submit serverless Dataproc batch jobs
List Batch Jobs: View all batch jobs in a region
Get Batch Job: Retrieve detailed batch job information
Delete Batch Job: Remove batch jobs

Installation

Prerequisites

Python 3.11 or higher
Google Cloud SDK configured with appropriate permissions
Dataproc API enabled in your Google Cloud project

Install from Source

# Clone the repository
git clone <repository-url>
cd dataproc-mcp

# Install with uv (recommended)
uv pip install --system -e .

# Or install with pip
pip install -e .

# Install development dependencies
uv pip install --system -e ".[dev]"

Configuration

Authentication

The server supports multiple authentication methods:

Service Account Key (Recommended for production):
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
Application Default Credentials:
gcloud auth application-default login
Compute Engine Service Account (when running on GCE)

Required Permissions

Ensure your service account or user has the following IAM roles:

roles/dataproc.editor - For cluster and job management
roles/storage.objectViewer - For accessing job files in Cloud Storage
roles/compute.networkUser - For VPC network access (if using custom networks)

Usage

Running the Server

The server supports multiple transport protocols:

# STDIO (default) - for command-line tools and MCP clients
python -m dataproc_mcp_server

# HTTP - REST API over HTTP using streamable-http transport
DATAPROC_MCP_TRANSPORT=http python -m dataproc_mcp_server

# SSE - Server-Sent Events for real-time communication
DATAPROC_MCP_TRANSPORT=sse python -m dataproc_mcp_server

# Run with entry point script (STDIO only)
dataproc-mcp-server

Transport Configuration

STDIO (default): Standard input/output communication for command-line tools and MCP clients
HTTP: REST API over HTTP using streamable-http transport
- Server URL: http://localhost:8000/mcp
- Accessible via web clients and HTTP-based MCP clients
SSE: Server-Sent Events for real-time bidirectional communication
- Server URL: http://localhost:8000/sse
- Supports streaming responses and live updates

Environment Variables

# Transport type (stdio, http, sse)
export DATAPROC_MCP_TRANSPORT=http

# Server host (for HTTP/SSE transports)
export DATAPROC_MCP_HOST=0.0.0.0

# Server port (for HTTP/SSE transports)
export DATAPROC_MCP_PORT=8080

# Authentication
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

MCP Client Configuration

Add to your MCP client configuration:

{
  "mcpServers": {
    "dataproc": {
      "command": "python",
      "args": ["-m", "dataproc_mcp_server"],
      "env": {
        "GOOGLE_APPLICATION_CREDENTIALS": "/path/to/service-account.json"
      }
    }
  }
}

Testing with MCP Inspector

You can test the server using the official MCP Inspector:

# Test STDIO transport
npx @modelcontextprotocol/inspector python -m dataproc_mcp_server

# Test HTTP transport
DATAPROC_MCP_TRANSPORT=http python -m dataproc_mcp_server &
npx @modelcontextprotocol/inspector --transport http --server-url http://127.0.0.1:8000/mcp

# Test SSE transport  
DATAPROC_MCP_TRANSPORT=sse python -m dataproc_mcp_server &
npx @modelcontextprotocol/inspector --transport sse --server-url http://127.0.0.1:8000/sse

The MCP Inspector provides a web interface to:

Browse available tools and resources
Test tool calls with custom parameters
View real-time protocol messages
Debug server responses

Example Tool Usage

Create a Cluster

{
  "name": "create_cluster",
  "arguments": {
    "project_id": "my-project",
    "region": "us-central1",
    "cluster_name": "my-cluster",
    "num_instances": 3,
    "machine_type": "n1-standard-4",
    "disk_size_gb": 100,
    "image_version": "2.1-debian11"
  }
}

Submit a PySpark Job

{
  "name": "submit_job",
  "arguments": {
    "project_id": "my-project",
    "region": "us-central1", 
    "cluster_name": "my-cluster",
    "job_type": "pyspark",
    "main_file": "gs://my-bucket/my-script.py",
    "args": ["--input", "gs://my-bucket/input", "--output", "gs://my-bucket/output"],
    "properties": {
      "spark.executor.memory": "4g",
      "spark.executor.instances": "3"
    }
  }
}

Create a Batch Job

{
  "name": "create_batch_job",
  "arguments": {
    "project_id": "my-project",
    "region": "us-central1",
    "batch_id": "my-batch-job",
    "job_type": "pyspark",
    "main_file": "gs://my-bucket/batch-script.py",
    "service_account": "my-service-account@my-project.iam.gserviceaccount.com"
  }
}

Development

Setup Development Environment

# Install development dependencies
uv pip install --system -e ".[dev]"

# Or with pip
pip install -e ".[dev]"

Running Tests

# Run all tests
pytest

# Run with coverage
python -m pytest --cov=src/dataproc_mcp_server tests/

# Run specific test file
pytest tests/test_dataproc_client.py -v

Code Quality

# Format code
ruff format src/ tests/

# Lint code
ruff check src/ tests/

# Type checking (with VS Code + Pylance or mypy)
mypy src/

Project Structure

dataproc-mcp/
├── src/dataproc_mcp_server/
│   ├── __init__.py
│   ├── __main__.py           # Entry point
│   ├── server.py             # MCP server implementation
│   ├── dataproc_client.py    # Dataproc cluster/job operations
│   └── batch_client.py       # Dataproc batch operations
├── tests/
│   ├── __init__.py
│   ├── test_server.py
│   └── test_dataproc_client.py
├── examples/
│   ├── mcp_server_config.json
│   └── example_usage.py
├── pyproject.toml
├── CLAUDE.md                 # Development guide
└── README.md

Troubleshooting

Common Issues

Authentication Errors:
- Verify GOOGLE_APPLICATION_CREDENTIALS is set correctly
- Ensure service account has required permissions
- Check that Dataproc API is enabled
Network Errors:
- Verify VPC/subnet configurations for custom networks
- Check firewall rules for cluster communication
- Ensure clusters are in the correct region
Job Submission Failures:
- Verify file paths in Cloud Storage are accessible
- Check cluster has sufficient resources
- Validate job configuration parameters

Debug Mode

Enable debug logging:

export PYTHONPATH=/path/to/dataproc-mcp/src
python -c "
import logging
logging.basicConfig(level=logging.DEBUG)
from dataproc_mcp_server import __main__
import asyncio
asyncio.run(__main__.main())
"

API Reference

Tools

Cluster Management

list_clusters(project_id, region) - List all clusters
create_cluster(project_id, region, cluster_name, ...) - Create cluster
delete_cluster(project_id, region, cluster_name) - Delete cluster
get_cluster(project_id, region, cluster_name) - Get cluster details

Job Management

submit_job(project_id, region, cluster_name, job_type, main_file, ...) - Submit job
list_jobs(project_id, region, cluster_name?, job_states?) - List jobs
get_job(project_id, region, job_id) - Get job details
cancel_job(project_id, region, job_id) - Cancel job

Batch Operations

create_batch_job(project_id, region, batch_id, job_type, main_file, ...) - Create batch job
list_batch_jobs(project_id, region, page_size?) - List batch jobs
get_batch_job(project_id, region, batch_id) - Get batch job details
delete_batch_job(project_id, region, batch_id) - Delete batch job

Resources

dataproc://clusters - Access cluster information
dataproc://jobs - Access job information

Contributing

Fork the repository
Create a feature branch
Make your changes
Add tests for new functionality
Run the test suite and linting
Submit a pull request

License

MIT License - see LICENSE file for details.

Support

For issues and questions:

Check the troubleshooting section
Review Google Cloud Dataproc documentation
Open an issue in the repository

Deploy Server

HTTP connection URL

security - not tested

license - permissive license

quality - not tested

How are these scores calculated?

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

Tools

View all tools

Enables AI assistants to manage Google Cloud Dataproc clusters and jobs through a standardized interface. Supports cluster creation/deletion, job submission (Spark, PySpark, Hive, etc.), and serverless batch operations.

Related MCP Servers

GCP MCP
henihaddad
A
security
F
license
A
quality
Enables managing Google Cloud Platform resources through natural language commands in Claude Desktop, supporting comprehensive operations across compute, storage, databases, networking, monitoring, and IAM without manual credential setup.
Last updated -
56
21
GCP MCP Server
enesbol
-
security
A
license
-
quality
Enables AI assistants to interact with and manage Google Cloud Platform resources including Compute Engine, Cloud Run, Storage, BigQuery, and other GCP services through a standardized MCP interface.
Last updated -
4
MIT License
GCP MCP Server
JayRajGoyal
-
security
A
license
-
quality
Enables AI assistants to interact with Google Cloud Platform services for log analysis and root cause investigation. Provides tools to query Cloud Logging, detect error patterns, and perform real-time log streaming across multiple GCP projects.
Last updated -
MIT License
Databricks MCP Server
winkash
-
security
F
license
-
quality
Enables AI assistants like Claude to interact with Databricks workspaces through custom prompts and tools. Supports running SQL queries, managing clusters, creating jobs, and accessing workspace resources via the Databricks SDK.
Last updated -

View all related MCP servers