Skip to main content
Glama
Sign In
PySpark MCP Server

PySpark MCP Server

by AnnasMazhar
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

PySpark MCP Server

SQL migration assistance, AWS Glue job template generation, and Spark code optimization — as an MCP server.

Not the live-Spark pyspark-mcp package. This project is SQL → PySpark / Glue source generation, published as pyspark-tools. SemyonSinchenko/pyspark-mcp introspects a running SparkSession. A deprecated pyspark-mcp console script remains here so old configs keep working; it prints a warning, then starts this server.

CI Pipeline Python 3.11+ License: MIT

What It Does

  • SQL Dialect Transpilation — Convert between PostgreSQL, Oracle, Redshift, MySQL, Snowflake, and Spark SQL using SQLGlot

  • PySpark DataFrame API Generation — Generate DataFrame API source text from SQL, with optimization hints

  • AWS Glue templates — Job script strings, DynamicFrame conversions, Data Catalog definitions, S3 layout advice

  • Batch Processing — Walk SQL files/directories and emit converted modules

  • Code Review & Optimization — Pattern-based review of existing PySpark source

  • Pattern Detection — Find duplicated snippets and suggest utilities

Related MCP server: mcp-server-sql-analyzer

What It Doesn't Do

  • Recursive CTEs → provides Spark SQL equivalent + guidance (PySpark has no native recursive CTE support)

  • MERGE/PIVOT/CONNECT BY → transpiles to Spark SQL, provides DataFrame API guidance

  • Perfect 1:1 DataFrame API transpilation for all SQL — complex queries get Spark SQL + recommendations

  • It does not start a SparkSession, submit Glue jobs, or execute SQL

  • optimize(mode="code") returns suggestions; it does not rewrite your code

  • glue_s3 is a path heuristic (no AWS call, no measured speedups)

  • It does not replace SemyonSinchenko/pyspark-mcp for live catalog/plans

Why this vs calling sqlglot yourself

SQLGlot already transpiles dialects. This MCP adds three things around that kernel: DataFrame-API pretty-printing with join/window/cast mappings that the conversion tests lock, Glue job boilerplate strings (bookmarks, DynamicFrames, catalog tables) so an agent can emit a file instead of assembling one, and a 14-tool FastMCP surface so an LLM picks convert / mode=sql instead of wiring sqlglot itself. If you only need sqlglot.transpile(...), use sqlglot.

Quick Start

pip install pyspark-tools
pyspark-tools

Zero-clone alternative: uvx pyspark-tools. run_server.py is a development convenience that inserts sys.path and prints startup banners. Prefer pyspark-tools in configs and production.

Try it

pip install pyspark-tools
python -c "from pathlib import Path; from pyspark_tools.sql_converter import SQLToPySparkConverter as C; from pyspark_tools.consolidated_tools import glue_job; c,s,o=C(),Path('examples'),Path('examples/out'); [(o/f'{n}.py').write_text(c.convert_sql_to_pyspark((s/f'{n}.sql').read_text(), dialect=d).pyspark_code) for n,d in [('postgres_orders','postgres'),('oracle_decode','oracle')]]; (o/'orders_etl_glue.py').write_text(glue_job(mode='template', job_name='orders_etl', sql_query=(s/'postgres_orders.sql').read_text())['template'])"

Writes the same files as examples/out/. MCP stdio CLI: pyspark-tools.

Example: SQL → PySpark

SELECT o.customer_id, c.name, SUM(o.amount) AS total
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'paid'
GROUP BY o.customer_id, c.name

Call convert with mode=sql. Captured converter output (dialect=spark):

from pyspark.sql import SparkSession
from pyspark.sql.functions import (
    col, lit, when, count, sum as spark_sum, avg, min, max, countDistinct,
    coalesce, concat, datediff, date_add, to_date,
    row_number, rank, lag, lead,
)
from pyspark.sql.window import Window

# Generated from SPARK SQL
spark = SparkSession.builder.appName('SQLToPySpark').getOrCreate()

# Load table: customers
customers_df = spark.table('customers')
# Load table: orders
orders_df = spark.table('orders')

# Main query
result_df = (orders_df.alias('o')
    .join(customers_df.alias('c'), (col('o.customer_id') == col('c.id')), 'inner')
    .filter((col('o.status') == lit('paid')))
    .groupBy(col('o.customer_id'), col('c.name'))
    .select(col('o.customer_id'), col('c.name'), (spark_sum(col('o.amount'))).alias('total')))

Exact output depends on dialect detection and fallbacks; conversion tests in tests/test_sql_conversion_fixes.py pin the important constructs. Notebook-style import * / show() is opt-in via style="notebook" on the converter.

MCP Configuration

Claude Desktop

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

Linux: ~/.config/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "pyspark": {
      "command": "pyspark-tools",
      "args": []
    }
  }
}

Hermes Agent

Add to ~/.hermes/config.yaml:

mcp:
  servers:
    pyspark:
      command: pyspark-tools
      enabled_tools: all

Docker

The image is stdio only (FastMCP over stdin/stdout). There is no HTTP server on port 8000. docker compose up is for local tests, not a health-checkable web service.

docker compose --profile test run --rm pyspark-tools-test

Tools

Three primary tools. The other eleven routers stay registered this minor version but are deprecated — prefer convert, glue_job, and review.

convert — SQL → PySpark (including mode=batch_dir)

convert(mode="sql", sql_query="SELECT id FROM users", dialect="postgres")
convert(mode="batch_dir", directory_path="etl/", output_dir="out")

glue_job — Glue 5.0 job template strings

glue_job(mode="template", job_name="orders_etl", sql_query="SELECT * FROM orders")

review — code review, patterns, duplicates

review(mode="code", code="df = spark.table('t')\ndf.collect()")

Legacy / deprecated: analyze, optimize, glue_schema, glue_s3, glue_data, refactor, search, context, batch_status, s3_source, analytics. Still callable; do not advertise to new agents.

Security

This MCP can read local files (SQL, TXT, PDF) and, if the [aws] extra is installed, list/read S3 with the host's default AWS credentials. File tools only allow paths under the process working directory (or an explicit base_path / FileHandler(base_directory=...)). That is not a sandbox.

Run the server under a restricted OS account. Do not point it at secrets directories. Do not attach AWS credentials with write access unless you intend S3 reads via s3_source / glue_s3. Optional extras:

pip install "pyspark-tools[aws]"    # boto3 for S3/Glue catalog helpers
pip install "pyspark-tools[spark]"  # pyspark — not required at runtime; generated code only

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Test
pytest tests/ -v --cov=pyspark_tools

# Format
black pyspark_tools tests
isort pyspark_tools tests

# Lint
flake8 pyspark_tools tests

Requires Python 3.11+ (matches the CI matrix).

Architecture

pyspark_tools/
├── server.py              # FastMCP server + helper implementations
├── consolidated_tools.py  # 14 @app.tool() routers
├── sql_converter.py       # SQLGlot-based transpilation + DataFrame API generation
├── aws_glue_integration.py # Glue job templates, DynamicFrame, Data Catalog
├── advanced_optimizer.py  # Performance analysis + optimization suggestions
├── batch_processor.py     # Concurrent file processing
├── code_reviewer.py       # PySpark code review patterns
├── duplicate_detector.py  # Code deduplication
├── data_source_analyzer.py # Data source analysis (optional boto3)
└── file_utils.py          # File I/O with allow-root checks

License

MIT — see LICENSE.

mcp-name: io.github.AnnasMazhar/pyspark-mcp

Available Tools

14 tools
analyticsA

Deprecated. Prefer review. Still registered this minor version.

Analytics on optimization effectiveness and usage patterns.

Modes:

optimization Get analytics on optimization effectiveness. Parameters: optimization_type, limit

usage Get usage statistics including conversion history and pattern stats. Parameters: limit

Inputs: mode (required), limit, optimization_type. Returns: object

analyzeA

Deprecated. Prefer convert / glue_job / review. Still registered this minor version.

Analyze SQL or PySpark code for context, data flow, or optimization opportunities.

Modes:

sql_context Analyze SQL context (schemas, tables, dialect, complexity). Parameters: sql_content or selected_text

data_flow Analyze data flow patterns in PySpark code. Parameters: pyspark_code (required), table_info

codebase Analyze a PySpark codebase directory for patterns and issues. Parameters: directory_path (required), include_optimization_suggestions, scan_depth

workspace Full workspace analysis including project structure. Parameters: sql_content or workspace_path, include_project_structure, workspace_name

Inputs: mode (required), scan_depth, table_info, sql_content, +7 more. Returns: object

batch_statusA

Deprecated. Prefer convert(mode="batch_dir"). Still registered this minor version.

Monitor and manage batch processing jobs.

Modes:

status Get the status of a specific batch job. Parameters: job_id (required)

cancel Cancel a running batch job. Parameters: job_id (required)

active List all currently active batch jobs. Parameters: none

recent List recent batch jobs. Parameters: limit, status

Inputs: mode (required), limit, job_id, status. Returns: object

contextA

Deprecated. Prefer convert. Still registered this minor version.

Store, retrieve, and work with SQL/PySpark conversion context.

Modes:

store Store additional context data for a conversion. Parameters: conversion_id (required), context_data (required)

get Retrieve stored context for a conversion. Parameters: conversion_id (required)

assist Real-time SQL assistance — analyze and convert as you edit. Parameters: sql_query or selected_text

Inputs: mode (required), sql_query, context_data, conversion_id, +1 more. Returns: object

convertB

Convert SQL to PySpark code or process SQL files in batch.

Modes:

sql Convert a single SQL query to PySpark code. Parameters: sql_query (required), table_info, dialect, optimization_level, include_glue_template, style (production | notebook), target (spark | glue)

batch_files Process multiple SQL files into PySpark. Parameters: file_paths (required list), output_dir, job_name

batch_dir Process all SQL files in a directory. Parameters: directory_path (required), output_dir, recursive, job_name

from_pdf Extract SQL from a PDF file and convert to PySpark. Parameters: pdf_path (required)

Inputs: mode (required), style, target, dialect, +10 more. Returns: object

glue_dataA

Deprecated. Prefer glue_job. Still registered this minor version.

Generate AWS Glue data processing jobs — incremental, CDC, bookmarks.

Modes:

incremental Generate Glue job with incremental processing and job bookmarking. Parameters: source_database (required), source_table (required), target_database (required), target_table (required), incremental_column (required), incremental_strategy, transformation_sql

cdc Generate Change Data Capture (CDC) Glue job. Parameters: source_database (required), source_table (required), target_database (required), target_table (required), cdc_column, cdc_strategy, primary_keys

bookmarks Generate job bookmark configuration for Glue jobs. Parameters: job_name (required), bookmark_strategy, transformation_context_keys

Inputs: mode (required), job_name, cdc_column, cdc_strategy, +10 more. Returns: object

glue_jobA

Generate and manage AWS Glue job configurations and templates.

Modes:

template Generate a complete AWS Glue job template. Parameters: sql_query, job_name, source_database, source_table, target_database, target_table, output_dir, source_format, target_format, include_bookmarking, template_type, script_name

dynamic_frame Convert PySpark DataFrame code to use DynamicFrames. Parameters: pyspark_code (required), source_database, source_table, target_database, target_table

properties Generate Glue job properties for AWS CLI/SDK/Terraform. Parameters: job_name (required), job_type, worker_type, number_of_workers, max_retries, timeout, glue_version, enable_continuous_logging, enable_metrics, enable_spark_ui

sql_conversion Generate a Glue job that includes SQL-to-PySpark conversion. Parameters: sql_query (required), job_name (required), source_database, source_table, target_database, target_table, source_format, target_format, include_bookmarking

Inputs: mode (required), timeout, job_name, job_type, +19 more. Returns: object

glue_s3A

Deprecated. Prefer glue_job. Still registered this minor version.

Analyze S3 data layouts using path heuristics (no AWS API call).

Modes:

analyze Path-heuristic layout suggestions. No AWS call; figures are not measured. Parameters: s3_location (required), database_name (required), table_name (required), data_format, query_patterns, data_size_gb

optimize Generate comprehensive S3 optimization strategy. Parameters: database_name (required), table_name (required), s3_location (required), data_format, target_file_size_mb, compression_type, enable_small_file_optimization, query_patterns

consolidate Generate Glue job for small files consolidation. Parameters: source_database (required), source_table (required), target_database (required), target_table (required), target_file_size_mb, consolidation_strategy

Inputs: mode (required), table_name, data_format, s3_location, +11 more. Returns: object

glue_schemaA

Deprecated. Prefer glue_job. Still registered this minor version.

Manage Glue Data Catalog schemas — detect, evolve, and define.

Modes:

detect Detect schema from sample data and generate table definition. Parameters: sample_data (required dict/list), table_name (required), infer_partitions

evolve Generate schema evolution strategy for handling schema changes. Parameters: current_columns (required), new_columns (required), merge_behavior, case_sensitive

catalog Generate AWS Glue Data Catalog table definition. Parameters: database_name (required), table_name (required), s3_location (required), data_format, columns, partition_keys, enable_schema_evolution

Inputs: mode (required), columns, table_name, data_format, +10 more. Returns: object

optimizeA

Deprecated. Prefer review. Still registered this minor version.

Optimize PySpark code and recommend performance improvements.

Modes:

code Return pattern-based suggestions for PySpark code. Does not rewrite the input. Parameters: code (required), optimization_level

joins Recommend join strategies based on estimated table sizes. Parameters: pyspark_code (required), table_info

partitioning Suggest optimal partitioning strategies. Parameters: pyspark_code (required), table_info

comprehensive Generate comprehensive optimization recommendations + performance estimates. Parameters: pyspark_code (required), table_info

Inputs: mode (required), code, table_info, pyspark_code, +1 more. Returns: object

refactorA

Deprecated. Prefer review. Still registered this minor version.

Refactor PySpark code and generate pipeline structures.

Modes:

patterns Refactor code by replacing duplicate patterns with utility function calls. Parameters: original_code (required), code_samples (required)

utilities Extract common utility functions from code patterns. Parameters: code_samples (required), patterns

pipeline Generate optimized PySpark data pipeline code or project structure. Parameters (pipeline): data_sources (required list), processing_requirements (required), target_format, include_monitoring Parameters (project): sql_content, workspace_name, workspace_path, output_dir, include_glue_template, dialect, include_batch_processing, include_visualization

Inputs: mode (required), dialect, patterns, output_dir, +12 more. Returns: object

reviewA

Review PySpark code for issues, patterns, and refactoring opportunities.

Modes:

code Review PySpark code for issues, best practices, and performance. Parameters: code (required), focus_areas

patterns Analyze code samples to discover common patterns. Parameters: code_samples (required list)

duplicates Detect duplicate patterns across code samples. Parameters: code_samples (required list)

Inputs: mode (required), code, focus_areas, code_samples. Returns: object

s3_sourceA

Deprecated. Prefer glue_job. Still registered this minor version.

Analyze S3 data sources and Delta tables.

Modes:

analyze Analyze S3 data source structure, format, and optimization opportunities. Parameters: s3_path (required), include_schema_inference

delta Analyze Delta table structure, properties, and optimization. Parameters: table_path (required), analyze_history

Inputs: mode (required), s3_path, table_path, analyze_history, +1 more. Returns: object

searchA

Deprecated. Prefer convert. Still registered this minor version.

Search stored conversions, code patterns, and context data.

Modes:

conversions Search previously converted SQL queries and history. Parameters: query, limit If query is empty, returns recent conversion history.

patterns Search stored code patterns by description or template. Parameters: query, limit, min_usage_count If query is empty, returns all stored patterns with min_usage_count.

context Retrieve stored conversion context. Parameters: conversion_id or key

Inputs: mode (required), key, limit, query, +2 more. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    Claude Code, inside the Godot 4 editorFlockbay puts an AI agent in the real Godot 4 editor: it writes GDScript, runs your game and fixes what breaks. Free, or bring your own Claude Code or Codex.flockbay.comAd

    TDQS

    B3.4/5.0

    Scored across 14 tools

    Disambiguation2/5

    Multiple tools have unclear boundaries: analyze, optimize, review, and refactor all target PySpark code analysis/improvement, while glue_s3, s3_source, glue_job, and glue_data overlap on Glue/S3 concerns. The deprecation notes help steer agents, but the large number of legacy tools still creates significant selection ambiguity.

    Naming Consistency2/5

    Tool naming mixes single-word verbs (convert, analyze, review, search), noun-style names (context, analytics), and compound names in inconsistent orders (glue_s3 vs s3_source, glue_job vs batch_status). Though all are lowercase snake_case, there is no predictable verb_noun or noun_noun convention across the set.

    Tool Count3/5

    Fourteen tools is a reasonable raw count, but 11 of them are explicitly deprecated, leaving only three actively preferred tools. The surface is bloated with redundant legacy tools that add noise without expanding genuine capability.

    Completeness4/5

    The core workflows—SQL-to-PySpark conversion, PySpark review, and Glue job generation—are well covered, with supporting capabilities for schemas, S3 layout analysis, batch processing, and context storage. Minor gaps exist, such as no non-deprecated tool for S3 source analysis or batch job status, but these are workable via the preferred tools.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Resources

    Related MCP Connectors

    Related MCP Servers

    • Claude MCP Data Engineer

      itsnikhile
      F
      license
      Not graded
      quality
      D
      maintenance
      Provides specialized tools for data engineering tasks like SQL formatting, dbt model generation, and Snowflake table creation. It enables users to analyze CSV data, validate pipeline configurations, and summarize ETL lineage through natural language.
      -

    • duckdb-iceberg-mcp

      ys1173
      A
      license
      A
      quality
      C
      maintenance
      Enables AI assistants to query Apache Iceberg tables on S3 via AWS Glue Data Catalog using DuckDB as the embedded query engine, supporting columnar Arrow reads with no data movement.
      4
      MIT