PySpark MCP Server
This server is an MCP toolset for SQL-to-PySpark conversion, AWS Glue job templating, and PySpark code review/optimization.
Convert SQL across dialects (PostgreSQL, Oracle, Redshift, MySQL, Snowflake, Spark SQL) into PySpark DataFrame API source code or Spark SQL, with optional optimization hints and notebook/production styles.
Process SQL files in batch via
convert(mode="batch_dir")orbatch_files, including PDF extraction of SQL.Generate AWS Glue 5.0 job template strings, DynamicFrame conversions, Glue job properties, and Data Catalog definitions.
Perform pattern-based code review of PySpark source (
review), including issue detection, common-pattern discovery, and duplicate snippet detection.Provide legacy/deprecated tools for deeper analysis:
analyze,optimize,refactor,search,context,batch_status,glue_schema,glue_s3,glue_data,s3_source, andanalytics.Support S3 read/list heuristics and optional AWS/Glue helpers if the
[aws]extra is installed, plus optional[spark]extras for generated code.Operate as a FastMCP stdio server, configurable with Claude Desktop, Hermes Agent, or Docker for local testing.
Allows converting SQL queries to and from MySQL dialect, and generating PySpark DataFrame API code from SQL.
Allows converting SQL queries to and from PostgreSQL dialect, and generating PySpark DataFrame API code from SQL.
Allows converting SQL queries to and from Snowflake dialect, and generating PySpark DataFrame API code from SQL.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@PySpark MCP ServerConvert this Redshift SQL to PySpark and generate a Glue job"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
PySpark MCP Server
SQL migration assistance, AWS Glue job template generation, and Spark code optimization — as an MCP server.
Not the live-Spark
pyspark-mcppackage. This project is SQL → PySpark / Glue source generation, published aspyspark-tools. SemyonSinchenko/pyspark-mcp introspects a running SparkSession. A deprecatedpyspark-mcpconsole script remains here so old configs keep working; it prints a warning, then starts this server.
What It Does
SQL Dialect Transpilation — Convert between PostgreSQL, Oracle, Redshift, MySQL, Snowflake, and Spark SQL using SQLGlot
PySpark DataFrame API Generation — Generate DataFrame API source text from SQL, with optimization hints
AWS Glue templates — Job script strings, DynamicFrame conversions, Data Catalog definitions, S3 layout advice
Batch Processing — Walk SQL files/directories and emit converted modules
Code Review & Optimization — Pattern-based review of existing PySpark source
Pattern Detection — Find duplicated snippets and suggest utilities
Related MCP server: mcp-server-sql-analyzer
What It Doesn't Do
Recursive CTEs → provides Spark SQL equivalent + guidance (PySpark has no native recursive CTE support)
MERGE/PIVOT/CONNECT BY → transpiles to Spark SQL, provides DataFrame API guidance
Perfect 1:1 DataFrame API transpilation for all SQL — complex queries get Spark SQL + recommendations
It does not start a SparkSession, submit Glue jobs, or execute SQL
optimize(mode="code")returns suggestions; it does not rewrite your codeglue_s3is a path heuristic (no AWS call, no measured speedups)It does not replace SemyonSinchenko/pyspark-mcp for live catalog/plans
Why this vs calling sqlglot yourself
SQLGlot already transpiles dialects. This MCP adds three things around that kernel: DataFrame-API pretty-printing with join/window/cast mappings that the conversion tests lock, Glue job boilerplate strings (bookmarks, DynamicFrames, catalog tables) so an agent can emit a file instead of assembling one, and a 14-tool FastMCP surface so an LLM picks convert / mode=sql instead of wiring sqlglot itself. If you only need sqlglot.transpile(...), use sqlglot.
Quick Start
pip install pyspark-tools
pyspark-toolsZero-clone alternative: uvx pyspark-tools. run_server.py is a development convenience that inserts sys.path and prints startup banners. Prefer pyspark-tools in configs and production.
Try it
pip install pyspark-tools
python -c "from pathlib import Path; from pyspark_tools.sql_converter import SQLToPySparkConverter as C; from pyspark_tools.consolidated_tools import glue_job; c,s,o=C(),Path('examples'),Path('examples/out'); [(o/f'{n}.py').write_text(c.convert_sql_to_pyspark((s/f'{n}.sql').read_text(), dialect=d).pyspark_code) for n,d in [('postgres_orders','postgres'),('oracle_decode','oracle')]]; (o/'orders_etl_glue.py').write_text(glue_job(mode='template', job_name='orders_etl', sql_query=(s/'postgres_orders.sql').read_text())['template'])"Writes the same files as examples/out/. MCP stdio CLI: pyspark-tools.
Example: SQL → PySpark
SELECT o.customer_id, c.name, SUM(o.amount) AS total
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'paid'
GROUP BY o.customer_id, c.nameCall convert with mode=sql. Captured converter output (dialect=spark):
from pyspark.sql import SparkSession
from pyspark.sql.functions import (
col, lit, when, count, sum as spark_sum, avg, min, max, countDistinct,
coalesce, concat, datediff, date_add, to_date,
row_number, rank, lag, lead,
)
from pyspark.sql.window import Window
# Generated from SPARK SQL
spark = SparkSession.builder.appName('SQLToPySpark').getOrCreate()
# Load table: customers
customers_df = spark.table('customers')
# Load table: orders
orders_df = spark.table('orders')
# Main query
result_df = (orders_df.alias('o')
.join(customers_df.alias('c'), (col('o.customer_id') == col('c.id')), 'inner')
.filter((col('o.status') == lit('paid')))
.groupBy(col('o.customer_id'), col('c.name'))
.select(col('o.customer_id'), col('c.name'), (spark_sum(col('o.amount'))).alias('total')))Exact output depends on dialect detection and fallbacks; conversion tests in tests/test_sql_conversion_fixes.py pin the important constructs. Notebook-style import * / show() is opt-in via style="notebook" on the converter.
MCP Configuration
Claude Desktop
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Linux: ~/.config/Claude/claude_desktop_config.json
{
"mcpServers": {
"pyspark": {
"command": "pyspark-tools",
"args": []
}
}
}Hermes Agent
Add to ~/.hermes/config.yaml:
mcp:
servers:
pyspark:
command: pyspark-tools
enabled_tools: allDocker
The image is stdio only (FastMCP over stdin/stdout). There is no HTTP server
on port 8000. docker compose up is for local tests, not a health-checkable
web service.
docker compose --profile test run --rm pyspark-tools-testTools
Three primary tools. The other eleven routers stay registered this minor
version but are deprecated — prefer convert, glue_job, and review.
convert — SQL → PySpark (including mode=batch_dir)
convert(mode="sql", sql_query="SELECT id FROM users", dialect="postgres")
convert(mode="batch_dir", directory_path="etl/", output_dir="out")glue_job — Glue 5.0 job template strings
glue_job(mode="template", job_name="orders_etl", sql_query="SELECT * FROM orders")review — code review, patterns, duplicates
review(mode="code", code="df = spark.table('t')\ndf.collect()")Legacy / deprecated: analyze, optimize, glue_schema, glue_s3,
glue_data, refactor, search, context, batch_status, s3_source,
analytics. Still callable; do not advertise to new agents.
Security
This MCP can read local files (SQL, TXT, PDF) and, if the [aws] extra is installed, list/read S3 with the host's default AWS credentials. File tools only allow paths under the process working directory (or an explicit base_path / FileHandler(base_directory=...)). That is not a sandbox.
Run the server under a restricted OS account. Do not point it at secrets directories. Do not attach AWS credentials with write access unless you intend S3 reads via s3_source / glue_s3. Optional extras:
pip install "pyspark-tools[aws]" # boto3 for S3/Glue catalog helpers
pip install "pyspark-tools[spark]" # pyspark — not required at runtime; generated code onlyDevelopment
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
# Test
pytest tests/ -v --cov=pyspark_tools
# Format
black pyspark_tools tests
isort pyspark_tools tests
# Lint
flake8 pyspark_tools testsRequires Python 3.11+ (matches the CI matrix).
Architecture
pyspark_tools/
├── server.py # FastMCP server + helper implementations
├── consolidated_tools.py # 14 @app.tool() routers
├── sql_converter.py # SQLGlot-based transpilation + DataFrame API generation
├── aws_glue_integration.py # Glue job templates, DynamicFrame, Data Catalog
├── advanced_optimizer.py # Performance analysis + optimization suggestions
├── batch_processor.py # Concurrent file processing
├── code_reviewer.py # PySpark code review patterns
├── duplicate_detector.py # Code deduplication
├── data_source_analyzer.py # Data source analysis (optional boto3)
└── file_utils.py # File I/O with allow-root checksLicense
MIT — see LICENSE.
mcp-name: io.github.AnnasMazhar/pyspark-mcp
Available Tools
14 toolsanalyticsA
Deprecated. Prefer review. Still registered this minor version.
Analytics on optimization effectiveness and usage patterns.
Modes:
optimization
Get analytics on optimization effectiveness.
Parameters: optimization_type, limit
usage
Get usage statistics including conversion history and pattern stats.
Parameters: limit
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| limit | No | ||
| optimization_type | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral disclosure burden. It discloses deprecation, defines two behavioral modes, and lists associated parameters. However, it doesn't mention side effects, error behavior, or whether this is purely read-only, though the analytics nature implies a query operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured, with the deprecation warning front-loaded and modes presented clearly. There is minor redundancy between the opening line and the mode descriptions, but no wasted words overall.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the output schema exists, return values don't need explanation. The description covers modes, deprecation, and parameter mapping. It remains incomplete around parameter semantics and when exactly to fall back to this tool versus 'review', leaving some ambiguity for invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It maps 'optimization_type' and 'limit' to the optimization mode and 'limit' to the usage mode, which adds useful structure beyond the schema. However, it doesn't explain acceptable values for 'optimization_type' or the precise meaning of 'limit' beyond the schema's type and default.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear purpose: providing analytics on optimization effectiveness and usage patterns. It further clarifies by defining two distinct modes, 'optimization' and 'usage', with explicit outputs for each. While the deprecation note adds context, it doesn't obscure what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Prefer review', giving the agent a clear alternative and strongly signaling not to use this tool unless necessary. The phrase 'Still registered this minor version' implies a fallback/backward-compatibility role. It doesn't detail conditional cases, but the deprecation guidance is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
analyzeA
Deprecated. Prefer convert / glue_job / review. Still registered this minor version.
Analyze SQL or PySpark code for context, data flow, or optimization opportunities.
Modes:
sql_context
Analyze SQL context (schemas, tables, dialect, complexity).
Parameters: sql_content or selected_text
data_flow
Analyze data flow patterns in PySpark code.
Parameters: pyspark_code (required), table_info
codebase
Analyze a PySpark codebase directory for patterns and issues.
Parameters: directory_path (required), include_optimization_suggestions,
scan_depth
workspace
Full workspace analysis including project structure.
Parameters: sql_content or workspace_path, include_project_structure,
workspace_name
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| scan_depth | No | ||
| table_info | No | ||
| sql_content | No | ||
| pyspark_code | No | ||
| selected_text | No | ||
| directory_path | No | ||
| workspace_name | No | ||
| workspace_path | No | ||
| include_project_structure | No | ||
| include_optimization_suggestions | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral disclosure burden: it clearly marks the tool as deprecated and describes what each mode analyzes. It does not disclose operational constraints or response behavior, though the output schema exists and the analysis-focused verbs imply read-only behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with bold headings and compact mode blocks; the deprecation warning is front-loaded and every listed parameter relates to a specific mode. There is no filler or redundant restatement of the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mode-based tool with 11 parameters and no annotation coverage, the description provides enough per-mode detail to invoke it correctly, and the output schema covers the return shape. It could be more complete by explaining optional parameter meanings and the relationship to preferred replacement tools, but the core calling contract is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description compensates by mapping every mode to its relevant parameters and marking required ones, such as pyspark_code for data_flow and directory_path for codebase. It stops short of defining the semantics of scan_depth, table_info, or workspace_name, but the mode-to-parameter mapping is the most critical missing guidance.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool analyzes SQL or PySpark code and enumerates four distinct modes: sql_context, data_flow, codebase, and workspace. However, it does not meaningfully differentiate analyze from its preferred siblings beyond a bare deprecation pointer, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The deprecation notice advises preferring convert/glue_job/review, and each mode lists its parameters, giving a rough sense of when a mode applies. There is no explicit when-to-use versus when-not-to-use guidance or conditions for choosing analyze over the preferred replacement tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_statusA
Deprecated. Prefer convert(mode="batch_dir"). Still registered this minor version.
Monitor and manage batch processing jobs.
Modes:
status
Get the status of a specific batch job.
Parameters: job_id (required)
cancel
Cancel a running batch job.
Parameters: job_id (required)
active
List all currently active batch jobs.
Parameters: none
recent
List recent batch jobs.
Parameters: limit, status
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| limit | No | ||
| job_id | No | ||
| status | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavior. It discloses deprecation status and the supported operations, and the cancel mode implies mutation, but it does not describe side effects, reversibility, permissions, or behavior beyond the mode names. Some added context exists, but not deep behavioral transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The deprecation warning is front-loaded, followed by a terse mode-by-mode breakdown. The markdown structure makes the conditional parameter requirements easy to scan, and there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers all modes and their parameters, deprecation, and the preferred alternative. The output schema exists, so return values need not be spelled out, but the status filter values and mode-specific edge behavior are not specified, leaving minor gaps for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema descriptions are absent (0% coverage), but the description compensates by mapping each mode to its relevant parameters and marking job_id as required for status and cancel. It does not define allowed values for status or clarify limit semantics, so it is helpful but not exhaustive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool as managing batch processing jobs, then enumerates four distinct modes (status, cancel, active, recent), each with a specific action and resource. It also differentiates itself from its sibling convert by stating that convert(mode="batch_dir") is preferred.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly marks the tool as Deprecated, instructs the agent to prefer convert(mode="batch_dir"), and notes it is still registered this minor version. This gives a clear when-not-to-use signal and names the alternative, while the mode list clarifies internal usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
contextA
Deprecated. Prefer convert. Still registered this minor version.
Store, retrieve, and work with SQL/PySpark conversion context.
Modes:
store
Store additional context data for a conversion.
Parameters: conversion_id (required), context_data (required)
get
Retrieve stored context for a conversion.
Parameters: conversion_id (required)
assist
Real-time SQL assistance — analyze and convert as you edit.
Parameters: sql_query or selected_text
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| sql_query | No | ||
| context_data | No | ||
| conversion_id | No | ||
| selected_text | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral burden. It does disclose deprecation status, the real-time behavior of the 'assist' mode, and the purpose of each mode. However, it does not mention side effects, permissions, data mutation, or error behavior, which is a notable gap for a tool with no annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized with the deprecation warning front-loaded and modes broken into clear labeled sections. The mode/parameter layout is easy to scan. The only minor issue is the vague 'work with' phrase in the overview, which adds little information, but overall the structure is efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has an output schema and the description covers modes, required parameters, and deprecation guidance, an agent can select and invoke each mode correctly. It lacks examples of context_data structure and a fuller explanation of what 'assist' returns, but for a deprecated compatibility tool with an output schema, the coverage is reasonably complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does by tying each parameter to a specific mode and marking required ones (e.g., conversion_id and context_data for store, sql_query or selected_text for assist). This adds meaning beyond the raw schema, although it does not describe the shape or expected format of context_data.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool is deprecated and that it stores, retrieves, and works with SQL/PySpark conversion context, then enumerates three distinct modes (store, get, assist). Each mode has a concrete verb and resource, so an agent can generally tell what the tool does. The phrase 'work with' in the overview is vague, but the mode list resolves most ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description opens with 'Deprecated. Prefer convert. Still registered this minor version,' which is explicit routing guidance away from this tool to a sibling. It clearly tells the agent when not to use it, though it does not describe under what conditions, if any, this tool should still be chosen over convert.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
convertB
Convert SQL to PySpark code or process SQL files in batch.
Modes:
sql
Convert a single SQL query to PySpark code.
Parameters: sql_query (required), table_info, dialect,
optimization_level, include_glue_template, style
(production | notebook), target (spark | glue)
batch_files
Process multiple SQL files into PySpark.
Parameters: file_paths (required list), output_dir, job_name
batch_dir
Process all SQL files in a directory.
Parameters: directory_path (required), output_dir, recursive,
job_name
from_pdf
Extract SQL from a PDF file and convert to PySpark.
Parameters: pdf_path (required)
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| style | No | production | |
| target | No | spark | |
| dialect | No | ||
| job_name | No | ||
| pdf_path | No | ||
| recursive | No | ||
| sql_query | No | ||
| file_paths | No | ||
| output_dir | No | ||
| table_info | No | ||
| directory_path | No | ||
| optimization_level | No | standard | |
| include_glue_template | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It explains what each mode does at a high level, but it does not disclose side effects such as creating files, overwriting, directory scanning behavior, error conditions, or performance characteristics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized with a clear opening purpose and structured mode sections. It is longer than average but the length is justified by the four distinct modes and their parameter lists.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (14 parameters, four modes), the description does a solid job of laying out mode-specific required parameters and basic semantics. It is not fully complete because several parameter meanings and behavioral outcomes are left implicit, but it is sufficient for basic invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It lists mode-specific parameters and some allowed values (style: production|notebook, target: spark|glue), but it does not explain the meaning of parameters like table_info, optimization_level, include_glue_template, output_dir, or recursive, leaving many parameters underspecified.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool converts SQL to PySpark code and supports batch file/directory processing and PDF extraction. It distinguishes itself by specific action and resources, though it does not explicitly name sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description defines four distinct modes with their required parameters, which gives an agent enough context to select a mode. However, it does not explicitly state when to use this tool over siblings like analyze, optimize, or refactor, nor does it offer exclusion guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
glue_dataA
Deprecated. Prefer glue_job. Still registered this minor version.
Generate AWS Glue data processing jobs — incremental, CDC, bookmarks.
Modes:
incremental
Generate Glue job with incremental processing and job bookmarking.
Parameters: source_database (required), source_table (required),
target_database (required), target_table (required),
incremental_column (required), incremental_strategy,
transformation_sql
cdc
Generate Change Data Capture (CDC) Glue job.
Parameters: source_database (required), source_table (required),
target_database (required), target_table (required),
cdc_column, cdc_strategy, primary_keys
bookmarks
Generate job bookmark configuration for Glue jobs.
Parameters: job_name (required), bookmark_strategy,
transformation_context_keys
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| job_name | No | ||
| cdc_column | No | last_modified | |
| cdc_strategy | No | upsert | |
| primary_keys | No | ||
| source_table | No | ||
| target_table | No | ||
| source_database | No | ||
| target_database | No | ||
| bookmark_strategy | No | enable | |
| incremental_column | No | ||
| transformation_sql | No | ||
| incremental_strategy | No | timestamp | |
| transformation_context_keys | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries full behavioral burden. It implies job generation but does not disclose side effects, resource provisioning, permissions, idempotency, or failure behavior. The deprecation warning adds some context, but the safety and mutation profile of the operation remains unclear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is moderately long but well-structured with mode headers and parameter lists. The deprecation warning is front-loaded, and each section conveys necessary mode-specific information, though some repetition across modes could be tightened.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 14 parameters, no annotations, and no schema descriptions, so the description carries a heavy burden. It covers mode-specific required parameters well and an output schema exists, but it leaves strategy value semantics and operational behavior unexplained. This is adequate but with notable gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does meaningfully map parameters to each mode and marks required ones, which is valuable. However, it does not explain the meaning or accepted values of parameters like incremental_strategy, cdc_strategy, bookmark_strategy, or transformation_sql beyond their names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Generate AWS Glue data processing jobs — incremental, CDC, bookmarks,' naming the specific action and resource. The opening 'Deprecated. Prefer glue_job' explicitly distinguishes it from its sibling, so an agent can tell it apart immediately.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Deprecated. Prefer glue_job,' giving a direct when-not-to-use instruction and naming the alternative. The mode breakdown further tells the agent which mode to select for incremental, CDC, or bookmark needs, though it does not deeply elaborate on conditions beyond deprecation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
glue_jobA
Generate and manage AWS Glue job configurations and templates.
Modes:
template
Generate a complete AWS Glue job template.
Parameters: sql_query, job_name, source_database, source_table,
target_database, target_table, output_dir, source_format,
target_format, include_bookmarking, template_type, script_name
dynamic_frame
Convert PySpark DataFrame code to use DynamicFrames.
Parameters: pyspark_code (required), source_database, source_table,
target_database, target_table
properties
Generate Glue job properties for AWS CLI/SDK/Terraform.
Parameters: job_name (required), job_type, worker_type,
number_of_workers, max_retries, timeout, glue_version,
enable_continuous_logging, enable_metrics, enable_spark_ui
sql_conversion
Generate a Glue job that includes SQL-to-PySpark conversion.
Parameters: sql_query (required), job_name (required),
source_database, source_table, target_database, target_table,
source_format, target_format, include_bookmarking
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| timeout | No | ||
| job_name | No | ||
| job_type | No | etl | |
| sql_query | No | ||
| output_dir | No | ||
| max_retries | No | ||
| script_name | No | ||
| worker_type | No | G.1X | |
| glue_version | No | 5.0 | |
| pyspark_code | No | ||
| source_table | No | ||
| target_table | No | ||
| source_format | No | parquet | |
| target_format | No | parquet | |
| template_type | No | standard | |
| enable_metrics | No | ||
| enable_spark_ui | No | ||
| source_database | No | ||
| target_database | No | ||
| number_of_workers | No | ||
| include_bookmarking | No | ||
| enable_continuous_logging | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly describes what each mode generates, but it does not mention side effects, credentials, filesystem writes, or whether 'manage' implies modifying existing AWS resources. The behavior is not misleading, but it is incomplete for a tool with no annotation safety hints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a one-sentence summary followed by clearly formatted mode sections and parameter lists. It is longer than average, but every section earns its place given the tool's four distinct behaviors, and the front-loaded summary helps an agent orient immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's high complexity, 23 parameters, and no annotations, the description does a solid job covering all modes and associating parameters appropriately. The presence of an output schema reduces the need to explain return values. Missing details like allowed values for template_type or source_format and mode-specific examples prevent a perfect score.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does by grouping all 23 parameters into their relevant modes and marking required parameters such as pyspark_code and job_name. It adds meaning beyond the bare schema, though it still relies on parameter names to carry most semantic weight rather than describing values or constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the resource ('AWS Glue job configurations and templates') and the action ('Generate and manage'), and it enumerates four distinct modes that make the tool's scope concrete. It does not explicitly contrast with sibling tools like glue_schema or glue_s3, so it misses the top tier, but the purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The mode breakdown gives clear context for which mode to choose: template, dynamic_frame, properties, or sql_conversion, each with its own parameter list. However, there is no explicit guidance on when glue_job should be preferred over sibling tools or when a mode should not be used, so usage guidance remains implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
glue_s3A
Deprecated. Prefer glue_job. Still registered this minor version.
Analyze S3 data layouts using path heuristics (no AWS API call).
Modes:
analyze
Path-heuristic layout suggestions. No AWS call; figures are not measured.
Parameters: s3_location (required), database_name (required),
table_name (required), data_format, query_patterns, data_size_gb
optimize
Generate comprehensive S3 optimization strategy.
Parameters: database_name (required), table_name (required),
s3_location (required), data_format, target_file_size_mb,
compression_type, enable_small_file_optimization, query_patterns
consolidate
Generate Glue job for small files consolidation.
Parameters: source_database (required), source_table (required),
target_database (required), target_table (required),
target_file_size_mb, consolidation_strategy
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| table_name | No | ||
| data_format | No | parquet | |
| s3_location | No | ||
| data_size_gb | No | ||
| source_table | No | ||
| target_table | No | ||
| database_name | No | ||
| query_patterns | No | ||
| source_database | No | ||
| target_database | No | ||
| compression_type | No | snappy | |
| target_file_size_mb | No | ||
| consolidation_strategy | No | coalesce | |
| enable_small_file_optimization | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full behavioral burden, and it does well: it explains that path heuristics are used, that no AWS API call is made, and that figures are not measured. It also says the consolidate mode 'Generate[s] Glue job' rather than executes one, but it does not fully clarify whether any side effects or external changes occur, leaving some ambiguity around resource mutation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the deprecation warning, followed by a one-line summary and clearly separated mode sections. The mode parameter lists are somewhat repetitive but serve a functional purpose because the schema alone does not indicate which parameters apply to which mode.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex 15-parameter, multi-mode tool, the description covers the primary context: deprecation, mode purposes, required parameters, and behavioral constraints like 'no AWS API call.' The output schema exists, so return-value documentation is not required here, but per-parameter semantics and side-effect clarity remain partially underserved.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description must compensate. It does add value by grouping parameters by mode and marking which are required per mode, which is not evident from the schema's sole 'mode' requirement. However, it does not explain the meaning of many parameters, such as query_patterns, data_size_gb, consolidation_strategy, or enable_small_file_optimization, relying mostly on self-explanatory names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear deprecation notice and explicit preferred alternative ('Prefer glue_job'), then states exactly what the tool does: 'Analyze S3 data layouts using path heuristics (no AWS API call).' The mode sections further define specific behaviors — analyze, optimize, consolidate — so an agent can distinguish this tool from siblings like glue_job, glue_schema, and glue_data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells the agent to prefer glue_job instead and notes that glue_s3 is 'Still registered this minor version,' which is clear when-not-to-use guidance. The mode breakdown also provides situational guidance for when each mode is appropriate, such as path-heuristic suggestions versus generating a Glue job for small-file consolidation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
glue_schemaA
Deprecated. Prefer glue_job. Still registered this minor version.
Manage Glue Data Catalog schemas — detect, evolve, and define.
Modes:
detect
Detect schema from sample data and generate table definition.
Parameters: sample_data (required dict/list), table_name (required),
infer_partitions
evolve
Generate schema evolution strategy for handling schema changes.
Parameters: current_columns (required), new_columns (required),
merge_behavior, case_sensitive
catalog
Generate AWS Glue Data Catalog table definition.
Parameters: database_name (required), table_name (required),
s3_location (required), data_format, columns, partition_keys,
enable_schema_evolution
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| columns | No | ||
| table_name | No | ||
| data_format | No | parquet | |
| new_columns | No | ||
| s3_location | No | ||
| sample_data | No | ||
| database_name | No | ||
| case_sensitive | No | ||
| merge_behavior | No | merge | |
| partition_keys | No | ||
| current_columns | No | ||
| infer_partitions | No | ||
| enable_schema_evolution | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosing side effects, permissions, or mutation risk. It says 'Manage' and lists actions like evolve/define, but never states whether catalog schemas are modified, whether operations are reversible, or what side effects occur. The deprecation note adds context but not behavioral transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a deprecation banner, one-line summary, and bold mode subsections that each list only relevant parameters. There is no filler; even the 'Still registered this minor version' line adds availability context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given output schema exists and all 14 parameters are covered by mode-based breakdowns, an agent has enough to invoke the tool correctly. Missing is any statement about side effects or exact mode string values, but the mode headings make those inferable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully compensates by mapping every parameter to its mode, indicating which are required, and adding type guidance ('dict/list') not present in the schema. This is far more informative than the bare property names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear deprecation warning and a preferred alternative, then identifies the resource (Glue Data Catalog schemas) and three specific actions with concrete outcomes: detect schema and generate table definition, generate evolution strategy, and generate catalog table definition. This distinguishes it from siblings like glue_job and glue_s3 by naming the exact domain and outputs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says 'Deprecated. Prefer glue_job', which tells an agent not to use this tool when a current alternative exists. Mode-specific parameter lists give contextual usage for each mode, though it does not state conditions under which glue_schema should still be used despite the deprecation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
optimizeA
Deprecated. Prefer review. Still registered this minor version.
Optimize PySpark code and recommend performance improvements.
Modes:
code
Return pattern-based suggestions for PySpark code. Does not rewrite
the input. Parameters: code (required), optimization_level
joins
Recommend join strategies based on estimated table sizes.
Parameters: pyspark_code (required), table_info
partitioning
Suggest optimal partitioning strategies.
Parameters: pyspark_code (required), table_info
comprehensive
Generate comprehensive optimization recommendations + performance estimates.
Parameters: pyspark_code (required), table_info
| Name | Required | Description | Default |
|---|---|---|---|
| code | No | ||
| mode | Yes | ||
| table_info | No | ||
| pyspark_code | No | ||
| optimization_level | No | standard |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that the code mode 'does not rewrite the input' and that outputs are recommendations/estimates, which is useful. However, it does not state whether the other modes have side effects, what happens with invalid inputs, or any environment prerequisites such as a running Spark session.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The deprecation warning is front-loaded, followed by a one-sentence purpose and compact mode bullets. The repeated parameter lists are a minor redundancy but every section earns its place and the format is easy for an agent to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema covers return values, so the description does not need to describe them. It covers deprecation, modes, and mode-specific parameters, but misses allowed values for optimization_level and the structure of table_info; for a 5-parameter tool with no annotations, this is an incomplete picture.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It adds a useful mode-to-parameter mapping and distinguishes code from pyspark_code by mode. It does not define allowed values for optimization_level or the expected shape of table_info, leaving important invocation details unresolved.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a deprecation notice and then states 'Optimize PySpark code and recommend performance improvements,' a specific verb and resource. The mode list further clarifies what each invocation produces. It identifies review as preferred but does not explain how review differs functionally, so it stops short of perfect sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says 'Deprecated. Prefer review,' which is a direct when-not-to-use directive with a named alternative. Each mode section also states the exact parameters required for that mode, giving clear invocation guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
refactorA
Deprecated. Prefer review. Still registered this minor version.
Refactor PySpark code and generate pipeline structures.
Modes:
patterns
Refactor code by replacing duplicate patterns with utility function calls.
Parameters: original_code (required), code_samples (required)
utilities
Extract common utility functions from code patterns.
Parameters: code_samples (required), patterns
pipeline
Generate optimized PySpark data pipeline code or project structure.
Parameters (pipeline): data_sources (required list), processing_requirements
(required), target_format, include_monitoring
Parameters (project): sql_content, workspace_name, workspace_path,
output_dir, include_glue_template, dialect, include_batch_processing,
include_visualization
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| dialect | No | ||
| patterns | No | ||
| output_dir | No | ||
| sql_content | No | ||
| code_samples | No | ||
| data_sources | No | ||
| original_code | No | ||
| target_format | No | delta | |
| workspace_name | No | ||
| workspace_path | No | ||
| include_monitoring | No | ||
| include_glue_template | No | ||
| include_visualization | No | ||
| processing_requirements | No | ||
| include_batch_processing | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure, and it does so well by explaining what each mode does: replacing duplicate patterns, extracting utilities, and generating pipeline or project code. It also discloses the deprecation status and that the tool remains registered. It does not mention side effects or return behavior, but the described behaviors are sufficiently concrete for an agent to anticipate what will happen.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the most important decision-related information: deprecation and the preferred alternative. It then uses a clear mode-based structure with bullet-style parameter lists. Despite its length, the size is justified by 16 parameters and three modes, and every section contributes actionable guidance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's high complexity—16 parameters, three modes, no annotations, and zero schema description coverage—the description is quite complete: it identifies all modes, their purposes, required parameters, and the preferred replacement. It does not explicitly restate the 'mode' parameter as a field in the schema, but the mode names are clearly used as the expected values. Since an output schema exists, the lack of return-format details is acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does by mapping parameters to specific modes and marking which are required. It adds meaning beyond the bare JSON schema by grouping parameters under 'patterns,' 'utilities,' 'pipeline,' and 'project' contexts, which the schema alone does not convey. Some parameter semantics remain implicit from names, but the mode-based organization materially improves parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear deprecation notice and names the preferred alternative, review, which immediately distinguishes this tool from its siblings. It then states the core action ('Refactor PySpark code and generate pipeline structures') and enumerates three specific modes with distinct behaviors, so an agent can tell what the tool does and how it differs from related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Deprecated. Prefer review. Still registered this minor version,' giving an unambiguous when-not-to-use instruction and naming the alternative. Within the modes, it lists mode-specific required parameters, which further guides invocation by telling the agent which parameters to supply for each mode.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
reviewA
Review PySpark code for issues, patterns, and refactoring opportunities.
Modes:
code
Review PySpark code for issues, best practices, and performance.
Parameters: code (required), focus_areas
patterns
Analyze code samples to discover common patterns.
Parameters: code_samples (required list)
duplicates
Detect duplicate patterns across code samples.
Parameters: code_samples (required list)
| Name | Required | Description | Default |
|---|---|---|---|
| code | No | ||
| mode | Yes | ||
| focus_areas | No | ||
| code_samples | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must carry the full burden. It describes the modes but does not explicitly state side effects, permissions, or that it is read-only. Given the output schema exists, some behavioral context is implied but not explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured with clear headers and backticks for modes. Every sentence adds value, and the main purpose is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the output schema exists, return values need not be explained. The description covers the three modes and parameter dependencies. It could mention limitations or prerequisites, but overall it is complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description compensates by explaining the role of each parameter per mode (e.g., 'code' required in code mode, 'code_samples' in patterns/duplicates). This adds significant meaning beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool reviews PySpark code for issues, patterns, and refactoring opportunities. It differentiates three modes, but does not explicitly distinguish it from the sibling tool 'analyze', which could be seen as similar.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit modes and the parameters required for each, guiding when to use each mode. However, it does not specify when not to use this tool or mention alternative sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
s3_sourceA
Deprecated. Prefer glue_job. Still registered this minor version.
Analyze S3 data sources and Delta tables.
Modes:
analyze
Analyze S3 data source structure, format, and optimization opportunities.
Parameters: s3_path (required), include_schema_inference
delta
Analyze Delta table structure, properties, and optimization.
Parameters: table_path (required), analyze_history
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| s3_path | No | ||
| table_path | No | ||
| analyze_history | No | ||
| include_schema_inference | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the behavioral burden. The verb 'Analyze' and phrases like 'Analyze S3 data source structure, format, and optimization opportunities' signal a read-only analysis operation. The deprecation status is also a useful behavioral disclosure. It does not explicitly state that no data is modified or mention permission requirements, but the analysis framing makes the safety profile reasonably clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: a deprecation banner followed by two short mode bullets. Every sentence adds useful information, and there is no redundant repetition of schema defaults or obvious filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists, the description does not need to describe return values. It covers the two modes, their parameters, and the preferred alternative. The only notable gap is the lack of an explicit statement about side effects or access requirements, but for a read-oriented analysis tool this is a minor omission.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does by mapping parameters to modes: 's3_path (required), include_schema_inference' and 'table_path (required), analyze_history.' It also clarifies conditional requiredness that the flat schema cannot express. The boolean parameters are named but not elaborated, though their purposes are fairly self-evident from their names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly opens with a deprecation notice and then states the tool's purpose: 'Analyze S3 data sources and Delta tables.' It enumerates two specific modes, 'analyze' and 'delta', each with its own resource target, making the tool's scope explicit and distinguishable from siblings like glue_job or glue_s3.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly instructs agents to prefer glue_job: 'Deprecated. Prefer glue_job. Still registered this minor version.' It also gives mode-specific guidance by listing which parameters apply to which mode, so an agent can decide when to call this tool and with what arguments.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
searchA
Deprecated. Prefer convert. Still registered this minor version.
Search stored conversions, code patterns, and context data.
Modes:
conversions
Search previously converted SQL queries and history.
Parameters: query, limit
If query is empty, returns recent conversion history.
patterns
Search stored code patterns by description or template.
Parameters: query, limit, min_usage_count
If query is empty, returns all stored patterns with min_usage_count.
context
Retrieve stored conversion context.
Parameters: conversion_id or key
| Name | Required | Description | Default |
|---|---|---|---|
| key | No | ||
| mode | Yes | ||
| limit | No | ||
| query | No | ||
| conversion_id | No | ||
| min_usage_count | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden. It adds useful behavior beyond the schema by explaining empty-query defaults for conversions and patterns and by using verbs like 'retrieve' and 'search' to imply read-only behavior. It does not discuss side effects, auth, or rate limits, but those are less critical for a search-style tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a deprecation warning up front, a one-line summary, and clearly labeled mode sections. Each sentence provides useful operational guidance with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The mode breakdown and parameter semantics give enough detail to call the tool correctly, and the output schema covers return-value shape. The main gap is not clarifying how the context mode relates to the sibling context tool or exactly how key differs from conversion_id.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does so well by mapping each mode to its applicable parameters and explaining empty-query behavior for conversions and patterns, plus the context mode's key/conversion_id inputs. This adds meaning far beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool searches stored conversions, code patterns, and context data, and it breaks behavior into three named modes. It names convert as the preferred alternative, but it does not fully differentiate from the sibling context tool, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The deprecation notice explicitly tells the agent to prefer convert, which is strong when-not-to-use guidance. It also explains mode-specific parameter usage, but it does not spell out when search should still be used over convert or other siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
2 tool updates
v0.1.0- Changed
convert2 fields changed- added
Input schema / properties / styleAdded value: +{ + "default": "production", + "type": "string" +} - added
Input schema / properties / targetAdded value: +{ + "default": "spark", + "type": "string" +}
- Changed
glue_job1 field changed- changed
Input schema / properties / glue_version / defaultPrevious value: -"4.0"New value: +"5.0"
14 tool updates
v0.0.4- First observed
analytics - First observed
analyze - First observed
batch_status - First observed
context - First observed
convert - First observed
glue_data - First observed
glue_job - First observed
glue_s3 - First observed
glue_schema - First observed
optimize - First observed
refactor - First observed
review - First observed
s3_source - First observed
search
TDQS
Multiple tools have unclear boundaries: analyze, optimize, review, and refactor all target PySpark code analysis/improvement, while glue_s3, s3_source, glue_job, and glue_data overlap on Glue/S3 concerns. The deprecation notes help steer agents, but the large number of legacy tools still creates significant selection ambiguity.
Tool naming mixes single-word verbs (convert, analyze, review, search), noun-style names (context, analytics), and compound names in inconsistent orders (glue_s3 vs s3_source, glue_job vs batch_status). Though all are lowercase snake_case, there is no predictable verb_noun or noun_noun convention across the set.
Fourteen tools is a reasonable raw count, but 11 of them are explicitly deprecated, leaving only three actively preferred tools. The surface is bloated with redundant legacy tools that add noise without expanding genuine capability.
The core workflows—SQL-to-PySpark conversion, PySpark review, and Glue job generation—are well covered, with supporting capabilities for schemas, S3 layout analysis, batch processing, and context storage. Minor gaps exist, such as no non-deprecated tool for S3 source analysis or batch job status, but these are workable via the preferred tools.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Your Databricks Lakehouse in natural language: run SQL on your SQL warehouses, track long-running qu
Query BigQuery, Snowflake, Redshift & Azure Synapse with natural language
Ask data questions in natural language. Get SQL, insights, and charts from your databases.
Remote data science agents for Snowflake, Databricks & BigQuery in Claude/Cursor via MCP
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceProvides specialized tools for data engineering tasks like SQL formatting, dbt model generation, and Snowflake table creation. It enables users to analyze CSV data, validate pipeline configurations, and summarize ETL lineage through natural language.-
- AlicenseAqualityDmaintenanceProvides SQL analysis, linting, and dialect conversion using SQLGlot, enabling validation, transpilation, and extraction of table/column references.432MIT
- AlicenseAqualityDmaintenanceEnables AI assistants to query Apache Iceberg tables on S3 via AWS Glue Data Catalog using DuckDB as the embedded query engine, supporting columnar Arrow reads with no data movement.4MIT
- FlicenseNot gradedqualityDmaintenanceEnables natural language-powered ETL workflows using Airflow, AWS Glue, Athena, and S3, allowing LLM agents to control and monitor data infrastructure.-
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/AnnasMazhar/pyspark_mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server