execute_pyspark_job
Run declarative PySpark transformations on DuckDB datasets for heavy aggregations, using a fixed set of validated operations like filter, select, groupBy, and orderBy.
Instructions
Run a declarative PySpark transformation pipeline against a DuckDB dataset -- for heavier aggregations/transformations than run_sql_query is meant for. NOT arbitrary code execution: each step must be one of a fixed set of operations, validated before running.
Supported operations (each a dict with an "op" key): {"op": "filter", "condition": ""} e.g. {"op": "filter", "condition": "revenue > 100"} {"op": "select", "columns": ["a", "b"]} {"op": "withColumn", "name": "new_col", "expression": ""} e.g. {"op": "withColumn", "name": "margin", "expression": "revenue - cost"} {"op": "groupBy_agg", "group_by": ["a"], "aggregations": {"b": "sum"}} aggregations map column -> function; functions: sum, avg, mean, count, min, max, stddev, variance {"op": "orderBy", "columns": ["a"], "ascending": true} {"op": "distinct"} {"op": "limit", "n": 100}
Steps run in the order given. A final row cap is always applied to the output regardless of what the pipeline itself requests.
Args: source_dataset: Exact table/view name, as returned by list_datasets. operations: Ordered list of pipeline steps (see above). row_limit: Desired max rows returned (capped at the server's max_row_limit).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| row_limit | No | ||
| operations | Yes | ||
| source_dataset | Yes |