submit_job
Submit a job to a Dataproc cluster by specifying project ID, region, cluster name, job type, main file, and optional arguments, JAR files, and properties. Supports Spark, PySpark, Hive, Pig, and Hadoop job types.
Instructions
Submit a job to a Dataproc cluster.
Args:
project_id: Google Cloud project ID
region: Dataproc region
cluster_name: Target cluster name
job_type: Type of job (spark, pyspark, spark_sql, hive, pig, hadoop)
main_file: Main file/class for the job
args: Job arguments
jar_files: JAR files to include
properties: Job properties
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| args | No | ||
| cluster_name | Yes | ||
| jar_files | No | ||
| job_type | Yes | ||
| main_file | Yes | ||
| project_id | Yes | ||
| properties | No | ||
| region | Yes |
Input Schema (JSON Schema)
{
"properties": {
"args": {
"default": null,
"items": {
"type": "string"
},
"title": "Args",
"type": "array"
},
"cluster_name": {
"title": "Cluster Name",
"type": "string"
},
"jar_files": {
"default": null,
"items": {
"type": "string"
},
"title": "Jar Files",
"type": "array"
},
"job_type": {
"title": "Job Type",
"type": "string"
},
"main_file": {
"title": "Main File",
"type": "string"
},
"project_id": {
"title": "Project Id",
"type": "string"
},
"properties": {
"additionalProperties": {
"type": "string"
},
"default": null,
"title": "Properties",
"type": "object"
},
"region": {
"title": "Region",
"type": "string"
}
},
"required": [
"project_id",
"region",
"cluster_name",
"job_type",
"main_file"
],
"title": "submit_jobArguments",
"type": "object"
}