Evaluation dataset builder
submit_eval_dataset_builderTurn a table of task examples into a JSONL evaluation set and a scoring rubric. When you have labelled examples and need an eval set and rubric. Do not choose this when you need synthetic examples generated, which needs a model. Example requests: Build an eval dataset from these examples. | Turn this table of task examples into a JSONL eval set. | Make an evaluation set and rubric for this task.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"inputs": "<inputs>",
"authorized_by": "<authorized_by>"
},
"name": "submit_eval_dataset_builder"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| inputs | Yes | ||
| quote_id | No | The quote_id returned by POST /v1/jobs. Required to authorize a priced capability; the service rejects a priced job that does not name the quote it issued. | |
| authorized_by | Yes | Identifier of the human or agent authorizing this job. | |
| accepted_price | No | The amount accepted, exactly as quoted (for example "5.00" or 5). Checked against the stored quote, so a price that moved between quote and authorize is refused rather than silently absorbed. Optional, but sent to make the acceptance explicit. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| quote | No | ||
| state | Yes | ||
| job_id | Yes | ||
| artifacts | No | ||
| access_token | No | Per-job credential. Present it as the X-Job-Token header to read the job or list it. Keep it; it is the only credential for this job. | |
| verification | No | Independent verification. verification.ok = false means the job failed, not a partial success. | |
| capability_id | No |