evaluate_job
Perform an asynchronous LLM evaluation of a completed job using a custom rubric, scoring job inputs and outputs against each criterion with numeric scores and justifications.
Instructions
Trigger an LLM-as-judge evaluation of a completed job against a rubric.
The evaluation runs asynchronously: the judge LLM scores the job's input/output against each criterion in the rubric and produces a numeric score with a textual justification. Results are retrievable via get_job_evaluations.
Write operation — recorded in the audit log.
Args: job_id: UUID of the job to evaluate (must be in "success" or "failed" state). rubric_id: UUID of the rubric to apply (from list_eval_rubrics).
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"job_id": "<job_id>",
"rubric_id": "<rubric_id>"
},
"name": "evaluate_job"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | ||
| rubric_id | Yes |