propose_eval_criteria
Generates proposed evaluation criteria from a use case hint and optional sample calls, using an LLM judge to draft rubrics for measuring prompt quality.
Instructions
Have an LLM judge (gpt-4o-mini) propose eval criterion candidates from a one-line useCaseHint (e.g. "customer support bot") and optional sampleCallIds (representative calls from your account, up to 5) (POST /v1/eval-criteria/propose). An AI agent can finish "propose criteria to measure our prompt quality" in one prompt. Pro+ only (the backend enforces the plan gate + budget gate); nothing is INSERTed (propose only — adoption is a separate step via create_eval_criterion, structurally limiting LLM-hallucination impact). Decrypt failures for sampleCallIds are reported in partialFailures (the LLM call still runs without samples). Privacy note for prompt samples: with sampleCallIds, the backend decrypts those calls' prompt/response and sends excerpts (1500 chars each) to OpenAI gpt-4o-mini. This re-sends data your SDK originally sent to OpenAI/Anthropic, so no new vendor is added, but be aware it may reach an OpenAI model different from your own LLM calls. If that is a concern, run with useCaseHint only. Results are advisory: the returned criteria are LLM proposals and may include semantically weak rubrics (overuse of "helpful", duplicates) even when structurally valid. User review before adoption is recommended; do not feed them blindly into create_eval_criterion. Returns { criteria: [{ name (snake_case 32 chars), rubric (1-200), scaleMin (=1), scaleMax (5 or 10), reasoning (1-200) }], partialFailures: string[], budgetSpentUsd, proposedRawCount (raw count returned by the LLM), droppedCount (entries removed by the validator) }. Audit: emits an eval.propose_criteria event to the audit log.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| maxCriteria | No | Max number of criteria to return (1-10, default 5) | |
| useCaseHint | Yes | 1-2 line description of the intended use case (e.g. "customer support bot for e-commerce returns + refunds"; 1-500 chars, required) | |
| sampleCallIds | No | Array of call_ids from your account passed as context (optional, up to 5, [A-Za-z0-9_-]{1,128}). Grounds the LLM's proposals in your own data |