errorbar

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v0.5.1
Disambiguation3/5
Most tools target distinct resource+action combos, but there are genuine overlaps: can_i_ship and get_eval_gate both serve as CI-gate verdicts on finished eval runs, and is_my_judge_trustworthy largely re-reads trust/TPR/TNR/drift data already present in get_criterion and list_criteria. Additionally, create_eval and screen_my_traffic both support 'is a cheaper model better' screening, though the descriptions try to disambiguate them.
Naming Consistency4/5
The dominant pattern is clean verb_noun snake_case (list_*, create_*, get_*, update_*, delete_*, cancel_*) applied consistently across 20+ resource domains. The exceptions are notable but few: the first-person sentence-style names can_i_ship, is_my_judge_trustworthy, and screen_my_traffic break the convention, and upsert_alias/auto_improve_criterion deviate slightly from standard CRUD verbs.
Tool Count2/5
81 tools is far beyond a typical MCP surface and will substantially bloat agent context and increase misselection risk, even though the underlying platform genuinely spans many domains (evals, fine-tuning, GRPO, dedicated endpoints, audit, labels, datasets). The breadth is real but the tool count is still excessive for an agent-facing interface, sitting at the extreme end of 'too many.'
Completeness3/5
Core lifecycles are thoroughly covered for most resources: criteria, evals, dedicated endpoints, batches, fine-tuning jobs, GRPO runs, model versions, and logs all have create/read/list/update/delete or cancel where appropriate. Notable gaps exist: datasets are create-only (no list/get/delete for created datasets), label sets cannot be deleted, training files cannot be removed, and labels have no delete operation—gaps that will matter for cleanup workflows in an audit-focused platform.
Average 4.6/5 across 81 of 81 tools scored. Lowest: 3.7/5.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 10 commits in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI is passing
This repository is licensed under Apache 2.0.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld, and non-destructive behavior, and the description reinforces this with 'Read-only.' It adds meaningful behavioral context by enumerating the output dimensions: trust verdict categories, TPR/TNR intervals, catch/pass rates with 95% intervals, κ, measurement count, recency, drift status, and the one state-changing action. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense and front-loads the main question, then lists the returned metrics in a compact, readable sequence. It is slightly longer than necessary due to the rhetorical opening and the long parenthetical list, but every element adds useful detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description does a good job listing what the caller will receive: verdict categories, confidence intervals, rates, κ, sample size, timing, drift status, and the one action that changes state. It does not fully clarify the return structure for the 'all judges' case or the exact meaning of 'the one action,' so it stops just short of complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description covers the only parameter, criterion_id, 100%: 'One criterion's id for its full numbers; omit for every judge in the workspace.' The tool description repeats the same idea ('pass criterion_id') without adding new format, value, or interpretation details, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the resource (workspace judges) and the core purpose (assessing trustworthiness with verdict categories and metrics). It distinguishes itself from siblings by focusing on trust measurement rather than settings or criteria management, though it opens with a rhetorical question instead of a direct action verb and the 'one criterion or all of them' phrasing is slightly ambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides scoping guidance: pass criterion_id for one criterion or omit it for all judges. However, it does not explicitly say when to use this tool versus alternatives like get_judge_settings or get_criterion, nor does it mention exclusions or when not to use it, so usage context is implied rather than fully explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish read-only, idempotent, non-destructive behavior, and the description adds meaningful behavioral details beyond that: the 'actual' value uses the interval's LOWER bound, omitted eval_id targets the newest DONE run, and sensible defaults apply. This is valuable behavioral context that the annotations alone do not provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense sentence with no filler, front-loaded with the core question and decision outcome before diving into defaults and lower-bound semantics. Every clause contributes either selection criteria, behavior, or output shape.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the full schema coverage, rich annotations, and no output schema, the description supplies the essential missing context: what the return is (ship/hold plus required vs actual), which runs it applies to, and how defaults work. It is complete enough for an agent to invoke correctly, though a bit more detail about all possible check types would fully round it out.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all six parameters; the baseline is 3. The description adds extra semantic value by explaining the certified-switch default (noninferiority_margin 0.05), min_win_rate 0.5 default, and how actual values map to interval lower bounds, which enriches parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool reads the gate on a finished eval run and returns a ship/hold decision with required vs actual values, which goes well beyond the name. It does not explicitly differentiate itself from the sibling get_eval_gate, so it falls just short of full sibling-distinguishing clarity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: use this when deciding whether evidence lets a change ship, including the CI-step role and the default to the latest DONE run. It does not name alternatives or state when not to use it, but the context is specific enough for an agent to select it appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
It goes well beyond the destructiveHint annotation by disclosing that calibration history is permanently lost and that aliases gated on this criterion lose their gate. It also specifies the permission requirement and error cases, providing a clear behavioral contract. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence serves a purpose: core action, endpoint/API scope, return shape, permission, error behavior, and side effects. It is dense but not bloated, and the most important information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter delete with no output schema, the description is complete: it documents the response, permissions, not-found behavior, and cascading side effect on aliases. No critical information is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already covers the only parameter (id) with 100% coverage, so the description does not need to add param-level detail. The baseline of 3 applies; the description does not contribute extra parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action on a specific resource: 'Permanently delete a criterion and its calibration history.' This clearly identifies the tool's purpose and distinguishes it from siblings like delete_alias and delete_eval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives useful context (OWNER/ADMIN only, workspace-scoped 404, alias side effects) but does not explicitly state when to use this tool versus alternatives such as update_criterion or delete_alias. The usage is implied by the delete verb rather than explicitly routed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare destructiveHint=true, readOnlyHint=false, and idempotentHint=true, and the description substantially enriches this with the hard-delete nature ('Hard delete'), the audit event side effect, the fineTuning feature-flag gate producing 404, the OWNER/ADMIN 403 condition, and the workspace-scoped 404 for unknown ids. The described 404-on-missing behavior is consistent with the idempotentHint annotation. No contradiction exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: purpose, endpoint and API-key scope, return shape, feature-flag gate, permission requirement, not-found case, and side effect. It is front-loaded with the purpose statement and uses compact parentheticals and labeled fields to preserve readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive single-parameter tool with no output schema, nothing critical is missing: the return value is stated ({ ok: true }), all expected error states (404 feature off, 403 permissions, 404 not found) are enumerated, auth scope and side effects are described, and the schema covers the only parameter. An agent has everything needed to invoke it correctly and anticipate failures.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema's id field already documents its source (GET /v1/env/tools) and workspace constraint, so the baseline of 3 applies. The description adds modest value by tying an out-of-workspace id to the 404 behavior, but it does not introduce new parameter-level details beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Revokes a declared agent tool registration (consent withdrawal) so training environments can no longer call that endpoint.' This precisely distinguishes the tool from its siblings, particularly register_env_tool and list_env_tools, by stating the action, the resource affected, and the consequence. Including the DELETE endpoint reinforces the scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The purpose statement implies when to use it (consent withdrawal for a previously registered agent tool), and the prerequisites are spelled out: fineTuning feature flag must be on, and an OWNER/ADMIN minting user is required. However, it never explicitly names alternatives or when-not-to-use conditions; given siblings like register_env_tool, explicit routing would have been valuable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and non-destructive. The description adds substantial behavioral context beyond those: recomputed on read so never stale, calibration null until pairs are labelled, corrected null until usable, and the same block stored on results.pairwise so gate/evidence read identical numbers. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose, and every clause carries information, but it is a single dense paragraph with deeply nested parentheticals describing the full return shape. It is thorough rather than concise, and would benefit from bulleted structure or clearer separation between response fields, error cases, and usage notes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description carries the full burden of explaining return values; it does so in exhaustive detail including null states, confidence intervals, floor half-width, and the corrected-rate formula. It also covers recomputation behavior, 400/404 conditions, and usage guidance about when to request more labels instead of samples. Nothing important is missing for a one-parameter read-only tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required id parameter with 100% description coverage: 'The eval run id (comparison runs only).' The tool description restates the endpoint path and comparison-run context but adds little parameter-specific meaning beyond the schema. Baseline 3 is appropriate because the schema already documents the only parameter fully.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-plus-resource statement: 'The pairwise judge's spec sheet and calibration for a comparison run,' and includes the exact GET endpoint. This clearly differentiates it from sibling tools like get_eval_evidence or get_eval_samples by tying it to pairwise comparison-run calibration and labelled-pair statistics.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states read-only, no-spend behavior, and gives concrete error boundaries: 404 when the run is not in the workspace and 400 for criterion runs. It does not name a sibling alternative for criterion runs, but the exclusion is clear enough that an agent knows this tool is only for comparison runs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only say not read-only and not idempotent, but the description adds substantial side-effect information: 'SPENDS MONEY: this starts billable work', wallet runway requirements, pricing freeze at submit, and who can execute. It also discloses return status and several error paths, giving the agent a full behavioral picture.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description front-loads the core purpose and then packs in auth, cost, return shape, and error codes. It is dense and somewhat run-on, but every clause carries information an agent needs to call the tool safely, so the length is justified.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description compensates by listing the returned batch object fields and 201 success status. It covers main error variants, permission requirements, and financial consequences, making it complete for a side-effectful paid API call.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already explains every parameter. The tool description adds no extra meaning beyond what the schema provides (e.g., it references input_file_id but the schema already describes the purpose and ownership requirement). Baseline 3 is appropriate when the schema handles parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence states a specific verb ('Submit'), resource ('batch of inference requests'), and key properties ('asynchronous, discounted', 'from a previously uploaded JSONL file'). It clearly distinguishes this creation action from sibling tools like list_batches, get_batch, and cancel_batch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly names the workload context ('for workloads that can wait up to the completion window') and gives preconditions (previously uploaded JSONL file, wallet runway, owner/admin). It does not name alternative tools or provide explicit when-not-to-use statements, but the context and error conditions make selection clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations, the description discloses the HTTP method, required permission (OWNER/ADMIN, 403), side-effect distinction (create does not spend vs judging spends), and the 201 response shape. It also states endpoint limitations on settable fields. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded purpose followed by endpoint, response, and notes in a compact structure; the response field enumeration is long but compensates for the missing output schema. Every clause adds information, though the single paragraph is dense.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description fully specifies the return payload, auth requirements, spend implications, and keyword casing. Combined with the 100%-covered input schema, an agent has everything needed to invoke the endpoint correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers all 7 parameters with individual descriptions (100% coverage), so the description need not redefine them. It adds only a snake_case serialization note and an exclusion of other knobs (coverage, pre-checks, contract rules), which is helpful but marginal semantic value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description opens with a precise verb+resource: 'Create a judge criterion' and defines it as 'a rubric prompt run by a judge model' that scores traffic online and calibrates against human labels. It includes the HTTP endpoint and API-key scope, making the operation unmistakable and distinct from sibling criteria/eval tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Notes give concrete usage context: OWNER/ADMIN only, POST /v1/criteria with evals:write scope, and 'Creating does not spend; judging (align, online monitoring) does.' It also warns that 'other criterion knobs ... are not settable through this endpoint,' but does not explicitly name an alternative tool like update_criterion or suggest_criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes far beyond the annotations, which only carry readOnlyHint=false, openWorldHint=true, idempotentHint=false, destructiveHint=false. The description discloses the critical trap that 'index: null means... NOTHING was checked — never treat that as clean,' scopes 'clean' to those benchmarks on that index date, notes one text can produce multiple hits, documents the 400 error conditions, and states it is free with Cache-Control: no-store. No contradiction with the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every section earns its place: purpose, usage timing, endpoint/scope, full return shape, error conditions, and caveats. With no output schema present, the detailed return documentation is necessary rather than padding. Slightly dense, but the tricky null-index semantics justify the length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a tool of this complexity. Since there is no output schema, the description fully documents the return object, error paths, and the most dangerous behavior (null index ≠ clean) that could silently corrupt downstream training decisions. The annotations cover the mutation/idempotency profile and the schema covers the single parameter, so nothing needed for a correct call is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%: the schema already documents that `texts` must be an array of strings, capped at 5,000 items, and that array positions map to response `index` values. The description adds the 400-on-invalid-input behavior and repeats the 5,000 cap, but the parameter's meaning itself is fully carried by the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource: 'Checks a batch of texts against the public-benchmark contamination index... and reports which inputs share material with which benchmark.' The mention of 13-word shingles and well-known test splits makes the mechanism concrete, and nothing in the ~130-tool sibling list covers benchmark contamination checking, so the tool is unmistakably distinguished.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs 'use it before training so later benchmark scores measure capability, not memorised answer keys' — a clear when-to-use directive with the rationale behind it. It doesn't name alternatives or when-not-to-use cases, but there is no competing contamination-check tool among the siblings, so that omission is not a practical gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent/destructive annotations, the description discloses important live-behavior details: best-effort reconciliation may return the stored row unchanged, billing is deferred to a background reconciler, and 404 behavior depends on workspace and feature flag. This gives the agent accurate expectations of side effects and failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense but well organized: core purpose first, then endpoint/scope, return shape, and edge-case notes. Every sentence contributes useful operational context, though the return-field list makes it slightly longer than minimal.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description compensates by enumerating the return fields. It also covers API-key scope, live reconciliation semantics, billing side effects, and 404 edge cases. For a simple one-parameter read tool, this is fully sufficient for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully describes the one required parameter with 100% coverage, including the distinction between platform id and upstream batch id. The description adds no new parameter-level semantics beyond restating the endpoint path, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource—'Fetch one batch job'—and clearly states the live status/count/file-id retrieval purpose. It is immediately distinguishable from sibling tools like list_batches (bulk listing) and cancel_batch (mutation).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'to poll for completion' gives a clear, actionable use case, and the 404 notes explain when the tool will fail. It does not explicitly name alternatives or state when not to use it, but the single-resource framing sufficiently guides selection among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, openWorld, idempotent, and not destructive, and the description adds substantial behavioral context: 'Read-only, no spend,' 404/412 error specifics, silent degradation of with_content when logging is off, unsigned bundles when no signing secret exists, and 'fail closed like the gate.' No contradiction with annotations exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but the length is largely justified by the absence of an output schema: the detailed return payload must live somewhere. It front-loads purpose and uses compact, structured formatting for the response fields. Some redundancy exists (e.g., 'Read-only, no spend' overlaps with annotations), but the density is mostly earned.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity, seven parameters, and no output schema, the description is remarkably complete: it specifies the exact response shape, conditional fields, error codes, silent degradation behavior, and authentication scope. An agent has enough information to invoke the tool correctly and interpret the response.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description mentions with_content degrading and gate parameters being embedded, but it does not add much beyond the schema's own parameter descriptions. It is adequate but not additive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific action ('Download the forwardable proof bundle') and a specific resource ('a DONE eval run'), then enumerates the bundle's contents in detail. It clearly differentiates this from sibling tools like get_eval_samples or get_eval_gate by emphasizing the audit/attestation purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description states when the tool applies ('for a DONE eval run', 'hand a reviewer or auditor') and gives operational guidance: poll until DONE, fail closed, 404 outside workspace. It does not explicitly name alternatives or say when not to use this versus get_eval_gate or get_eval_samples, but the purpose and conditions are clear enough for correct selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations carry only bare hints (readOnlyHint:false, idempotentHint:false, openWorldHint:true), and the description goes far beyond them: billable spend with per-label economics ('one judge call per label; a new run is new spend'), auth restrictions, dual sync/async execution with a threshold (<=50 labels), every major error mode with its status code and message, the polling endpoint, and a 300s maxDuration. It also clarifies the openWorldHint by disclosing that background work continues after the response. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Well front-loaded: purpose first, then endpoint/auth, then the capitalized money warning, then behavioral branches and errors. Nearly every sentence earns its place, and the verbose return schemas are justified by the absence of an output schema. Minor inefficiency: the money warning is stated twice ('SPENDS MONEY' and again in 'Notes: MONEY...').
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a genuinely complex tool — dual sync/async execution, billing, permissions, timeout, and no output schema — and the description covers every branch: both return payloads with field details, polling instructions, the 30-label minimum, all key error codes with example messages, and the 403 auth constraint. Nothing an agent needs to invoke and interpret this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% — the single 'id' parameter is documented as 'Criterion id.' — so the baseline of 3 applies. The description only incidentally confirms the parameter via the URL template POST /v1/criteria/{id}/align and adds no format, validation, or usage nuance beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource with rich detail: 'Calibrate a criterion by re-judging every in-scope human-labeled trace and measuring agreement (TPR/TNR with Wilson intervals, Cohen's kappa)'. The 'earns a judge the trust needed to gate on it' clause gives operational purpose, and the detail distinguishes this from siblings like get_criterion_alignment (fetch results) and auto_improve_criterion (improve the criterion).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides clear context for invocation: it is the step that earns gating trust, it explicitly warns OWNER/ADMIN only (403) and that it spends wallet money, and it instructs how to follow up for large sets (poll GET /v1/criteria/{id}/alignment). However, no sibling alternatives are named (e.g., get_criterion_alignment for checking existing results without spending) and there is no explicit 'when not to use' statement.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Extensive disclosure far beyond annotations: billable wallet spend with a pre-placed cost-ceiling hold, 402 if the hold fails, OWNER/ADMIN requirement (403), rate limiting (429), a thorough 400 taxonomy, feature-flag gating, scopes-not-enforced, and the 202 queued-run lifecycle with polling follow-up. Annotations (readOnlyHint=false, idempotentHint=false) are consistent with this mutating, non-idempotent, non-destructive action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Dense but well-ordered: purpose front-loaded, then HTTP shape, then the money warning, then auth/rate/error conditions. The length is justified for a spend-incurring operation with complex failure modes, though the cost warning appears twice ('SPENDS MONEY' and the Notes ledger/hold detail), creating minor redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the return-value burden and delivers: 202 {trigger_run_id}, queued status, and polling guidance. Combined with the full error taxonomy, auth requirement, rate limit, and cost-ceiling hold behavior, an agent has everything needed to decide, call, and follow up correctly. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents id and judge_criterion_id, including the judged pass-rate comparison details. The description adds only summary-level phrasing ('optionally a judged win-rate') and does not meaningfully extend what the schema provides, matching the baseline for full schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (start), resource (held-out bake-off), and scope (proves a succeeded supervised fine-tune against its base model via NLL/perplexity and optionally a judged win-rate on an ephemeral GPU box). This distinguishes it from siblings like get_fine_tuning_bakeoff (which polls status) and create_fine_tuning_job (which creates the job being baked off).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clear context for when to use it: proving a SUCCEEDED supervised fine-tune on a held-out split without deploying the model, with explicit preconditions and a detailed 400-condition list. It routes the agent to the GET bakeoff endpoint for status/verdict, but never explicitly names get_fine_tuning_bakeoff as the sibling alternative, so differentiation is implicit rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Extremely rich for a tool whose annotations only state readOnly=false, openWorld=true, idempotent=false, destructive=false. The description discloses billing (SPENDS MONEY, billed under assist:suggest), auth requirements (OWNER/ADMIN, evals:write scope), error conditions (400s with counts), the valid body-less POST, dedup behavior (same-name skipped, never overwritten), the trust model (drafts never gated until human review), and maxDuration 300s. No contradiction with annotations — readOnly=false and idempotent=false align with a billable draft-creating call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Information-dense and front-loaded: the core purpose leads, followed by endpoint/scope, cost warning, return shape, and edge cases. Every sentence earns its place. The only minor redundancy is the billing warning appearing twice ('SPENDS MONEY' and 'SPENDS THE WALLET'), which can be read as deliberate emphasis on a critical fact.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description carries the full burden of explaining return values — and does so exhaustively, including the exact criterion object shape plus critiques_used and skipped_duplicates. Prerequisites, failure modes, auth, timeout, and the dedup/trust behavior are all covered. Nothing essential is missing for an agent to invoke this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% — judge_model is fully documented with its default model, the playground-catalog requirement, and whitespace-only fallback. The description adds the body-less POST default-judge behavior, but since the schema already carries the parameter meaning, the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (clusters), resource (failure critiques → up to 5 DRAFT judge criteria, one per failure mode), and the workflow position (after grading fails with reasons). The draft/assist-suggested nature clearly distinguishes it from siblings like create_criterion, update_criterion, and auto_improve_criterion without needing to open their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context: use it after grading a batch of fails with reasons to bootstrap criteria that are then reviewed and aligned. It also states the prerequisite (≥10 FAIL grades with critiques, only the 200 most recent considered) and follow-up workflow. However, it never explicitly names alternatives such as create_criterion for hand-writing a single criterion or run_criterion_alignment for the post-draft step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes far beyond annotations by disclosing that the gateway resolves aliases in ~10s, that a 200 is returned for both create and update, that evidence policy can block repointing with a 412, that a brand-new alias is never blocked, and that billing follows the model that actually ran. This is rich, non-obvious behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long due to the tool's complexity, but it is well structured and front-loaded: purpose first, then HTTP/scope, return shape, then behavioral notes and error semantics. Each section earns its place, though the dense list of 400-mode failures makes it slightly heavier than necessary.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 12 parameters, meaningful behavioral side effects, and no output schema, the description is remarkably complete: it covers the return object, auth requirements, idempotency, timing, billing, and the main 400/404/412 failure modes. An agent has enough information to invoke this tool correctly and understand consequences.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the input schema already fully documents all 12 parameters. The description adds high-level context about optional canary/gate/evidence attachments and the returned alias object, but does not meaningfully expand on individual parameter semantics beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb plus resource: 'Create or repoint a model alias by name (idempotent upsert)'. It clearly distinguishes this from the sibling list_alias and delete_alias operations, and states the production purpose of moving traffic without redeployment.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives strong contextual guidance: when to use it ('so production traffic moves to a new model without a redeploy'), optional uses (canary split, quality gate, evidence policy), and the OWNER/ADMIN prerequisite. It does not explicitly name alternative tools or state when not to use it, but the use case is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses side effects well beyond the sparse annotations: attaching replaces prior bindings, detaching sets label_set_id to null, and unfrozen sets trigger a 400. It also provides the exact return shape and error messages for foreign ids. This adds rich behavioral context that the annotations (readOnly=false, openWorld=true) do not cover.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but efficiently structured: it front-loads the action and purpose, then gives the endpoint, return shape, error notes, and key constraint in a logical order. Every sentence conveys necessary information, and the 'attaching replaces the previous one' caveat is clearly stated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description specifies the exact response shape, error codes, prerequisite freeze condition, and side effects. It also includes the required API-key scope, making it self-contained for an agent to invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides 100% parameter coverage, including that id must be frozen and is ignored when detach=true, and that criterion_id is non-empty. The description reaffirms these constraints but does not add significant new parameter-level meaning. Therefore the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Attach a frozen golden set to a judge criterion' and explains the downstream consequences for calibration and certificate kappa. It also covers the detach variant, making the dual purpose explicit. This clearly differentiates it from sibling tools like freeze_label_set, which operate on the set's freeze state rather than the criterion binding.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear situational context: use this to bind a frozen set to a criterion for calibration, or to clear that binding via detach. It specifies the prerequisite that the set must be frozen, and warns that a criterion holds at most one set, so attaching replaces the previous one. It does not name alternative tools, but no direct alternative exists in the sibling list.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, the description discloses the HTTP endpoint, required API-key scope, expected status values, common error codes, ownership restrictions, audit logging, feature-flag behavior, and billing consequences. This is substantial behavioral context that helps an agent anticipate side effects and failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and information-dense, with the core action and endpoint front-loaded, followed by high-value notes on permissions, errors, auditability, and billing. Every sentence adds necessary context without repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With one simple parameter, no output schema, and minimal annotations, the description covers everything needed to invoke the tool correctly: endpoint, auth scope, permission constraints, response behavior, error conditions, and side effects. Nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully documents the single parameter 'id' as 'The platform batch id' with 100% coverage. The description does not add new parameter-level detail, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Request cancellation'), a specific resource ('in-flight batch job'), and the exact endpoint. It clearly identifies the tool's scope and distinguishes it from sibling cancellation tools like cancel_fine_tuning_job or cancel_eval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for use ('in-flight batch job') and important operational constraints such as OWNER/ADMIN only, 403 for unauthorized callers, feature-flag-related 404s, and billing implications. It stops short of explicitly naming alternatives or stating when not to use this tool, but the guidance is otherwise strong.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations, disclosing auth requirements (OWNER/ADMIN minting user, 403), terminal-state refusal (400), workspace scoping (404), billing implications, race behavior for in-flight slices, and the exact response shape and error message. This is exceptionally transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Though lengthy, every sentence conveys a distinct operational fact: conditions, auth, refusal behavior, error codes, race semantics, and response contents. The most important purpose is front-loaded, and the detail is justified by the tool's subtle billing and terminal-state behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete for a mutation tool with no output schema and one parameter. It covers prerequisites, failure modes, response shape, side effects on billing, and edge cases. An agent has everything needed to call this tool correctly and predict the outcome.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single required id parameter, so the schema already documents it fully. The description references the id in the endpoint path but adds no meaningful parameter semantics beyond what the schema provides, matching the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Stop a PENDING or RUNNING eval run.' It also states the motivating purpose (misconfigured or no longer needed) and clarifies the billing rationale, making the tool's role clear and distinct from siblings like delete_eval or cancel_batch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use the tool ('when a run was misconfigured or is no longer needed') and clearly lists when not to use it: a run already DONE, ERROR, or CANCELLED is refused. It does not name alternative sibling tools, but the when/when-not guidance is strong enough to route an agent correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations, the description discloses the upsert side effect ('Upserting an existing seq replaces the reason'), the exact API-key scope (platform:write), the 403 role restriction, the 201 return shape, and specific 400 failure modes. This is far richer than the annotation set alone and gives the agent a precise behavioral model.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence contributes: purpose, endpoint/scope, return shape, auth requirement, and error conditions. The purpose is front-loaded and the technical details are packed into efficient clauses without repetition or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description provides the 201 response fields and error cases. It covers authentication, side effects, and preconditions. Nothing needed to call the tool correctly is missing, and the reference to GET /v1/audit/verify in the schema complements the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed descriptions for both seq and reason, so the schema carries the parameter meaning. The description reinforces the same constraints (seq must be a gap, reason >= 10 chars) but does not add material new semantics beyond the schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Acknowledge a verified audit-chain gap with a written reason' and explains the effect ('stops reporting it as unexplained'). It also characterizes itself as 'a platform-admin repair action', clearly separating it from read-only siblings like list_audit_tombstones and get_audit_verification.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: it is for verified gaps, requires platform-level admin, and 'never a way to hide a gap'. It enumerates 400 refusal conditions, so the agent knows when it will be rejected. It stops short of explicitly naming an alternative tool to use instead, but the conditions are strong enough to infer appropriate use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
This is an exceptionally transparent description. It discloses auto-drop categories, the 50,000 row fetch cap, judge-mode billing and gating, failure-of-whole-build semantics, idempotent retry billing, eval-split failure behavior, disjointness guarantees, and human-FAIL exclusion. These details go far beyond what the sparse annotations (readOnlyHint/idempotentHint/destructiveHint) convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence carries real content and the length is arguably justified by the tool's complexity, but the description is one dense wall of text with critical constraints buried in the middle. It would benefit from structured sections or bullets for billing, error cases, and prerequisites, and the most important usage guidance could be front-loaded more clearly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Even though there is no output schema, the description supplies a detailed response shape, error statuses, auth requirements, failure modes, billing caveats, and edge cases like contamination and refusal ledgers. An agent has nearly everything needed to call this tool correctly and predict what will happen on success or failure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the schema descriptions are already rich, so the baseline is 3. The description adds meaningful non-obvious consequences: the fetch cap, judge spending and billing behavior, no re-billing on retry, eval split naming, and the fact that a failed eval split still leaves the training set created. These details help an agent reason about parameter impact beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Curates logged gateway traffic into a managed training dataset') and immediately distinguishes this from related tools like create_eval or upload_training_file by describing the source, the curation action, and the optional eval split. It also gives the concrete endpoint and auth scope, so an agent knows exactly what this operation is.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use the tool ('use it to turn production logs into fine-tuning or eval data') and states critical prerequisites such as OWNER/ADMIN minting and evals:write scope. It does not explicitly name alternative tools or say 'use X instead', but the context is strong enough for an agent to determine appropriate use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations establish that this tool is destructive and not read-only; the description adds substantial beyond-annotation context: API-key scope (aliases:write), role restrictions (OWNER/ADMIN only, 403), not-found behavior (404 when the id is not in the workspace), the fact that users of the alias will break, and that the action is audited with the model it pointed at. No contradiction with the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: purpose, consequence, endpoint, scope, return value, permission error, not-found error, and audit trail. The most decision-relevant behavior is front-loaded in the first clause.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter destructive operation with no output schema, the description fully compensates: it states the success response, error cases, required permissions, workspace behavior, and audit side effect. Nothing an agent needs to safely invoke or reason about this tool is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is only one parameter, id, and the schema already fully describes it including the important caveat that it is the alias id from GET /v1/aliases, not its name. The description adds no additional parameter-level meaning beyond restating the endpoint, so the high-coverage baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Remove a model alias') and immediately adds the key semantic distinction ('cutover step, not cleanup'), so an agent can tell it apart from alias creation, listing, and general cleanup operations. The HTTP method and endpoint reinforce the action.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says this is a cutover step rather than cleanup and warns that requests still using the alias will fail afterward. It gives clear context for when to call it, though it does not name a specific sibling alternative for non-cutover cleanup.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark it destructive and non-readonly, but the description adds crucial behavioral context: a final meter/billing step, soft-delete semantics, exclusion from lists, non-reversibility, authorization scope, and the 404 behavior for already-deleted endpoints. This goes well beyond the annotations and fully discloses consequences.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place, covering action, billing effect, API details, permissions, error cases, and irreversibility. It is front-loaded with the core purpose and then layers supporting details efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive, billing-affecting operation with no output schema, the description is complete: it states the response shape, required permissions, error behavior, side effects, and irreversibility. An agent has everything needed to invoke it correctly and understand consequences.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides 100% coverage for the single 'id' parameter, including a clear description. The tool description adds no additional parameter-level meaning beyond what the schema states, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Tears down') and resource ('dedicated endpoint'), and explains the result: billing accrued GPU-hours, freeing GPUs, marking DELETED, and stopping payments. It clearly distinguishes this from related endpoint operations like update or create.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explains exactly when to use this tool ('the way to permanently stop paying for an endpoint') and gives prerequisites (OWNER/ADMIN, workspace ownership). It doesn't explicitly name alternatives or say when not to use it, but the context is clear enough for an agent to select it appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent annotations, the description discloses numerous non-obvious behaviors: the 10,000-row cap with no limit/offset, the X-Omnia-Export-Capped truncation header, skipped malformed rows rather than failures, a 409 when request logging is disabled, and exclusion of aborted exchanges. These add real context beyond what annotations provide, and there is no contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every clause earns its place: endpoint, auth scope, response content type, line format, ordering, headers, cap behavior, malformed-row handling, and error shape. It is front-loaded with the core purpose and then layers operational specifics without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema present, the description fully compensates by detailing the JSONL body format, headers, error cases, truncation detection, and filter advice. An agent has everything needed to invoke and interpret the result correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%: all 7 parameters (start, end, model, tag, segment, cache_hit, finish_reason) have individual descriptions in the schema. The tool description adds general filtering context (e.g., 'narrow the filter') but does not elaborate on individual parameters, so the schema carries the burden; baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Export the filtered logged exchanges as JSONL in chat format', naming the verb, resource, and output format in one precise sentence. It further specifies the assistant reply appended and the training/eval use case, which cleanly distinguishes it from siblings like list_logs or export_audit_log without needing to inspect their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly states the intended use case ('ready to pipe into your own training or eval tooling') and gives actionable advice when the cap is hit ('narrow the filter, e.g. a time range'). However, it does not explicitly name sibling alternatives or state when not to use this tool, leaving some inference to the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses important behaviors beyond annotations: re-freezing re-measures kappa but preserves original frozen_at and membership, kappa is null rather than 'good' when insufficient raters exist, 400 is returned when inter-rater stats are unavailable, and the endpoint requires evals:write scope. This is rich, honest behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core purpose, then organized into endpoint, response shape, and edge-case notes. Every sentence carries useful information; there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully compensates by enumerating the complete returned object and its field types. It also covers idempotency nuances, null semantics, and error conditions, making the tool's behavior fully understandable for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the id parameter is already documented as workspace-scoped with a 404 case. The description adds the endpoint path and no-body requirement, but does not substantially expand parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: freeze a golden set, seal membership with a hash, and measure inter-rater agreement (kappa). The unique behavior clearly distinguishes it from sibling label-set tools like create_label_set or attach_label_set.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides clear context by saying freezing is required before a judge can calibrate on the set, and explains re-freezing behavior. It does not explicitly name alternatives or state when-not-to-use, but the usage context is unmistakable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, and the description adds substantial context: it is free, never re-judges, returns a persistent report, supports polling via a specific 400 error, and explains when threshold_analysis is null. There is no contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is detailed but well-structured, front-loading purpose and scope before the return shape and error notes. Every sentence adds relevant information, especially since there is no output schema to document the response.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderately complex return object, lack of an output schema, and single simple parameter, the description is complete: it documents the full return structure, error cases, polling behavior, historical null caveat, and key naming convention. An agent has enough to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has one parameter, id, described as 'Criterion id,' and schema description coverage is 100%. The description adds the endpoint pattern using {id}, but does not need to elaborate further because the parameter is simple and already documented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb ('Read') and a precise resource: the persistent report from the criterion's last calibration run. It enumerates the report contents and explicitly distinguishes itself by saying it never re-judges, making it easy to separate from run_criterion_alignment.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly communicates when to use this tool: to read an existing alignment report, poll an in-progress run, or check why no report exists. It says it is free and never re-judges, which implies a safe alternative to running alignment, but it does not explicitly name run_criterion_alignment as the counterpart tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the readOnly/idempotent annotations: it discloses that the call is free and does no judging, returns 404 for out-of-workspace criteria, uses Cache-Control: no-store, returns unsigned certificates when no signing secret is configured, and honestly reports unmeasured calibration. It also explains the camelCase/verbatim emission so the signature can be re-derived, which is critical behavioral context. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but tightly structured: a one-sentence summary, endpoint and auth scope, response shape, then operational notes. Every block earns its place, especially because there is no output schema, so the detailed inline JSON shape is necessary rather than filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a complete description for a one-parameter GET: endpoint, auth scope, full response shape, signature verification flow, error condition, caching behavior, and degenerate uncalibrated case are all specified. An agent has everything needed to decide when to call it and what to do with the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single `id` parameter, so the schema already carries the semantic weight. The description adds the 404-out-of-workspace behavior and repeats the path/workspace membership, but it does not materially change or extend parameter semantics beyond what the input schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description opens with a specific verb/resource: 'Returns the signed judge certificate for a criterion' and names the exact contents and use cases (audits, evidence bundles, proving a judge's calibration to a third party). This distinguishes it from siblings like get_criterion or is_my_judge_trustworthy by focusing on a signed, verifiable certificate rather than the criterion definition.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit use contexts: 'for audits, evidence bundles, or proving a judge's calibration to a third party.' It does not explicitly say when not to use it or name a sibling alternative, but the context is clear enough for an agent to select it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, and non-destructive; the description adds meaningful behavior beyond that: the exact GET route, API-key scope, note that the local apiKeyActor does not enforce scopes, the fact that it is implemented via a full live-reconciled list call, and the 404 behavior. This is rich, non-obvious operational context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and use cases, then provides the route, return shape, and critical caveats. Every sentence carries useful information, and despite its length, no part is redundant or wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Even without an output schema, the description enumerates the full snake_case object shape, explains auth scope behavior, ownership requirements, implementation cost, and error semantics. For a one-parameter read tool, this is complete enough for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the single parameter's description already covers the workspace ownership constraint and deleted-endpoint 404 behavior. The description repeats the 404 and scoping context but adds little beyond the schema for the parameter itself, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Returns one dedicated endpoint's current view' with concrete fields (live status, frozen hourly price, unbilled accrued cost, routing key). It clearly distinguishes itself from list_dedicated_endpoints by returning a single object, and even names its use cases, so an agent can tell it apart from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to call it: 'use it to poll a deploy until RUNNING or to check spend.' This gives clear context and intent. It does not explicitly name alternatives or when-not-to-use it, but the primary use cases are well conveyed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false. The description goes well beyond annotations by disclosing that results is null until DONE, that a 404 indicates the run is not in the key's workspace, how progress_ratio is computed, and the screening result semantics. No contradiction exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but the complexity of the return object justifies the length. It is front-loaded with the core purpose, followed by endpoint, response shape, and edge-case notes. Every sentence carries useful information, and there is no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description carries the full burden of explaining the return value. It does this thoroughly: listing the run object's fields, status enum, error semantics, results behavior, and progress_ratio. For a complex polling tool, this is complete enough for an agent to call and interpret the response correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is only one parameter, id, and the schema already fully documents it as 'The eval run id returned by POST /v1/evals.' Since schema description coverage is 100%, the baseline is 3. The description adds endpoint context and key-scope information, but not substantial new parameter meaning beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Fetch one eval run's status, progress and — once DONE — its full results.' This clearly identifies the tool as a single-run polling/retrieval operation, distinguishing it from siblings like list_evals and compare_evals without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells the agent when to use this tool: 'poll this after creating a run.' It also clarifies that results are null until DONE, which guides polling behavior. It does not explicitly enumerate when not to use alternatives, but the usage context is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this as read-only, idempotent, and non-destructive, and the description adds valuable behavioral detail beyond them: the special 403 owner/admin requirement, 404 conditions, status semantics, the holdout_present meaning, and a closing 'Read-only, no spend' confirmation. No contradiction with the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Despite being long, every clause earns its place because there is no output schema to carry return semantics. The opening sentence front-loads the core purpose, then the return contract and edge cases are organized in a readable sequence. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one simple parameter, no output schema, and meaningful conditional behavior, the description is unusually complete: it covers the response shape, failure modes, permission requirements, feature-flag behavior, and the no-holdout case. An agent has enough information to invoke it correctly and interpret its results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents the only parameter, id, with 100% coverage, so baseline 3 applies. The description confirms the id appears in the endpoint path but adds no additional parameter-level format, constraints, or examples beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description starts with a precise verb and resource: 'Read a fine-tune's bake-off state, verdict ... and ledger-true spend'. It also states the decision context ('to decide whether the tuned model is worth deploying'), making the purpose unmistakable and distinguishing it from sibling mutation tools like start_fine_tuning_bakeoff.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides clear context on when to use the tool and important exclusions: requires an OWNER/ADMIN minting user, returns 404 when the job is not in the workspace or when the feature flag is off, and explains when a comparison cannot be offered. It does not explicitly name sibling alternatives, but the read/bakeoff naming plus the 'Unlike other GETs' note gives sufficient routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this read-only, idempotent, and non-destructive. The description adds meaningful behavioral detail that annotations cannot convey: live progress is fetched best-effort from the training backend and falls back to DB state, a SUCCEEDED job remains not-servable until deployed, and scopes are not enforced on this route. Nothing contradicts the annotations; instead it reinforces the read-only nature with 'Read-only, no spend'.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every sentence earns its place: purpose, endpoint, auth scope, full return contract, error conditions, backend fallback, deployment nuance, feature flag, and safety notes are all essential for correct invocation. The front-loaded sentence states the core purpose immediately, followed by structured details. No redundant filler exists.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description must disclose return fields, which it does exhaustively, including null states and enums. It also covers auth, errors, feature flags, live-data behavior, and deployment semantics, making it complete for an agent to call the tool and interpret the result correctly. Given the tool's complexity, nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single parameter `id` is already well described in the schema ('The job id returned by POST /v1/fine_tuning/jobs, not the provider_job_id'). The description does not need to add parameter semantics, so it sits at baseline 3. It does indirectly clarify id usage by returning `provider_job_id` in the response, but that is not directly about the input parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Get one fine-tuning job's live status, step progress, trained tokens, output model name, deployment state and billed cost'. It enumerates exactly what is returned and distinguishes itself from list/cance/other fine-tuning tools by targeting a single job. It also provides the exact endpoint, leaving no ambiguity about the operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit guidance is given: 'poll this after creating a job', which tells an agent when this tool is appropriate. It also communicates contextual constraints like 404 when the job is not in the workspace and feature-flag gating. It does not explicitly contrast with alternatives such as list_fine_tuning_jobs or cancel_fine_tuning_job, though the one-job scope implies the distinction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnly, openWorld, idempotent, non-destructive), the description discloses key behaviors: the exact response shape, outcome being null while ACTIVE or without artifact, feature-flag gating causing 404, OWNER/ADMIN key requirement causing 403, and foreign-id behavior. This is substantial context that helps the agent anticipate failures and interpret results.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every section earns its place: purpose and usage first, followed by the HTTP endpoint and auth note, then a full return payload since no output schema exists, then edge-case notes. It is dense but well-structured and free of filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has only one parameter, no output schema, and rich annotations, the description is complete: it specifies the endpoint, auth/scope requirements, response object, null semantics for outcome, and error cases. An agent has all necessary information to call and interpret this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully documents the single 'id' parameter, including its source ('from the runs list') and scoping behavior. The description adds no additional parameter-level meaning beyond what the schema provides, so the high schema coverage baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Fetch') and resource ('one online-RL (GRPO) run') and enumerates exactly what is returned: status, spend, and training outcome. This clearly distinguishes it from sibling tools like list_grpo_runs and get_grpo_run_weights.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says to use it to poll a run you started, which establishes the primary usage context. It does not explicitly name alternatives or say when not to use it, but the single-run scope and purpose make the intended use clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this as read-only, idempotent, and non-destructive, and the description adds substantial behavioral context beyond that: presigned URLs expire in 15 minutes, calls are audit-logged as weight exports, OWNER/ADMIN permission is required, feature-flag gating returns 404, and partial checkpoints are handled differently. This is rich and useful transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every sentence carries operational value: purpose, timing, endpoint, auth scope, response shape, error semantics, expiry, audit logging, and feature-flag behavior. It is front-loaded with the core purpose and usage condition before diving into return details and edge cases.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no output schema, the description fully specifies the return JSON structure, field meanings, sort order, partial-checkpoint semantics, error cases, permission requirements, expiry behavior, and storage-unavailable handling. An agent has everything needed to call this tool correctly and interpret its response.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully documents the single `id` parameter, including the requirement that the run be in a terminal status (COMPLETED, STOPPED, or FAILED). The description mostly repeats that status constraint; it adds error-case context (400/404/403) but does not add significant meaning about the parameter itself beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Get... presigned download links'), a specific resource ('a finished online-RL (GRPO) run's trained adapter files'), and a clear goal ('so you can self-host the weights'). It is easily distinguished from sibling tools like get_grpo_run, which is about run metadata, not weight export.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly states when to use it: after a run completes or stops with a partial checkpoint, and explicitly excludes ACTIVE and OVERBUDGET runs. It does not name an alternative tool to use instead, but the timing and error conditions provide strong contextual guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool readOnly, idempotent, and non-destructive. The description enriches this with concrete behavioral details: the GET endpoint, API-key scope `read`, the note that it is 'Not feature-flag gated', the OWNER/ADMIN key requirement with a 403 error, and the exact return shape. This significantly exceeds the annotation baseline.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and only includes high-signal details: use case, endpoint, auth, return shape, and relevant caveats. Every sentence earns its place, and the structured layout makes it scannable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description explicitly lists all return fields and states the response shape equals list rows. It covers the endpoint, required permissions, error condition, feature-flag status, and intended workflow, making the context complete for a single-fetch tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%; the sole `id` parameter is already described as workspace-scoped with error behavior. The description only mentions `{id}` in the URL path and does not add new meaning to the parameter, so it correctly stays at the baseline for fully covered schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: 'Fetch one model version's record', and it enumerates the contents (lineage, artifact, served model name, verdict, pinned hashes, adoption time). It also differentiates the tool from siblings by framing it as inspecting a specific training round before adopting, making its identity clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear usage context: 'use it to inspect a specific training round before adopting it', which implies this is a detail view for one version. It does not explicitly name alternatives like list_model_versions or state when not to use the tool, so it falls just short of full explicit guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and non-destructive behavior. The description adds substantial context: newest-first ordering, offset pagination mechanics, the fact that all sources are included, and the exact response shape. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core verb, resource, ordering, and use case, then supplies endpoint, response shape, and pagination notes. Every sentence adds value, and the structure makes the pagination and source-inclusion caveats easy to find.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully specifies the return JSON, including field types and enums. It also covers pagination and source semantics, leaving little ambiguity for an agent deciding whether and how to call this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents limit, offset, and verdict thoroughly. The description adds no parameter-specific meaning beyond the schema, but it does clarify that rows include all sources regardless of verdict filter, which slightly reinforces optional-filter behavior. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'List the workspace's human pass/fail grades... on logged requests, newest first.' It clearly differentiates from siblings like list_label_sets and create_label by focusing on ground-truth verdicts rather than label sets or label creation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit use context: 'use it to audit or export the verdicts every judge is measured against.' It doesn't name alternative tools or exclusions, but the purpose is specific enough to guide tool selection against the large sibling list.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already carry the safety profile (readOnlyHint, idempotentHint, non-destructive). The description adds substantial behavioral context beyond that: the owner/admin authorization requirement with 403, the 200-record cap with no pagination, newest-first ordering, that the endpoint is deliberately NOT feature-flag gated, and the comparable_to_parent discontinuity warning for interpreting deltas. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Dense but every sentence earns its place: purpose, use case, HTTP verb and API-key scope, complete return shape with field types and null semantics, ordering, cap, and three edge-case notes (feature-flag gating, auth, pagination). The field-by-field return listing is justified because no output schema exists. Core purpose is front-loaded before any detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a 1-parameter list tool. With no output schema, the description documents every return field including served_model null meaning, verdict type, the comparable_to_parent discontinuity flag, and the 200-cap. Auth requirements, gating behavior, and ordering are all disclosed. Nothing an agent needs to correctly call and interpret the results is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% — base_model's 'Filter to one base model's lineage (exact match)' is fully documented in the schema. The description's return-shape listing of base_model adds minor context about what 'lineage' refers to, but it contributes no parameter syntax, format, or filtering semantics beyond the schema. Baseline 3 is appropriate since the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'List the workspace's model-version chain — one immutable record per completed training round, pinning the judge, curriculum and holdout its verdict depended on'. The downstream use ('pick a version to adopt or roll back to') distinguishes it from siblings like adopt_model_version (which performs adoption) and get_model_version (single-version fetch). Purpose is precise and not confusable with any sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit use context: 'use it to review improvement history and pick a version to adopt or roll back to'. This is clear enough to route an agent toward listing rather than adopting or single-getting. However, it never names sibling alternatives explicitly (e.g., 'for a single version use get_model_version') or states when not to use it, so it falls short of explicit when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent/non-destructive annotations, the description discloses append-only semantics ('a refusal later overridden is still listed'), ordering tie-break ('created_at desc, id desc'), Cache-Control: no-store, and pagination behavior. It also documents truncation ('reason ... truncated to 500 chars') and the null last-page cursor, giving the agent a full behavioral picture.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense but every sentence earns its place: purpose first, then endpoint/scope, return shape, ordering, append-only caveat, cache behavior, and cost. No filler or repetition is present; it is long because the tool has meaningful behavioral specifics to disclose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully substitutes by specifying the exact response shape, cursor semantics, ordering, truncation, and pagination. It also covers auth scope, cache behavior, cost, filter semantics implied by examples, and append-only visibility — nothing necessary for a correct call is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter is already documented in the input schema. The tool description adds some operational context such as the pagination flow and the meaning of next_cursor, but it mostly restates schema-level semantics, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('Pages through the workspace's refusal ledger') and enumerates exactly what counts as a refusal with concrete examples, making it unmistakably distinct from sibling list tools like list_alerts or list_audit_tombstones. Saying 'newest first, for audit and compliance reporting' further pins the tool's purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for when this is appropriate ('for audit and compliance reporting') and notes the required API-key scope and cost ('Free'). It does not explicitly contrast it with alternative list/audit tools such as list_audit_tombstones or export_audit_log, so exclusion guidance is missing, but the intended use is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations by disclosing billable spend, the exact endpoint and API-key scope, per-type scan limits, quiet periods, pre-flight wallet gates, billing prefixes, adjudication semantics, and maxDuration. Annotations only indicate non-read-only, non-idempotent, open-world behavior; the description supplies the operational details an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every sentence carries operational value: purpose, endpoint, billing, return shape, auth, errors, scan limits, and review lifecycle. The most important scoping and cost warnings are front-loaded, and no filler is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has no output schema, so the description correctly explains the return object shape and zero-case behavior. It also covers auth, workspace constraints, billing, limits, quiet periods, calibration requirements, and error codes, making the tool safely callable by an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the only parameter, id, is already documented as a criterion id belonging to the key's workspace. The description reinforces this with the path and 404 condition, but it does not add substantial parameter-level meaning beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: it judges a bounded batch of unlabeled traffic with a criterion and queues FAIL verdicts as pending suspects for human review. It also frames its purpose as the fastest way to grow a judge's failure-label set from live traffic, which clearly distinguishes it from generic scan or review tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides strong contextual guidance: when to use it (live, unlabeled, unscanned traffic), prerequisites (OWNER/ADMIN, completed calibration for trace-unit criteria), and failure conditions (403, 404, 400, 402). It does not explicitly name alternative sibling tools or state when not to use it, but the context is clear enough for an agent to decide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the PUT endpoint, required API-key scope (platform:write), return payload shape, and the metering/wallet behavior. It also explains the 'clear' semantics via null, which complements the idempotentHint annotation without contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core action, followed tightly by endpoint, return shape, and rationale. Every sentence contributes essential information with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With one optional parameter and no output schema, the description covers all necessary call context: auth scope, model constraints, return shape, and clearing behavior. An agent has everything needed to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the single parameter is thoroughly documented in the schema including null/omitted/empty-string behavior and error cases. The description adds context about what the judge model is for, but no new parameter semantics beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Set (or clear)') and resource ('workspace's default judge model for eval runs that don't specify one'), making the tool's function immediately clear. It is easily distinguishable from the sibling get_judge_settings and other model-related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use the tool (for eval runs that don't specify a judge) and what models qualify (chat models from the platform catalog offered to the workspace). It also clarifies why provider keys cannot be used, though it does not name alternative tools or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes far beyond the sparse annotations by explicitly warning that this tool SPENDS MONEY, holds the commitment on the wallet, requires OWNER/ADMIN, is rate limited, is feature-flag gated, and returns specific failure codes (402, 403, 404, 429, 400) with their causes. This gives an agent a strong model of side effects and failure behavior before invoking it.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but every sentence carries load-bearing operational information for a high-stakes, 22-parameter, money-spending tool. It front-loads purpose, endpoint, spending warning, and return value, then groups notes logically by billing, validation, auth, rate limits, and refusal reasons.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no output schema but many non-obvious failure modes, the description supplies the return shape (201 with trigger_run_id), wallet implications, auth requirements, rate limits, feature gating, and the full set of 400 refusal reasons. This is comprehensive enough for an agent to call the tool confidently and diagnose failures.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of parameters with rich descriptions, so the baseline is 3. The description adds useful body-level conventions (camelCase, selected snake_case exceptions, validation error shape) but does not deepen individual parameter semantics beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
It states a specific verb ('start'), a precise resource ('online-RL (GRPO) training run'), and the mechanism ('improves a fine-tunable base model against a calibrated judge as reward'), with clear boundaries (hard reward and GPU-hour budgets). The REST endpoint and the explicit goal of turning logged traffic or the candidate queue into a trained adapter make it readily distinguishable from siblings like create_fine_tuning_job or stop_grpo_run.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear when-to-use signal ('use it to turn logged traffic or the candidate queue into a trained adapter') and notes prerequisites such as a fine-tunable model and calibrated judge. It does not explicitly contrast with alternative tools or state when not to use it, so it stops short of a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses far more than annotations provide: the exact POST endpoint, API scope, 200 response, transition to STOPPED, refusal of further scoring, orchestrator exit, immediate hold release, already-metered spend staying billed, feature-flag gating, auth requirements, 404 behavior, and that no body is read. This is comprehensive behavioral disclosure with no contradiction to annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: main purpose is front-loaded, followed by endpoint, response, behavioral side effects, error cases, and auth. No filler or redundancy; the structure makes the information easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool with no output schema, the description is complete. It covers the return value ('200 { ok: true }'), side effects, error states, auth, feature-flag gating, billing implications, and idempotency behavior. An agent has everything needed to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the single required 'id' parameter is already documented in the schema as 'Must be ACTIVE and belong to the workspace.' The description repeats ACTIVE/workspace context but adds no new parameter-level semantics beyond what the schema already provides, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: 'Stop an ACTIVE online-RL (GRPO) run and release its wallet hold immediately.' It clearly identifies this as the stop operation for GRPO runs, distinguishing it from sibling tools like start_grpo_run, get_grpo_run, and list_grpo_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an explicit when-to-use statement: 'use it to cut a run short when spend or results aren't what you expected.' It also documents key conditions such as 404 for non-ACTIVE runs and OWNER/ADMIN key requirements. However, it does not explicitly name alternative tools or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations by disclosing side effects: instrument changes void calibration, judge_prompt changes delete stored confusion rows, retiring removes the criterion from online scoring, and enabling online monitoring spends wallet funds capped by online_cap_usd. It also states the return shape. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: purpose first, then endpoint/auth, errors, return value, and high-impact consequences. Every sentence carries distinct information, and there is no filler or repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description specifies the 200 response shape, auth, expected error codes, side effects, and cost implications. Combined with the exhaustive input schema, an agent has everything needed to select and invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents every parameter including constraints like unit being create-only and judge_prompt voiding calibration. The description adds a helpful high-level grouping and money caveat, but does not substantially add parameter-level semantics beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource—'Update a criterion's prompt, judge model, population scope, online-monitoring settings or lifecycle status'—and names the exact fields and endpoint. This clearly distinguishes update_criterion from sibling tools like create_criterion, get_criterion, delete_criterion, and run_criterion_alignment.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description states the operation context (PATCH update), required API-key scope (evals:write), access restriction (OWNER/ADMIN), and error cases (403, 404, 400). It does not explicitly name sibling alternatives or say when not to use it, but the field list and lifecycle notes make the intended use clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations only provide generic hints (readOnlyHint false, idempotentHint false), so the description carries the burden and fully delivers. It discloses the API route, required auth scope, OWNER/ADMIN minting requirement, multipart request constraints, rate limits, feature-flag gating, billing implications, and unenforced scopes — far beyond the annotation signals.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence carries distinct operational value: purpose, return value, downstream usage, request format, auth, rate limits, feature flag, and billing. It front-loads the core purpose and return contract before diving into edge cases.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, yet the description provides the 201 response shape and the key field to use downstream. It also covers failure modes, rate limits, permissions, feature-flag gating, scopes, and billing, making the tool fully callable without external documentation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema already documents the only parameter, 'file', including the multipart field requirement and empty-file behavior. The description adds surrounding error messages and operational context, but it does not significantly enrich the parameter semantics beyond what the schema already states.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the exact operation — uploading a JSONL training or validation file for fine-tuning — and explains the returned provider_file_id is what job creation references. This clearly separates it from sibling tools like list_training_files and create_fine_tuning_job by tying it to the upload lifecycle step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly states when this tool fits in the workflow: before creating a fine-tuning job, and it even specifies which fields in POST /v1/fine_tuning/jobs should use provider_file_id. It does not explicitly list when-not-to-use or alternatives, but there is no same-purpose sibling, so the context is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations are mostly negative (readOnlyHint false, idempotentHint false, destructiveHint false), so the description carries the full burden and succeeds. It discloses that the tool MOVES PRODUCTION TRAFFIC, replaces the alias target, clears any live canary split, stamps adopted_at only on first adoption, and is audit-logged. Error semantics, feature-flag behavior, auth requirements, and response shape are also covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but densely informative, with the core purpose front-loaded in the first sentence and high-impact side effects flagged explicitly. Some details like the Zod failure shape are somewhat niche, but they are still useful for error handling. The structure could be tightened, but every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a high-risk mutation with no output schema and sparse annotations, the description is unusually complete: it covers purpose, HTTP method, API-key scope, required key level, response format, side effects on canary, error codes, feature-flag behavior, and audit implications. An agent has enough information to decide whether and how to call this tool without needing external docs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaningful operational context: id must correspond to a deployed version with weights, and aliasName must reference an existing workspace alias. It also maps failure modes to each parameter (400 for undeployed version, 404 for foreign id or unknown alias), which helps the agent validate inputs before calling.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource — 'Point a model alias at a deployed model version' — and clearly identifies the two use cases: promoting a trained round into production or rolling back. It also distinguishes itself from related alias/GET tooling by noting this is the audited repoint operation and that it is feature-flag gated unlike the GET routes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use it: 'use it to promote a trained round into production or roll back to an earlier one.' It also gives a practical exclusion signal by noting the fineTuning flag off yields a 404 'unlike the GET routes,' helping an agent avoid attempting the call when feature-gated. It does not name exact sibling tools as alternatives, but the guidance is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false and idempotentHint=false, so the description's real contribution is cost and side-effect disclosure: 'SPENDS MONEY' and 'one metered rewriter call (billed under assist:iterate) plus the queued alignment run'. It also exposes partial-failure behavior (nothing created, only the rewriter call spent) and the deliberate single-round design, which no annotation conveys. Nothing contradicts the annotation hints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The definition is long but front-loaded: the first sentence carries purpose, mechanism, and when-to-use. The exhaustive return-field list earns its place only because no output schema exists, but the billing warning is stated twice ('SPENDS MONEY' and 'SPENDS THE WALLET'), which is redundant and could be tightened.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a money-spending, non-idempotent, queueing operation with one required parameter and no output schema, everything an agent needs is present: return shape (201 with full criterion object), error conditions (404/400 with their triggers), cost, auth scope (evals:write), post-call expectations (loop yourself; adoption stays human), and maxDuration. No material gap remains.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single parameter already has a descriptive schema entry ('Criterion id (must belong to the key's workspace)'), so the baseline is 3. The description only restates the parameter's role via the endpoint path criteria/{id} and adds no new semantic meaning beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Runs one auto-improvement round on a judge criterion' and decomposes it into three concrete steps (mines tune-half disagreements, rewrites the judge prompt,creates a successor DRAFT criterion with alignment queued). This clearly differentiates it from siblings like run_criterion_alignment and create_criterion by spelling out what auto-improve means as a distinct post-alignment operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use guidance is given: 'use it when a calibrated judge still disagrees with your grades and you want a better candidate without hand-editing the prompt.' When-not conditions are enumerated as 400 triggers (alignment in progress, fewer than 80 judged rows, zero disagreements, unusable rewriter), and the 'Deliberately single-round: loop it yourself' note plus 'adoption stays a human act' tells the agent how to orchestrate multi-round use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description is exceptionally transparent about behavior beyond annotations: it warns that money is spent, explains that billing starts once running, states the prepay requirement, notes that the price is frozen at create time, and explains how to stop billing via PATCH or DELETE. This goes far beyond the annotations and gives the agent critical operational knowledge.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but densely informative, front-loading the core purpose and billing warning. There is slight redundancy between 'billing starts as soon as it is running' and the later 'SPENDS MONEY' and 'MONEY' sections, but every substantive detail earns its place given the financial and operational stakes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a high-complexity tool with 11 parameters, no output schema, and significant cost implications, the description is remarkably complete. It covers the success response, polling flow, auth requirements, validation failures, catalog constraints, prepay wallet gate, and billing-stop mechanisms. An agent has enough information to call this tool correctly and anticipate consequences.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides 100% parameter coverage, so the baseline is 3. The description adds meaningful cross-parameter context, such as the prepay calculation involving min_replicas x gpu_count x hourly price, the required-field validation after alias lifting, and catalog/replica-range failure modes. This adds value beyond the schema without duplicating it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description begins with a specific verb and resource: 'Provisions a new dedicated inference endpoint', and expands on what that means (reserved GPUs, frozen per-GPU-hour price, guaranteed capacity, custom weights, predictable latency). It is clearly distinct from sibling tools like get, update, delete, and list_dedicated_endpoints.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool: 'use it for guaranteed capacity, custom fine-tuned weights, or predictable latency'. It also gives important usage context around billing, auth scope, and polling for status, but it does not explicitly name alternative tools or say when not to use it, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations are sparse (readOnlyHint false, openWorldHint true, idempotentHint false, destructiveHint false), and the description carries the behavioral burden thoroughly: it discloses that the call 'SPENDS MONEY', requires the key's minting user to be workspace OWNER/ADMIN, returns 201 with status PENDING, requires polling until DONE|ERROR|CANCELLED, and enumerates 400/403/404/402/422 error paths. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but it is front-loaded with the core purpose and then organized into labeled operational sections ('Returns', 'Notes', 'Screening-specific'). The density is justified by the tool's complexity, though some redundancy in the money warnings and error enumerations prevents a perfect conciseness score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Because there is no output schema, the description supplies the return shape (status PENDING, results null, progress_ratio 0, arms[]), the polling/gating pattern, authentication requirements, cost model, error schema, and screening-specific failure modes. This is unusually complete for a tool with 14 parameters and no structured output definition.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaningful operational semantics beyond the schema: snake_case-to-camelCase conversion, best-of-N cost multiplication, screening-mode parameter precedence, and creation-time funds gating. It does not replace the per-parameter schema descriptions, but it augments them with cross-cutting details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Queue an eval run', and immediately enumerates the three run modes (pairwise comparison, absolute criterion, screening). This clearly differentiates create_eval from siblings such as list_evals, get_eval, compare_evals, and create_criterion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear context for when to use the tool ('so a customer can measure a model, prompt, tool or index change before shipping it') and explains the mode alternatives. It does not explicitly name sibling tools or state when not to use create_eval, so it stops short of full when/when-not routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes far beyond the annotations by disclosing that the call 'SPENDS MONEY' from the workspace wallet, that a prepay runway is required or the call fails 402, and that the final charge is metered from real trained tokens. It also discloses role requirements, rate limits, the 'scopes not enforced today' caveat, snake_case/camelCase conventions, and exact 400/404 trigger conditions. No contradiction with annotations — readOnlyHint=false, idempotentHint=false, and openWorldHint=true are all consistent with billable, externally-visible side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the two most decision-relevant facts — what the tool does and 'SPENDS MONEY' — before any schema-level detail, and every section carries operational value. It is not a 5 because it is a single dense wall of text that repeats information already present in the schema (e.g., hyperparameter bounds are summarized again), so it could be tightened or lightly structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a money-spending mutation with 12 parameters, nested objects, and no output schema, the description covers the response shape (201 {id, provider_job_id}), the mandatory follow-up (poll GET for status and fine_tuned_model), all major error codes with their triggers, the workspace-membership constraint, and the feature-flag gate. Nothing an agent needs to invoke it correctly is left to inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so a baseline of 3 applies, but the description adds genuine cross-parameter semantics absent from the schema: training_file_id is 'Ignored when training is present,' top-level keys accept camelCase while nested keys are snake_case only, camelCase wins on base_model if both are present, validation is what enables a later bake-off, and spec_draft_hyperparameters apply only when method='spec-draft'. These precedence and conditional-usage rules meaningfully help an agent construct a valid request.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Start a supervised (SFT/LoRA) or spec-draft fine-tune of a catalog base model on an uploaded file or a workspace dataset.' It names both training modes and both data-source flavors, and makes clear this is the creation entry point versus sibling list/get/cancel tools without needing to inspect them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit preconditions and failure gates: wallet runway (402), OWNER/ADMIN requirement (403), feature-flag gating (404), per-workspace rate limit (429 with Retry-After), and the follow-up flow ('Poll GET /v1/fine_tuning/jobs/{id} for status'). It stops short of a full 5 because it never explicitly contrasts alternative training-related creation tools (e.g., start_grpo_run or create_batch), so the when-not-to-use guidance is implied through failure conditions rather than named alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds substantial behavior beyond the annotations: row ordering, silent cap at 50000, truncated indicator semantics, empty-not-error behavior, Cache-Control no-store, and exact CSV/JSON shapes. Annotations already declare readOnlyHint=true and idempotentHint=true; the description fully aligns with these, noting the API key scope and read-oriented export behavior. No contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is complete and front-loaded with the core export purpose, then format and endpoint details, then notes. It is long but each sentence carries functional value (ordering, truncation, empty set, cache), and the structure keeps the most decision-relevant information near the top. It could be tightened slightly, but for a tool with four parameters and two output formats this is appropriately dense.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given four optional parameters, full schema coverage, no output schema, and annotations covering safety, the description covers the necessary invocation details: endpoint, auth scope, exact JSON return fields, CSV header and Content-Disposition, truncation indicator, ordering, cap, empty-set behavior, and caching. An agent has enough to select, call, and interpret results without guessing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema coverage is 100%, so the baseline is 3; the description adds useful behavioral context around parameters: limit is silently capped at 50000 (schema says the same, but description reinforces the default and cap effect), since/until constrain row timestamp, and format's default is csv, matching the schema. It slightly clarifies that the date bounds are inclusive and HTTP 400 on unparseable values, which is already in the schema, but also explains how the format changes the response (JSON vs CSV attachment).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a precise verb (export) and resource (this workspace's audit rows), and details the hash-chain fields (seq, prev_hash, row_hash) that distinguish it from other audit-related tools. It clearly differentiates export_audit_log from siblings such as list_logs/export_logs and the audit tombstone/verification tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description states the tool returns audit rows for verification of hash consistency, says it exports as CSV or JSON, and notes the empty-set behavior when the log store is unavailable. It explains the accepted formats and response details, though it does not explicitly enumerate when to prefer this over sibling tools or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool as read-only, idempotent, and non-destructive; the description adds valuable behavior beyond that: the exact returned field list, the 404 'Criterion not found' condition for ids outside the workspace, and the note that trust/drift derivations are identical to GET /v1/criteria.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and API endpoint before the detailed field list. The field enumeration is long but justified because there is no output schema; still, it could have been slightly more compact without losing value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully compensates by enumerating the return fields, documenting the 404 error case, and noting the read scope. For a single-resource read operation, nothing essential is missing for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with 'Criterion id.', so the baseline is 3. The description enriches this by showing the id is used as a path parameter in GET /v1/criteria/{id} and clarifying that a missing or out-of-workspace id results in a 404 rather than an empty result.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear verb and resource: 'Fetch one criterion' with its calibration metrics, trust verdict, and derived intervals. This distinguishes it from sibling tools like list_criteria or get_criterion_certificate by specifying exactly what payload this single-criterion fetch returns.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly indicates this is the tool to retrieve one specific criterion by id, including metrics and trust verdicts. It does not explicitly name alternative tools or exclusion conditions, but the 'one criterion' phrasing and the note that derivations match GET /v1/criteria provide usable context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent annotations, the description discloses the hourly cache and how 'force' bypasses it, the metered model-call billing on fresh clustering, deduplication of online FAILs and pending suspects, the 'without_reason' exclusions from clustering, and the 400 error for invalid window_days. These are meaningful behavioral details that annotations alone cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but dense and well ordered: purpose, endpoint/scoping, full return shape, and then cache/error/billing notes. Every section earns its place because there is no output schema to rely on. Minor redundancy like 'Cache-Control: no-store' could be dropped, but it does not detract much.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully specifies the response structure, including the cache flag, criteria array, cluster fields, and the representative rationale example. It also covers error behavior, billing, and edge cases such as criteria with fewer than four reasons. An agent has everything needed to call the tool and interpret results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents both parameters fully (defaults, ranges, and force's cache-bypass effect), so the baseline is 3. The description adds further value by specifying the cost implications of force=true, the 40-reason cap per criterion, and the exact 400 error message when window_days is out of range.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a precise resource ('live production failures grouped into systemic causes per criterion') with a clear action ('See') and the user purpose ('find what to fix first rather than reading failures one by one'). It is not a tautology of the title and is easily distinguishable from sibling tools that scan or list individual failures.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: use this to get an aggregated, prioritized view of production failures instead of inspecting them individually. It does not explicitly name an alternative tool or a when-not-to-use condition, but the intended scenario is unmistakable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description significantly expands beyond annotations by disclosing Cache-Control: no-store, 200/401 semantics, the camelCase vs snake_case discrepancy with every other /v1 route, the meaning of null in loggedConversations, and the Accept: text/plain behavior. None of this contradicts the readOnly/idempotent/openWorld annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but dense and front-loaded with a clear purpose statement before diving into the endpoint and response details. Every sentence carries substantive information (null semantics, casing, cache behavior, auth meaning, header variant). Only minor redundancy: the casing difference is highlighted twice, but the second mention adds cross-route context, so it earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no parameters, the description bears the full burden of explaining return values and behavior. It fully specifies the JSON shape with field types, nested objects, enum values for next.action, null semantics, the text/plain alternative, and success/error meaning — nothing an agent needs to invoke or interpret this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema is empty (0 parameters), so the baseline is 4. The description adds value by documenting the optional Accept: text/plain header that switches the output format to flat snake_case key=value lines, and example like grades_total=12, next_action=grade.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific and memorable purpose: 'Answer where am I and what should I do next?' and enumerates the exact resources covered (workspace identity, logging state, traffic counts, grade counts, judge calibration, next step). It distinguishes itself from sibling action-tools by presenting a one-call status/setup overview rather than a mutation or query of a specific resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear usage context: it is the 'cheapest way to check that an API key is live and which workspace it belongs to', and is the one-call for setup status. It does no explicit name alternatives or exclusions, but the usage intent is unambiguous and sufficient for an agent to select it over the many sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and does so thoroughly: it discloses the bare JSON array response, the 100-object cap, the stored-vs-live status caveat, and the feature-flag failure mode. This goes well beyond a minimal definition.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and route, and the Returns/Notes structure makes the dense information scannable. The field enumeration and status list are long but necessary given the absence of an output schema, so it earns a high score rather than a perfect one.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the full response shape, field semantics, status enum, nullable fields, pagination cap, failure mode, and the live-refresh caveat. Nothing essential is missing for an agent to select and invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so the baseline is 4; the description confirms no parameters are needed beyond the implicit workspace scope. There is no parameter ambiguity to resolve.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action and resource: listing the workspace's batch inference jobs, newest first, with status, request counts, and billed cost. This clearly distinguishes it from sibling tools like create_batch, get_batch, and cancel_batch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly identifies the scope (this workspace), the HTTP route, and the read-only API-key scope, and notes the feature-flag 404 that signals when the capability is unavailable. It also implicitly routes the agent to GET /v1/batches/{id} for live status refreshes, but it does not explicitly enumerate when to prefer get_batch versus list_batches.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint/idempotentHint annotations, the description discloses the feature-flag gating (404 when off), the OWNER/ADMIN requirement even for GET (403), the fact that credentials are encrypted at rest and never returned (only a prefix), the auth_prefix display semantics, name sorting, and the exact HMAC_SHA256 signing scheme. This is rich behavioral context that neither annotations nor the empty schema provide, and it does not contradict the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but front-loaded: purpose first, then endpoint/scope, a structured Returns block, and Notes on gating, auth, and security. Every sentence carries load-bearing information, though minor overhead like the 'snake_case:' label and dense parentheticals keep it just short of a perfect 5.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and zero parameters, the description carries the full burden and succeeds: it specifies the HTTP endpoint, API-key scope, every return field with its semantics, the two primary failure modes (404/403), and the security guarantees. Nothing an agent needs to call or interpret the tool is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With zero parameters, the baseline is 4; there is no parameter meaning for the description to add. The description productively uses the space instead to document the return structure in detail, which is the semantically relevant information for this tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb (Lists) with a precise scope: agent tools declared for training environments (the egress allowlist) plus the per-workspace verification secret. It is immediately distinguishable from sibling register_env_tool and delete_env_tool, and the audit purpose is stated outright.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly frames the intended use ('use it to audit which endpoints and credentials environments may call'), which gives an agent a clear trigger condition for selecting it. It does not name alternatives or state when-not-to-use conditions, but the audit framing makes the distinction from the register/delete siblings evident.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this as read-only/idempotent/non-destructive, but the description adds substantial behavioral context: ledger-true spend figures, newest-first ordering, 50-run cap, outcome only fetched for the 10 newest non-ACTIVE runs, feature-flag 404 behavior, and OWNER/ADMIN permission requirement. This goes well beyond annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but dense, and every section earns its place: purpose, endpoint, response shape, ordering, limits, permissions, and error behavior. It is front-loaded with the primary use case. Slightly verbose, but justified by the absence of an output schema to document the response.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully documents the JSON response shape, field semantics, ordering, limits, and caveats. It also covers auth, feature-flag failure, and ledger accuracy. An agent has everything needed to invoke the endpoint and interpret results correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. The description correctly avoids inventing parameter guidance and instead documents the fixed request scope and response behavior, which is appropriate for a parameterless endpoint.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description opens with a specific verb and resource: it lists the workspace's online-RL (GRPO) training runs. It also clearly distinguishes from sibling tools like get_grpo_run, start_grpo_run, and stop_grpo_run by framing this as the monitoring/list operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description states when to use the tool: to monitor training and decide whether to start another run. It doesn't explicitly name alternatives or exclusions, but the list-vs-single-run distinction is clear enough given the sibling names and the stated purpose.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses a remarkable amount beyond annotations: update-by-name re-enabling, camelCase request vs snake_case response with no aliasing, exact error conditions (400/403/404), encrypted credential storage, omit-vs-clear authHeader behavior, registration-time-only host validation, DNS rebinding caveat, audit event writing, and the feature-flag gate. It does not contradict the annotations; the operation is a write, non-idempotent, non-destructive call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense and long, but every clause carries useful information: purpose, HTTP route, auth scope, response shape, error modes, prerequisites, and security caveats. It front-loads the core purpose before diving into details; a bulleted layout would improve scanability, but the current structure is well-ordered and free of filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, yet the description supplies the 201 response object with field names, covers success and error paths, documents authorization and feature-flag requirements, explains updating vs creating, and flags the security limitation around DNS rebinding. For a registration tool with this many edge cases, nothing crucial is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Since schema coverage is 100%, the baseline is 3, but the description adds meaningful cross-cutting semantics: the request/response casing mismatch, update defaults and re-enable behavior, authHeader clearing semantics, encrypted-storage failure mode, and host-validation limitations. These details go beyond the individual parameter descriptions and help an agent avoid common mistakes.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Registers (or updates, by name) an agent tool that training environments for this workspace are allowed to call.' It also captures the upsert nature ('or updates, by name') and the essential consent-granting purpose, making it clearly distinct from sibling operations like list_env_tools and delete_env_tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description makes the operational context explicit: it is the consent grant naming the HTTPS endpoint, credential, read-only flag, and call cap, with prerequisites such as OWNER/ADMIN minting user and the fineTuning feature flag. It does not explicitly name alternative tools or say when not to use it, but the sibling names and the 'Registers (or updates)' framing give an agent sufficient guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate idempotent and destructive, and the description adds substantial context: metering stops, status becomes CANCELLED, cancel is authoritative despite backend lag, backend errors do not block local cancel, permissions require OWNER/ADMIN minting user, 404/400 cases, feature-flag gating, and scopes not enforced. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action and consequence, then organized caveats (return, status, errors, permissions, backend behavior). The length is justified because every sentence adds an operationally relevant detail for a cancellation endpoint.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With only one parameter and no output schema, the description supplies everything needed to call it correctly: full HTTP semantics, success response, failure modes, auth context, idempotency, backend behavior, and feature-flag behavior. Nothing material is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single id parameter is already documented as 'The job id.' The description references {id} in the DELETE path but does not add much semantic detail beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Cancel a queued or running fine-tuning job'), includes the HTTP path, and explicitly distinguishes this from deletion ('This is a CANCEL, not a delete — the job record remains listed'). It is clearly differentiated from sibling tools like cancel_batch or delete_alias.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit conditions for use: queued or running jobs, and when-not: SUCCEEDED or FAILED jobs return 400, while already-CANCELLED jobs are idempotent. It also clarifies it is not a delete operation, which is the main alternative behavior an agent might confuse.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations, disclosing upsert semantics, overwrite behavior, pass-clears-failures, OWNER/ADMIN authorization requirements, and downstream side effects on judge calibration and recalibration slices. It also specifies 201, 400, and 403 outcomes, giving an unusually complete behavioral picture.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is substantial but every sentence earns its place: purpose, endpoint, auth, return shape, upsert behavior, side effects, and validation errors. It is front-loaded with the core purpose and organized with clear notes, avoiding redundancy with the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with no output schema, the description provides the return payload, auth requirements, side effects, and validation behavior. Combined with a fully documented input schema, an agent has everything needed to invoke the tool correctly and anticipate consequences.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents every parameter. The description adds meaningful behavioral semantics beyond the schema: re-labeling replaces the prior judgment, the newest wins, and a 'pass' clears prior failure attributions — all of which clarify how verdict, critique, and request_id interact.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a precise verb and resource: recording a pass/fail verdict on a logged request or agent run, and explicitly links it to ground-truth judge calibration and training rewards. It is clearly differentiated from read-only siblings like list_labels and from pairwise eval labelling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description establishes clear context for when to use this tool: when a human or downstream system needs to record the authoritative verdict on a logged request. It explains scope choices and side effects, though it does not explicitly name alternatives or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this as a non-read-only, non-idempotent write operation, and the description adds substantial behavioral context beyond that: the HTTP endpoint, API-key scope, exact 201 response shape, the 'NOT frozen yet' state, minimum/maximum size limits, verifier-source exclusion, and the subsequent freeze step. This gives the agent a thorough understanding of side effects and constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but well-structured, with the core action front-loaded followed by endpoint, return shape, constraints, and next steps. Every sentence carries actionable information; the explicit response object and error-message snippets earn their place, and the 'Notes:' section keeps constraints scannable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description supplies the full 201 return object; with only sparse annotations, it also provides endpoint, auth scope, size limits, verifier exclusion, and the follow-up freeze workflow. Combined with 100% schema coverage for parameters, an agent has everything needed to invoke this tool correctly and interpret its result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaningful behavioral semantics: it explains the conceptual distinction between 'explicit request ids or the newest N grades', reinforces that verifier-sourced labels never count, and highlights the minimum distinct graded requests requirement. These details clarify how to choose and validate parameter values beyond the schema text.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource ('Create a golden set from human-graded requests') and clearly distinguishes this tool from siblings like freeze_label_set and attach_label_set by emphasizing the set is 'NOT frozen yet' and must be frozen before attaching. It is immediately clear what the tool produces and where it fits in the workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear sequencing context — this is the 'first step' toward a frozen calibration set, and 'freeze the set before attaching it to a criterion' — so an agent understands when to use it relative to freeze_label_set and attach_label_set. It does not explicitly name alternative tools or give direct when-not-to-use conditions, but the workflow guidance is strong.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already flag destructiveHint=true, but the description adds extensive behavioral context: irreversibility, cascading deletion of samples/outputs/verdicts, preservation of models/judges/criteria, OWNER/ADMIN requirement, 400/403/404 error conditions, return payload, post-deletion GET behavior, and 'No spend'. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence carries operational value: action, endpoint, auth scope, response, preconditions, alternative action, errors, and side effects. Although long, it is front-loaded with the core purpose and then systematically adds necessary caveats for a destructive operation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive tool with no output schema, the description is exceptionally complete. It covers prerequisites, authentication, error codes, cascading effects, unaffected resources, return value, and post-deletion observable behavior. Nothing needed to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single parameter 'id' is already documented as 'The eval run id'. The description reinforces that the id identifies the eval run and explains 404 behavior for a missing id, but adds no meaningfully new parameter semantics beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a precise verb ('Permanently delete'), a specific resource ('finished eval run'), and the exact state scope (DONE|ERROR|CANCELLED). It clearly distinguishes this from sibling delete tools like delete_criterion or delete_dedicated_endpoint by naming the eval-run resource and its completion states.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use it ('cleaning up runs a customer no longer needs as evidence'), when not to use it (PENDING|RUNNING refused), and what to do instead (call POST /v1/evals/{id}/cancel first). This gives the agent complete routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the strong readOnly/idempotent/non-destructive annotations, the description discloses material behavior: verification is null until the first nightly run, the response schema and fields, Cache-Control: no-store, API-key scope, and the platform-global chain scope. These add real operational context an agent cannot infer from annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence carries necessary information: the operation, endpoint, auth, full return shape, null behavior, and the key distinction from export_audit_log. The key purpose is front-loaded and the structured return example is easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no parameters, the description fully compensates by specifying the complete response object, the null-before-first-run behavior, the tombstone list, and the relationship to audit export. Nothing needed to invoke and interpret this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and an empty schema, so there is no parameter ambiguity to resolve. The description still supplements the schema by documenting the endpoint and authentication scope, meeting the baseline for a parameterless tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description starts with a specific verb and resource: 'Read the latest nightly whole-chain integrity verification of the audit log... plus the tombstone list.' It clearly identifies this as the platform's integrity statement and distinguishes it from export_audit_log by noting that own-row hashes come from GET /v1/audit/export.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: this reports the platform-global integrity verification, not your own rows' hashes, which are handled by GET /v1/audit/export. It doesn't explicitly contrast with sibling list_audit_tombstones or verify_document, so alternative guidance is partial rather than exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior, but the description adds meaningful behavioral context beyond that: null means house default/auto, Cache-Control is no-store, runtime precedence order, and the screening edge case where a default that would judge its own sibling model is swapped. This is exactly the kind of disclosure agents need.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but tightly organized: purpose first, then endpoint and scope, then return shape, then Cache-Control, then precedence and edge-case notes. Every sentence adds information and there is no filler or repetition of the title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, read-only GET with no output schema, this description is remarkably complete. It documents the exact JSON shape, the semantics of null, the API-key scope, HTTP caching behavior, runtime precedence, and even the screening exception. An agent can understand the tool's behavior without needing additional context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the input schema is an empty object, so there is no parameter ambiguity to resolve. The description goes further by explaining the meaning of the two return values, which compensates for the absence of an output schema, though this is more output semantics than parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb 'Read' and names the exact resources: the workspace's default judge model and the platform's house default. It clearly distinguishes itself from the sibling set_judge_settings by being read-only, and it never merely restates the tool name.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description makes the read-only usage context clear and the contrast with set_judge_settings is implicit through the GET method and 'read' API-key scope. However, it does not explicitly state when to prefer this tool over alternatives or list exclusions, so it stops short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior. The description goes further by disclosing ordering, inclusion of aborted partial runs, the 409 flat-shape error, the need for request logging, and the absence of pagination/filters — all genuinely useful behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the primary purpose and use case, then efficiently covers endpoint, auth scope, return shape, ordering, errors, and exclusions. Every sentence carries information; none is filler, and the structure guides the agent from purpose to behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the detailed return JSON is essential and provided. The description covers prerequisites, error responses, ordering, trace grouping, differences from alternatives, and limitations, making the tool fully usable without further inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already describes traceId fully, including its X-Omnia-Trace-Id origin and the 404 behavior, so the parameter meaning is completely covered. The description adds no new parameter-level semantics beyond what the schema provides, justifying the baseline score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the specific verb and resource: 'Fetch every logged step of one agent run or conversation' grouped by trace ID. It distinguishes this from related log tooling by emphasizing the ABORTED partial rows and the trace-scoped granularity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says to use this for error analysis of a multi-step run, contrasts itself with /v1/logs, and notes that ABORTED partial rows are the finding. It also mentions prerequisites and error conditions, leaving little ambiguity about when this tool is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With annotations already providing readOnly/idempotent/destructive hints, the description adds substantial behavioral context: one-label-per-pair overwrite semantics, companion DELETE/GET endpoints, the returned pairwise calibration block, 400/404 error cases, API-key scope, 'No spend', and the integrity rule about not fabricating preferences.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but well-organized, front-loading the core purpose before covering endpoint details, return shape, mutation semantics, error handling, and constraints. Every sentence earns its place by communicating a distinct fact an agent needs to call this tool correctly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is complex, yet the description covers selection criteria, request semantics, response contents, side effects, error handling, authentication scope, and cost. Since there is no output schema, the description compensates by explicitly describing the returned pairwise calibration block and linking it to get_eval_pairwise.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and each parameter already has a descriptive explanation, including the meaning of verdict values and sample_index ordering. The description adds contextual framing like candidate-vs-baseline and overwrite behavior, but it does not need to restate parameter formats, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Record a HUMAN's verdict on one candidate-vs-baseline pair of a comparison run.' It clearly frames this as calibration evidence and differentiates it from read-side siblings like get_eval_pairwise by emphasizing the write action and human-labeling role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly scopes usage to comparison runs, states that criterion runs are refused, and warns that labels must come from actual user verdicts, never invented. It also cites the 30-label corrected-rate threshold, giving the agent concrete conditions for when this tool should be invoked.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the readOnlyHint/idempotentHint annotations by detailing newest-first ordering, the exact response envelope, keyset pagination semantics, 400 error conditions, and Cache-Control: no-store. It also clarifies that alerts carry notification payloads, which is non-obvious behavioral context. No contradiction with annotations exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense but structured effectively: core purpose first, then endpoint/auth scope, then return shape, then pagination/error notes. Every sentence earns its place, and the most important behavioral facts are front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description fully defines the return structure including alert fields, payload subfields, and next_cursor. It also covers pagination, validation failures, and caching, making the tool safely callable end-to-end without needing external documentation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaningful extra semantics by explaining how next_cursor is passed back as cursor until null, and by specifying when since and limit cause 400 errors; these constraints enrich the schema's otherwise adequate parameter descriptions without redundancy.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description begins with a precise verb-resource pair: 'List every alert this workspace has fired, newest first, with the payload the notification carried.' It clearly distinguishes alerts as their own resource type from sibling log/tombstone/refusal tools, gives the HTTP endpoint, and explains the practical purpose (reacting to quality, cost, or drift events).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for when to use this tool: to retrieve fired alerts with their payloads for pipeline reactions, and it documents pagination and error behavior. It does not explicitly name sibling alternatives or exclusion conditions, but the resource distinction from list_logs, list_criteria, and list_refusals is strong enough for an agent to select it correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark it read-only/idempotent, and the description adds the endpoint, API-key scope, sorting behavior (name ascending), and nuanced null semantics for last_evidence_run_id. It also exposes the full response envelope so the agent knows exactly what to expect beyond the safety profile.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is structured with a concise purpose sentence, endpoint/auth note, explicit return shape, and a focused note about null behavior. Each part provides actionable information without duplication, so it is appropriately sized for a rich read-only list operation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema provided, the description compensates with a full field list, enum values, null semantics, sort order, and authentication scope. This is complete enough for an agent to invoke the tool and interpret the result correctly in a workspace context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, and the schema coverage is 100% (empty schema). The baseline of 4 applies because there are no parameter semantics to clarify; the description does not need to compensate for undocumented parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('List'), resource ('this workspace's model aliases'), and enumerates exactly what is included (target, canary split, quality-gate config, evidence policy, authoring eval run). This clearly distinguishes the read-only listing from mutation siblings like upsert_alias and delete_alias.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear context for when the tool is relevant: inspecting the stable names your code calls and their current routing details. It does not explicitly name alternatives or state when not to use it, but the read-only 'GET /v1/aliases' phrasing plus sibling names make the usage context unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Even though annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint false, the description adds substantial behavioral detail: the API-key scope ('read'), the response format sorted by seq ascending, the platform-global nature, and Cache-Control: no-store. This enriches the agent's understanding beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact, front-loaded with the core purpose, then gives endpoint, scope, return format, and a critical scoping note. No sentence is redundant or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless read-only tool with no output schema, the description covers everything needed: what it returns, the field types, sort order, authentication scope, and the cross-workspace consistency caveat. An agent can invoke it correctly without further context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters and the schema is empty, so there are no parameter semantics to clarify. The description correctly documents the endpoint and return fields, satisfying the baseline for a parameterless tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('List'), a precise resource ('acknowledged audit-chain gaps (tombstones)'), and the reason for using it ('so a known loss can be distinguished from tampering'). It also includes the HTTP endpoint and return shape, making it unmistakably distinct from siblings like create_audit_tombstone or get_audit_verification.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear use case: distinguish a known audit loss from tampering. It also notes the audit chain is platform-global and identical across workspaces, which is relevant context for when to call it. It does not explicitly name alternatives or state when not to use it, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, and idempotentHint, so the safety profile is covered. The description goes well beyond that by explaining the endpoint, API-key scope, the list response shape, the difference between trust and tier, and the derived nature of drift_status/drift_signal with detailed meanings for each enum value. This is substantial added behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Although long, the description is dense and every section earns its place: purpose is front-loaded, the return shape is included because there is no output schema, and the notes clarify ambiguous computed fields. The structure is logical and avoids repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With zero parameters, rich annotations, and no output schema, the description carries the full burden of explaining the response and key semantics. It enumerates every returned field and explains trust, tier, drift_status, and drift_signal sufficiently for an agent to interpret results correctly. Nothing essential appears missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and schema coverage is 100%, so the baseline is 4. There are no parameter semantics to explain, and the description does not need to compensate for any schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'List this workspace's judge criteria' and enumerates what is included (calibration metrics, trust verdict, drift status, online-monitoring config). It also states the intended decision ('to see which judges are proven enough to gate on'), which clearly distinguishes this list-all tool from siblings like get_criterion or list_criterion_templates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear context and intended use case: reviewing judges to determine which ones are safe to gate on. It does not explicitly name alternatives or state when not to use it, so it stops short of full exclusion guidance, but the purpose is concrete enough for an agent to select it appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint/idempotentHint annotations, the description discloses that the endpoint is static and free, requires only authentication, and returns a fixed set of 11 templates. It also explains important semantic behaviors: universal templates are safe unscoped, non-universal ones should be scoped, and trace-unit templates need trace-scoped labels. This is rich behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every sentence earns its place: purpose, endpoint, auth scope, full return shape, template count, and behavioral notes. The most decision-relevant information is front-loaded, and the detailed notes are justified because no output schema exists to carry that burden.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter read-only list tool, the description covers everything an agent needs: the exact endpoint, auth requirement, response object shape with field semantics, example slugs, and usage caveats about universal/trace templates. There is no output schema, so the inline return specification is essential and is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema is empty with 100% coverage, so no parameter documentation is needed. The description adds value by specifying the output record shape and field meanings, which matters more here since there is no output schema. The score reflects the baseline for a zero-parameter tool with complete schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action and resource: listing shipped judge-criterion templates, and explicitly gives the HTTP route GET /v1/criteria/templates. It clearly distinguishes these read-only template entities from sibling tools like list_criteria or list_dedicated_templates by framing them as starting-point judge prompts for later instantiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when this tool is useful: to obtain a starting-point judge prompt and instantiate it via POST /v1/criteria with an edited judge_prompt. It also adds practical usage warnings, such as templates needing alignment against workspace labels and task-specific templates needing traffic scoping. It does not explicitly name alternatives, but the instantiation workflow makes the intended context clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent annotations, the description discloses several non-obvious behaviors: the dedicated routes do not enforce key scopes, deleted endpoints are excluded, status is live-reconciled with DB fallback, catalog/provider failures return a 400, and billing accrues per GPU-hour. This is exactly the kind of behavioral context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but dense and well-organized: the core purpose is front-loaded, followed by a structured field list and then essential caveats. Every sentence adds operational value, and there is no tautology or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully compensates by covering return shape, field meanings, auth caveats, reconciliation behavior, deletion filtering, error cases, and the billing model. An agent has enough information to call the tool correctly and interpret its results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and schema description coverage is 100%, so there is no parameter ambiguity to resolve. The description still adds meaning by explaining response-field semantics such as hourly_rate_usd being frozen at deploy and pending_cost_usd being unbilled accrued GPU-hours.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: lists the workspace's dedicated (reserved-GPU) inference endpoints, with live-reconciled status, frozen hourly price, and unbilled cost. It clearly distinguishes this from the create/update/delete/get dedicated endpoint siblings and from list_dedicated_templates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly frames the use case: 'use it to monitor what is running and what it is costing,' and provides the HTTP method and read scope. It does not explicitly mention when to prefer get_dedicated_endpoint for a single endpoint, but the context is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, it discloses auth behavior (the local dedicated apiKeyActor does not enforce scopes), pricing semantics (null means deployment will be refused; prices are quoted at request time and frozen at creation), and specific 400 error cases. This is substantial behavioral context that structured annotations cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every section carries distinct value: purpose, route/scope, exact return shape, pricing/error caveats. It is front-loaded with the most decision-relevant information and stays organized despite the dense catalog schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema or parameters, the description carries the full burden of explaining the response and the call's safety profile, and it does so exhaustively: templates schema, prices schema, error conditions, auth notes, and pricing freeze behavior. Nothing needed to invoke or interpret the call is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters and the schema coverage is complete for that, so there are no parameter semantics for the description to add. The description instead documents the returned data structure, which is the relevant semantic information for this stateless read-only call.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies a specific resource ('deployable model catalog for dedicated endpoints') and explains its purpose: reading it lets you build a valid POST /v1/dedicated request and estimate cost. This clearly distinguishes it from siblings like list_dedicated_endpoints and create_dedicated_endpoint.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit trigger context: use it before creating a dedicated endpoint to get valid model/flavor/GPU values and to estimate costs. It does not explicitly spell out when not to use it or name an alternative, but the intended usage is unmistakable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnly/openWorld/idempotent/non-destructive hints, and the description adds substantial behavioral detail beyond them: no wallet spend, API-key scope, no pagination/filtering, snake_case conversion, model-id key pass-through, status lifecycle, and frozen criterion snapshots. This is rich disclosure beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the purpose, ordering, limit, and audience, then moves through the full response shape and ends with critical notes. Despite its length, every sentence carries operational value, especially because there is no output schema to encode the return structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With zero parameters and no output schema, the description carries the full burden of documenting behavior and return values, and it does so comprehensively: response shape, nested run details, statuses, key casing, scope, and read-only/no-spend behavior. Nothing needed to invoke and interpret this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there are no parameter semantics to document. The description reinforces this by stating there is no filtering or pagination, and the baseline for a no-parameter tool is 4, which is appropriate here.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('List'), resource ('this workspace's eval runs'), ordering and limit ('newest first, most recent 50'), and the kinds of runs included ('comparison, criterion run and screening'). This clearly distinguishes it from siblings like create_eval, delete_eval, and compare_evals by enumerating its exact scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear usage context: a customer can see every queued or finished comparison, criterion run, and screening. It also states constraints ('No pagination or filtering: always the 50 newest runs'), but it does not explicitly name alternative tools such as get_eval or compare_evals or state when to prefer them.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false; the description goes well beyond these by disclosing live reconciliation against the training backend, best-effort status on backend error, that locally terminal statuses are never resurrected, markup stripping, feature-flag gating (404), read-only/no-spend guarantees, and the fact that scopes are not enforced. This is rich, non-redundant behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Although the description is long, it is tightly structured: purpose, endpoint/auth scope, return shape, and behavioral notes. Every sentence adds necessary information, especially because there is no output schema to carry return-value documentation. The key purpose is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and zero parameters, the description carries the full burden of explaining what the agent will receive and how the backend behaves. It documents the exact array shape, all notable fields, status enums, deployment statuses, pricing fields, error cases, and live-vs-terminal semantics. The only minor omission is explicit pagination behavior, but for a monitoring list endpoint this is not a significant gap given the complete field-level return specification.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the schema coverage is 100%, so there is nothing for the description to add about parameter semantics. The baseline for 0-parameter tools is 4, and the description correctly omits irrelevant parameter details while focusing on output and behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('List this workspace's fine-tuning jobs'), adds meaningful qualifiers (newest first, live status, progress, output model, price), and is clearly distinct from sibling tools like get_fine_tuning_job, create_fine_tuning_job, and cancel_fine_tuning_job. It also names the exact HTTP endpoint, leaving no ambiguity about scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly frames the intended use case ('for monitoring training from CI or a script') and states the read-only, no-spend nature. It does not explicitly name alternatives or give when-not-to-use conditions, but the context is clear enough that an agent can select it correctly among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and non-destructive. The description adds valuable behavioral detail beyond annotations: the exact endpoint, API-key scope, bare-array response format instead of an envelope, newest-first ordering, and precise null semantics for kappa. No contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is somewhat long but every sentence earns its place: purpose, endpoint/scope, complete return shape, and caveats are all separated clearly. The critical 'bare array' warning is isolated in a Notes section rather than buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully compensates by enumerating every returned field and its type/null behavior. It also covers ordering, endpoint, scope, and the kappa edge case, making the tool callable without any external documentation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the schema is an empty object with 100% coverage, so there is no parameter semantics to add. The rubric's baseline of 4 applies, and the description instead enriches the response semantics extensively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List') and a specific resource ('the workspace's golden sets') while defining what golden sets are. It also clarifies the distinguishing scope ('named collections of human-graded requests...') so it will not be confused with list_labels or the label-set management siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly frames when to use the tool: 'use it to see which sets exist, which are frozen, and which judges calibrate on them.' It gives clear practical context though it does not name alternatives or state when not to use it, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so the description correctly focuses on behavior annotations cannot express: newest-first ordering, successful (non-aborted) exchanges only, secret-scrubbing, the 409 logging_disabled error that prevents misreading an empty list as 'no traffic', and the exact offset-pagination loop. The 409 disclosure alone is high-value operational context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Dense but every sentence earns its place: purpose front-loaded, then endpoint+auth scope, then a complete return shape (critical since there is no output schema), then ordering, error mode, pagination pattern, and a cross-tool tip. No fluff, no repetition of annotation content, and the structure follows a natural read→call→interpret flow.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 9-parameter, 0-required, no-output-schema read tool, the description covers everything needed to call it correctly: full return payload shape, ordering and filtering semantics, the one critical error case (409 when logging disabled), the pagination contract, and how results feed into evals. Nothing an agent needs to invoke this safely and correctly is left to inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds genuine value beyond the schema by mapping the filter surface in prose, clarifying time-range semantics for start/end, and especially the segment round-trip workflow (row's segment → ?segment= or eval sample_filters.segment), which is operational guidance no schema field conveys.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource+scope: 'Browse the workspace's logged chat exchanges (request messages + assistant reply, secret-scrubbed)' with named filters. The 'inspect real traffic before grading, building datasets, or running evals' phrase distinguishes it from siblings like export_logs (bulk export) and get_trace (single trace lookup) without needing their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear when-to-use context ('inspect real traffic before grading, building datasets, or running evals') and even shows downstream reuse ('Pass a row's segment back as ?segment= or into an eval's sample_filters.segment'). However, it never names alternatives or states when NOT to use it versus export_logs or screen_my_traffic, stopping short of the explicit routing the top band requires.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Even with strong annotations (readOnly, idempotent, non-destructive), the description adds substantial behavioral context: feature-flag gating (404), required OWNER/ADMIN key (403), empty array as a valid non-error response, 200-round cap, and the meaning of null fields. This goes well beyond what annotations alone convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but well-organized: purpose, endpoint, response schema, limits, and error conditions are all present without wasted words. Information is front-loaded with the core purpose before the API details, making it easy for an agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the empty input schema and lack of an output schema, the description fully equips an agent to invoke and interpret the tool correctly. It covers the endpoint, auth scope, response fields with null semantics, max result count, flag gating, and non-error empty behavior. No meaningful gap remains for a zero-parameter read-only list call.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. There are no parameter semantics to document, and the description does not need to compensate for any schema gaps. It instead documents the output shape in detail, which is appropriate for this parameterless call.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('List') and resource ('self-improvement rejection-sampling fine-tuning rounds') with meaningful scope ('workspace's', 'newest first', 'started from the dashboard'). It also enumerates the data returned, clearly distinguishing it from sibling tools like list_fine_tuning_jobs or list_grpo_runs through the explicit 'raft/rounds' resource and endpoint path.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear context for when to use the tool: read-only observability of dashboard-started raft rounds, and explicitly notes there is no public start endpoint. However, it does not name alternative sibling tools or state when one should use them instead, stopping short of the full when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description prominently warns that the tool spends money, bills the workspace wallet, and can return 402 when the funds gate is not met. It also discloses that it waits for the run and returns verdicts, adding behavioral context beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense and front-loaded with the screening question, then mechanics, then cost, then routing. It is longer than strictly necessary and uses a long dash-laden sentence, but every sentence carries distinct value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
It covers prerequisites, cost, high-level output, and candidate selection, which is strong for a tool with no required parameters and no output schema. It stops short of fully describing the polling path when wait_seconds expires and the exact return shape, though wait_seconds is already documented in the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All parameters are already fully described in the schema, so the baseline is 3. The description adds meaning by explaining that candidate_models are the challengers that override the auto-picked cheaper set and that the tool is zero-config with all parameters optional.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a concrete question and defines the tool as a zero-config screening of the workspace's own traffic, naming the incumbent, challengers, and the verdict/recommendation output. It also explicitly contrasts with create_eval, so an agent can distinguish this tool from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states prerequisites ('Needs request logging on and logged traffic') and gives an explicit routing rule: 'Prefer this over create_eval for the is X better/cheaper question.' The cost gate and zero-config nature further clarify when the tool is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations say readOnlyHint=false and destructiveHint=false, and the description is fully consistent with that. It goes far beyond annotations by disclosing that the call spends money, bills the workspace wallet, immediately meters accrued GPU-hours on stop/resize, requires an OWNER/ADMIN minting user, triggers specific status transitions, and returns only { ok: true }. This is exemplary behavioral disclosure for a mutating, money-spending operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every section earns its place: the opening sentence states the full purpose, the money warnings are prominent, return shape and status transitions are explicit, and error cases are enumerated. The MONEY note is repeated for emphasis, but given the financial impact and complexity of this tool, the length is justified and well organized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex operation with no output schema, the description is complete: it covers the exact action, HTTP method and scope, billing consequences, wallet gating, authorization requirements, status transitions, and likely 4XX error cases. An agent has enough behavioral and environmental context to invoke the tool correctly and anticipate the important edge cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already covers 100% of parameters with rich descriptions, so the baseline is 3. The description adds cross-parameter meaning: enabled=false stops billing, gpu_type/gpu_count triggers price re-freeze and a wallet gate, and replica counts must satisfy max >= min >= 1. This goes beyond the schema's per-field descriptions and helps an agent reason about combinations.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb-resource pair ('Scales, starts/stops, renames, or changes the GPU configuration of a dedicated endpoint') and clearly differentiates this update operation from sibling tools like create_dedicated_endpoint, get_dedicated_endpoint, and delete_dedicated_endpoint. It also anchors the HTTP method and path, leaving no ambiguity about what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives concrete usage scenarios: use enabled=false to stop billing without deleting, and use gpu_type/gpu_count to re-size. It contrasts stopping with deletion, which helps the agent choose this tool over delete_dedicated_endpoint. However, it does not explicitly list when not to use the tool or name alternative tools beyond that implied contrast.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the sparse annotations: it discloses that the POST writes nothing, nothing is stored, Cache-Control is no-store, verification canonicalises the document, and successful well-formed requests always return 200 with enumerated failure reasons. readOnlyHint=false and idempotentHint=false do not affirmatively claim side effects, so there is no contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense but every clause earns its place: purpose, trigger, HTTP details, response variants, and canonicalisation behavior are all covered without redundancy. The response reason enum is presented compactly and readably.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description fully enumerates the response shape and all `ok: false` reasons. Combined with complete schema coverage for the single parameter, nothing an agent needs to call this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully documents the single `document` parameter, so the baseline is 3. The description adds useful meaning by explaining what a signed document looks like, how HMAC re-derivation works, and why key order does not matter while value changes do.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Verify') and resource ('downloaded certificate or evidence bundle'), and explains the mechanism ('re-deriving its HMAC signature'). This clearly distinguishes it from verification-adjacent siblings like get_criterion_certificate or get_audit_verification.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit trigger: 'use it when a third party hands you a document and you need to trust its numbers.' It does not name specific alternatives or exclusions, but the described scenario is sufficiently distinct to guide an agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark readOnly/idempotent/non-destructive, and the description adds substantial behavioral detail: exact response shape, Cache-Control: no-store, sign convention (delta = b − a), Newcombe 95% interval, significance rule, no spend, and precondition failure semantics. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but densely informative and well structured: purpose, endpoint, response shape, error conditions, best-use guidance, and safety note. Every clause earns its place, and the core purpose is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema present, the description fully specifies the return object including nested fields, arm keys, metrics, confidence intervals, unmatched sets, and error cases. It also covers operational context (workspace scoping, judge-model precondition, sample_filters), making it complete for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds the crucial directional meaning: a is the BEFORE/baseline run and b is the AFTER run, with delta explicitly defined as b.rate − a.rate. This goes beyond the schema's short descriptions and prevents a common misuse.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('Compare two finished eval runs') and distinguishes the tool from single-run eval tools by explaining the arm-by-arm delta output. The phrase 'instead of eyeballing two reports' makes the intent unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly states when the tool applies (finished runs, same eval kind, same judge model, ideally identical sample_filters) and documents rejection conditions (400/404/412), including the important 'delta between judges measures the judges' caveat. It does not explicitly name sibling tools like get_eval for single-run needs, but the comparison context is otherwise well specified.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent/non-destructive annotations, the description discloses critical behaviors: 412 is not an error shape but the verdict body, fail-closed on non-DONE runs, 404 semantics, invalid query values treated as absent, key casing, and 'Read-only, no spend.' This adds substantial context the annotations alone do not provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence carries useful operational information: verdict semantics, response shape, 412 cases, parsing rules, and authentication. It is front-loaded with the core one-call CI decision concept, and the details are organized logically from response shape to edge cases.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully compensates by specifying the return object shape, check types, the conditions for 200/412/404, the meaning of 'reason', and polling guidance. For a tool with 7 parameters and complex threshold semantics, nothing essential for an agent to invoke it correctly appears to be missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Even with 100% schema coverage, the description adds deep semantics beyond the schema: thresholds are compared against CI lower bounds, not point estimates; it explains corrected vs. printed rates, fail-closed behavior, calibration prerequisites, and the noninferiority switch test. This is a strong value-add over the raw parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource+outcome: 'Turn a finished eval run into a CI deploy decision with one call' and gives the exact endpoint GET /v1/evals/{id}/gate. It clearly distinguishes the tool's role as a deploy gate from the broader eval-related sibling tools by emphasizing the 200/412 binary verdict for pipeline blocking.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly establishes the CI context: 'so a pipeline can `curl -f` it and block a bad model/prompt change' and advises pairing with a read-only scoped key. It doesn't explicitly name alternatives or specify when not to use this tool, but the intended workflow and fail-closed polling behavior are clear enough for an agent to select it appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations by disclosing the exact response shape (bare JSON array, no envelope), the field semantics (especially provider_file_id being the opaque upstream id to use), error shape for feature-flag gating (plain string error not nested), and the lack of spend. This is useful operational detail that annotations alone do not convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but packed with useful information: endpoint, scope, return format, field meanings, error behavior, auth nuance, and cost. It is front-loaded with the core purpose and ordering, then layers supporting details. Slightly dense, but every clause earns its place given the absence of an output schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no parameters, no output schema, and rich annotations, the description covers everything an agent needs: what is returned, which field to use downstream, what errors to expect, auth behavior, and side-effect/cost implications. No material gap remains for correct invocation and interpretation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters and the schema coverage is 100%, so the description has no parameter burden. It instead clarifies what the returned fields mean, especially provider_file_id, which is the kind of semantic detail that would otherwise be missing. A 4 reflects the baseline for a no-parameter tool plus the added value of explaining result semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies a specific verb and resource: 'List the training files this workspace has uploaded for fine-tuning, newest first'. It also clarifies the purpose—finding the file id to start a job—and distinguishes itself from related operations by naming the exact endpoint (GET /v1/fine_tuning/files) and emphasizing the provider_file_id as the value to pass to job creation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use this tool: when a customer needs to find a file id to start a fine-tuning job. It also provides critical context about feature-flag gating (404 when fineTuning is off) and authentication behavior (any valid workspace key works; management scopes not enforced), which helps the agent decide whether this tool is accessible in the current context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already carry the safety profile (readOnly, idempotent, non-destructive), so the description focuses on behaviors annotations cannot express: server-side 2000-char clipping with a '[clipped]' marker, outcome enum differences between comparison ('win'|'loss'|'tie'|'failed') and criterion ('pass'|'fail'|'unparsed') runs, the '__stored__' baseline fallback semantics, empty-array-before-sampling behavior, and late-appearing human_verdict fields. This is dense, valuable behavioral disclosure beyond the structured data.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded in sentence one, and the subsequent length is earned: with no output schema, the description must document a nested return payload, and it does so in a logical progression (endpoint/auth → array shape → field semantics → enums → caveats → alternative endpoint). Each sentence carries distinct information; nothing is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with a nested return structure, run-state-dependent edge cases, and no output schema, the description covers everything an agent needs: the full item shape (prompt, baseline_answer variants, candidates array), per-run-type outcome enums, clipping limits with the escape hatch, human_verdict timing, empty/partial states, and the workspace 404. No critical gap remains.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% — the lone 'id' parameter is already documented as 'The eval run id' — so the baseline is 3. The description adds meaning on top: the id appears in the endpoint path, is workspace-scoped (404 for runs outside the workspace), and is valid across any run status. This modestly exceeds what the schema alone provides, warranting a 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence states a specific verb and resource: 'Inspect the test cases behind a run's score', then enumerates exactly what is included (sampled prompt, each arm's answer with reasoning traces stripped, per-sample verdict). It differentiates from the sibling get_eval (run metadata) and get_eval_evidence (full transcripts) by naming the evidence endpoint explicitly as the alternative for full content.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit routing: use GET /v1/evals/{id}/evidence?with_content=true when clipped 2000-char texts are insufficient. Also gives clear availability conditions — works on any status, partial data while RUNNING, empty array before sampling, 404 when the run is not in the workspace — so an agent knows when the call is valid vs. when results will be incomplete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![errorbar-mcp MCP server](https://glama.ai/mcp/servers/omnia-v/errorbar-mcp/badges/card.svg)](https://glama.ai/mcp/servers/omnia-v/errorbar-mcp)

Score Badge

Copy to your README.md:

[![errorbar-mcp MCP server](https://glama.ai/mcp/servers/omnia-v/errorbar-mcp/badges/score.svg)](https://glama.ai/mcp/servers/omnia-v/errorbar-mcp)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/omnia-v/errorbar-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server