The Aggregate — LLM benchmark aggregate

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive, and closed-world, so the safety profile is covered. The description adds genuinely useful information beyond that: it discloses the payload contents (IRT fusion explanation, coverage counts, cadence, citation), which matters because there is no output schema. It still doesn't say the response format or length.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence that lists its payload categories with zero filler or restatement of the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of conveying what comes back, and listing the four content areas does that adequately for a documentation tool. It could still state the return format (prose vs. structured) to be fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline of 4 applies; there is nothing for the description to disambiguate and no schema gap to compensate for.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the resource (the aggregate dataset) and enumerates its subject matter: IRT fusion methodology, coverage counts, update cadence, and citation guidance. It is clearly a metadata/documentation tool and distinguishable from the data-fetching siblings (get_benchmark, search_models), though it never uses an explicit verb like 'returns' or 'describes'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the content list — an agent would call this when it needs methodology, coverage, or citation details — but there is no explicit when-to-use or when-not-to-use statement, and no alternative tool is named or excluded.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive and closed-world behavior, and the description adds the substantive part: what the comparison returns (aggregate ranks, a rating gap with a significance note derived from standard errors, shared benchmarks). That output-shape disclosure matters because there is no output schema. It still says nothing about error behavior for unknown model names or missing coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence with the core purpose front-loaded before a colon-delimited list of outputs. Every clause carries information, though the output enumeration is compressed enough to read slightly like a spec dump.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter read-only tool with no output schema, the description supplies the essential missing piece (what the result contains). It is nearly self-sufficient; only failure modes and whether benchmarks shown are filtered or exhaustive remain unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a single well-described parameter, so the schema already carries the semantics. The description restates the 2-4 cardinality but adds no format guidance (slug vs display name, case sensitivity) beyond what the schema provides. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific action (head-to-head comparison) and a bounded resource set (2-4 models), then enumerates the concrete outputs: aggregate ranks, rating gap with significance, shared benchmarks. This clearly separates it from get_model or get_leaderboard, though it never explicitly names a sibling to differentiate against.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The 2-4 model framing implies the use case (pairwise/multi-way comparison rather than single-model lookup or full leaderboard), but there is no explicit when-to-use statement, no prerequisites, and no pointer to alternatives such as get_prediction_duel for duel-level questions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so the safety profile is covered. The description usefully discloses the shape of the returned payload, but says nothing about error behavior for an unknown benchmark name, rate limits, or data freshness.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence that front-loads the scope ('one benchmark in depth') and then lists the returned fields; nothing is wasted, though the parenthetical list is slightly enumeration-heavy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description does the work of enumerating return contents, and both parameters are fully documented in the schema. It is close to complete for a simple read tool, with only routing/usage context left unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with both the benchmark name/slug and the top-N range documented inline, so the schema carries parameter meaning. The description adds no syntax, format, or default guidance beyond it, making the baseline 3 appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific resource ('one benchmark') and enumerates the concrete contents returned (IRT stats, skill weights, top models), so the agent knows exactly what it fetches. It implicitly separates itself from search_benchmarks by emphasizing 'in depth' for a single benchmark, but never names the sibling, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: 'in depth' suggests this is the detail step after discovery via search_benchmarks, but the description never states when to use it versus get_leaderboard, search_benchmarks, or get_model. No exclusions or prerequisites are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and a closed world, so the safety profile is covered. The description adds genuinely useful behavior beyond that: default output is one row per model fused across reasoning-effort settings, plus paging support. It does not describe ordering stability, tie-handling, or scale units, so it remains adequate rather than rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with what the ranking is before the sibling pointer and the paging note. Tight and readable, though the closing paging sentence largely duplicates what the schema already states.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of conveying the return shape, and it does so reasonably: one row per model, fused across reasoning-effort settings, pageable. It omits the exact fields per row and the ranking direction, but for a read-only listing tool it is close to complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with each of limit, offset and include_variants fully documented in the schema, so the baseline is 3. The description restates paging via limit/offset and the fused-by-default behavior tied to include_variants, adding no format or constraint detail the schema lacks.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Top of the cross-benchmark aggregate ranking', 'one rating scale by an IRT model fit over public benchmark leaderboards') and distinguishes itself from siblings like get_benchmark and get_model by describing a fused cross-benchmark aggregate. An agent can identify exactly what this tool returns without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear context for use and routes one adjacent need explicitly ('call about_the_aggregate for the current coverage counts'). However, it never states when to prefer this over siblings like compare_models or get_benchmark, so the alternative-selection guidance is partial rather than complete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive, closed-world), so the bar is lower. The description adds real value by disclosing what data is returned — aggregate rank, rating with standard error, cost-per-task availability ('where known') — which is important since there is no output schema. It does not state failure behavior for unknown model names.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with the core concept ('One model in depth') front-loaded, followed by a compact enumeration of returned fields. Every clause carries information, though the list-like tail is dense.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of describing the return payload, and it does so thoroughly. Only edge-case behavior (unknown model, missing cost data beyond 'where known') is left unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is a single parameter with 100% schema description coverage, and the schema already gives example formats ('Claude Opus 4.5' or 'gpt-5-5'). The description adds nothing about name-vs-slug resolution or case sensitivity, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific resource ('One model in depth') and enumerates the exact payload: rank, Aggregate Rating with standard error, provider, description, cost per task, and benchmark results. This clearly separates it from list-oriented siblings like search_models or get_leaderboard, though no sibling is named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the phrasing 'One model in depth' — an agent can infer this is the single-entity lookup versus the multi-entity siblings. However, no explicit when-to-use guidance, prerequisites, or alternative routing (e.g., compare_models for side-by-side) is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, non-destructive and openWorldHint=false, so the safety profile is covered. The description adds useful scope context (daily cadence, predictions made before scores are seen, errors scored) and states the return payload includes wins and losses, but says nothing about freshness or how monthly standings reset.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, with the resource identified up front via 'the public prediction duel'. The middle clause about LLMs and the IRT model is somewhat wordy but does real explanatory work for a novel concept, so it mostly earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless read-only tool with no output schema, the description supplies enough: it names the entity, describes the underlying mechanism, and previews the returned content (monthly standings, wins, losses). Only the absence of any refresh/period detail keeps it from a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline of 4 applies. There is nothing for the description to clarify beyond confirming the call needs no arguments.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource: returns current monthly standings for the 'Guesswork' prediction duel, including wins and losses. It is clearly distinguishable in substance from the data-lookup siblings, though it never names or contrasts with get_leaderboard, which is the closest relative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains what the duel is but gives no when-to-use guidance, no prerequisites, and no mention of alternatives such as get_leaderboard or compare_models. The agent must infer the call context entirely.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so the safety profile is handled. The description adds genuinely new context by naming the returned payload (model coverage, difficulty, benchmark page URL), which matters because there is no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with the search action front-loaded and the return payload second. Nothing is wasted and nothing is buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter read-only search with a fully documented schema and annotations carrying the safety profile, the description covers what it does and what it returns. Minor omissions like result ranking or behavior when no match is found are tolerable at this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters are already documented with range, default, and an example fragment. The description only echoes that the match is partial, adding no syntax or behavior beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (find benchmarks) plus the match semantics (partial name) and what comes back. It contrasts implicitly with the singular get_benchmark sibling through naming, but never explicitly differentiates the two in the text.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'by (partial) name' implies the use case - you have a fuzzy fragment rather than an identifier - which distinguishes it from get_benchmark in practice. However, there is no explicit when-to-use or when-not-to-use statement, and no named alternative.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this a safe read (readOnlyHint=true, idempotentHint=true, destructiveHint=false), so the bar is lower. The description adds real behavioral context beyond them: the returned fields (rank, Aggregate Rating, page URL) and the default de-duplication/fusion across reasoning-effort settings, which explains why a caller may see fewer rows than expected.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the capability and followed by return shape and the default row behavior. Nothing is repetitive or decorative; every sentence carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly takes on the return-value burden by naming the returned fields and the one-row-per-model default. It stops short of covering result ordering/pagination specifics, though limit is already documented in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents query, limit and include_variants. The description reinforces the variant-fusion semantics of include_variants and the partial-match nature of query, but adds no syntax or format detail beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource plus the matching key (partial name or provider), so the agent knows exactly what comes back: ranked models. It does not name or distinguish itself from siblings like get_model or compare_models, which is the only thing keeping it from a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the description (search when you have a name/provider fragment), but there is no explicit when-to-use, when-not-to-use, or pointer to an alternative such as get_model for an exact single-model lookup. Minimum viable guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.