IA-QA Eval & RAG — 38 tools

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, non-destructive behavior. The description adds value by specifying the statistical metrics returned (mean, median, std deviation, winner, improvement percentage), which goes beyond the structured annotations. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that efficiently covers purpose, input, and output without fluff. It is front-loaded with the primary action 'Generate an A/B test report'.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema and simple input schema, the description adequately covers what the tool does. It could mention edge cases like equal scores or unequal array lengths, but for a straightforward statistical tool, it is sufficiently complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds semantic meaning by calling the variants 'prompts or model configurations', which is not explicit in the schema's generic 'variant configuration'. It also clarifies that 'scores' are arrays for statistical comparison, aiding understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool's function: 'Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns statistical comparison' with specific outputs. This distinguishes it from sibling tools like compare_models or compare_responses, which likely have different comparison methods.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly implies when to use: when you have two sets of scores to compare statistically. It mentions 'two prompts or model configurations' and 'arrays of scores', which is sufficient context. It doesn't explicitly name alternatives, but the usage context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, and non-destructive behavior. The description adds valuable context beyond annotations: it runs locally, requires no LLM or API key, and describes the two operational modes and their outputs. No contradiction exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose, modes, local execution, CI relevance, input requirements, and sibling routing. It is front-loaded with the core purpose and ends with the alternative tool, which is an effective structure.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich annotations, complete schema, and output schema, the description covers everything an agent needs: what the tool does, when to use it, how modes differ, what inputs are required, and which sibling to choose instead. Nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds meaningful semantics by explaining the role of the reference parameter and clarifying that responses must be for the same task and can be plain strings or labeled objects. This goes beyond the schema's basic field descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: semantically analyze N model outputs for the same task, with distinct consensus and reference-based modes. It explicitly differentiates itself from the sibling compare_responses, so an agent can disambiguate without inspecting schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit guidance on when to use the tool versus compare_responses, and explains which mode is appropriate for CI gating versus as a non-failing signal. This is strong, actionable usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so safety is covered. The description adds that scores are normalized and it handles one or more documents, but does not detail other behavioral traits like tokenization or comparability. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with the main action front-loaded. The second sentence provides useful context about the algorithm's industry usage, though it could be trimmed slightly. Overall very concise with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present and annotations covering read-only/idempotent safety, the description adequately covers the tool's purpose and return behavior. It lacks explicit alternative guidance but is sufficiently complete for a computational scoring tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, providing descriptions for all parameters (query, documents, top_k, k1, b). The description adds no additional parameter-specific meaning beyond the schema, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Compute BM25 relevance score between a query and one or more documents', identifying the specific verb, resource, and unique algorithm. This distinguishes it from siblings like rag_relevance_rank or similarity_score by naming BM25 explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides clear context by stating BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search, implying when to use it. However, it does not explicitly name alternatives or exclusions such as 'for semantic similarity use embedding_similarity instead'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish read-only/idempotent/non-destructive safety, so the bar is lower. The description adds valuable behavioral context by enumerating internal steps (token budgeting, citation numbering, system instruction injection, source attribution), which helps predict output structure. It doesn't disclose edge-case behaviors (e.g., truncation strategy) but exceeds baseline.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core action. Every phrase adds information—no filler or redundancy. Appropriate length for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description plus rich schema and output schema provide enough for selection and basic invocation. It covers the main functional areas and hints at all parameter roles. It omits details like truncation strategy and citation ordering, but these are secondary given the strong schema context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers 100% of parameters, so baseline is 3. The description adds meaning by connecting parameters to their effects: max_context_tokens↔token budgeting, cite_sources↔citation numbering, system_instruction↔system instruction injection, chunks.source↔source attribution. This contextualizes the schema without duplicating it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Assemble a complete RAG prompt', combining a specific verb with resource and data source. The additional capabilities (token budgeting, citations, system instruction) distinguish it from sibling tools like few_shot_formatter or system_prompt_builder.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

States it operates on 'retrieved context chunks', making the input precondition clear. However, it doesn't explicitly name alternatives or state when not to use it, though the name and RAG specificity imply these boundaries.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so safety is covered. The description adds that a recommendation is generated and the table contents, but says nothing about behavior with unknown model names, invalid counts (e.g. 6 models) or error handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

A single tightly-packed sentence with the action and scope front-loaded and the return value last. Nothing is redundant or padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema and annotations are present, so the description needn't explain return structure or safety, and it correctly summarizes both. The only shortfall is absence of guidance on edge cases such as invalid model names or too many models.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and both parameters are self-documented, including the enum values and the 2-5 bound, so baseline 3 applies. The description restates the 2-5 range and the compared fields but adds no syntax or format detail beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (compare) and resource (2-5 AI models) and enumerates the attributes compared (context window, pricing, multimodal, reasoning, provider). It also names the output (comparison table + recommendation), so an agent can distinguish it from single-model siblings like model_info. It never explicitly names those siblings, keeping it just short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The use_case parameter and 'recommendation based on your use case' imply when to reach for this tool (multi-model selection decisions), but no exclusion or alternative is stated. An agent must infer that model_info is for one model and list_llm_models for browsing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, non-destructive behavior. The description adds valuable behavioral context beyond that: it lists exactly what metrics are returned, explains that a `reference` causes scoring against ground truth, and that providing `model`+`api_key` triggers an LLM judge call. It also notes 'No re-execution' and the schema notes the API key is 'never stored,' which bolsters transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact—five sentences that front-load the core purpose, then layer conditional behavior and end with a clear constraint. Every sentence earns its place; no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given an output schema, rich annotations, and 100% schema parameter coverage, the description is complete enough. It covers the two major optional modes (reference-based scoring and LLM judging), the deterministic metrics, and the no-re-execution constraint, leaving no significant gaps for a tool of this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with good per-parameter descriptions, yielding a baseline of 3. The description adds semantic value by explaining how parameters interact: `reference` makes each output scored against it, `model`+`api_key` enables the judge, and the metric list (token cosine, ROUGE-L, Jaccard, JSON diff) clarifies what the tool computes from `response_a` and `response_b`.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Compare two ALREADY-PRODUCED outputs ... side by side.' It clearly differentiates from siblings by emphasizing no re-execution and listing concrete deterministic metrics plus an optional LLM judge verdict.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly states when to use the tool: when outputs already exist and you want a side-by-side comparison. It gives conditional guidance for `reference` and `model`+`api_key`, and explicitly says 'No re-execution — you bring the outputs,' which implies it is not for generating outputs. It does not name alternatives explicitly, so it's strong context without formal exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes beyond the annotations by revealing the algorithmic limitation (surface-level word matching) with a concrete example. It also adds traits such as 'fast' and 'deterministic', which are not present in the annotations but are important behavioral context. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise, using only two sentences to convey purpose, method, limitations, and alternatives. Every sentence adds value, and the text is well-structured with a clear separation of strengths, limitations, and use case.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, has full parameter documentation, and an output schema. The description covers use cases, limitations, and alternatives, making it complete for an agent to decide when to use it. There are no obvious gaps given the available structured information.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already provides 100% coverage for both parameters, so the baseline is 3. The description adds slight context by explaining that check_facts deals with 'fact drift (number comparison)', but this is already implied by the schema description. No additional parameter-level detail is provided in the description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: compare multiple LLM responses to the same prompt and detect inconsistencies via Jaccard similarity and fact drift. It names specific methods and distinguishes from sibling tools like run_semantic_tests by explicitly referencing embedding mode for semantic consistency.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides clear when-to-use context: 'Essential for determinism testing' and highlights the fast/deterministic/no-API-key benefits. It explicitly states when not to rely on it (semantic equivalence) and directs to run_semantic_tests as an alternative for true semantic consistency.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only and idempotent behavior, so the bar is lower. The description adds valuable context: the operation is an 'estimate' (implying approximation) and it 'warns about truncation risk'. This goes beyond the schema and annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the most important action, and contains no filler or redundant repetition of annotations. Every word contributes to understanding the tool's purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity, full schema coverage, and presence of an output schema, the description sufficiently covers inputs, purpose, and key behavior. It does not need to describe return values because the output schema handles that, and it adequately sets expectations about estimation and warnings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description repeats the message structure already present in the schema and does not add new parameter details. It provides no additional syntax or format guidance beyond what the schema already documents.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function with specific verbs ('estimate', 'check', 'warns') and names its resource (message array and target model's context window). It distinguishes itself from siblings like count_tokens by adding the context-window fit check and truncation warning.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implicitly communicates when to use the tool (when you need to estimate token usage and verify context fit), but it does not explicitly name alternatives or exclusions. Given the clear context stated, this meets the 'clear context, no exclusions' level.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false, and idempotentHint=true. The description adds valuable behavioral context by specifying the input format (messages array) and the exact analysis dimensions. It does not contradict annotations and provides extra detail about what the tool evaluates.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, with three sentences each serving a distinct purpose: stating the core action, specifying input format, and giving usage context. It is front-loaded with the primary verb and resource, and every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a single well-documented parameter, a full schema, clear annotations, and an output schema. The description covers purpose, input format, and use case, which is sufficient for an agent to select and invoke it correctly. Minor gaps like message ordering constraints are not mentioned, but the tool's simplicity makes these non-critical.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% coverage for the single 'messages' parameter, which is already described as 'Conversation messages in order'. The description repeats the array format '[{role, content}]' but adds no additional semantic info beyond what the schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with a specific verb ('Analyze') and resource ('multi-turn conversation'), and lists concrete analysis dimensions (context retention, topic drift, instruction following, repetition). It also mentions the input format and the primary use case (chatbot QA), which distinguishes it from sibling tools like analyze_responses or consistency_check.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description says 'Essential for chatbot QA', providing a clear context for when to use the tool. However, it does not explicitly mention when not to use it or name alternative tools, so it falls short of full differentiation but still gives strong usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and idempotentHint=true, so safety is covered. The description adds valuable behavioral insight by listing what the analysis detects (step count, backtracking, reasoning depth, etc.), which goes beyond the structured fields and helps the agent anticipate the tool's output dimensions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly written sentences with no filler. The first sentence states the action and key capabilities; the second gives the evaluation context. Every clause adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete for a tool with rich annotations and an output schema. It explains what the tool does, what it detects, and when it is useful. There is no missing context about safety or return values because annotations and output schema cover those aspects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3; the description does not need to repeat parameter details. It adds implicit alignment by mentioning 'conclusion presence' which relates to the optional expected_conclusion parameter, but does not explicitly explain parameter semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Analyze' with a precise resource ('Chain-of-Thought (CoT) or reasoning trace') and enumerates concrete detection outputs (step count, logical flow, conclusion presence, backtracking, reasoning depth). It also distinguishes this tool from siblings by explicitly targeting reasoning traces and naming model families (o1/o3/DeepSeek-R1) for evaluation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context by stating the tool is 'Useful for o1/o3/DeepSeek-R1 evaluation,' which signals when to use it. It does not explicitly mention when not to use it or suggest alternatives, but the use case is specific enough to guide selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, and non-destructive behavior. The description adds the approximation method (cl100k_base, ~4 chars/token), which sets expectations about accuracy, and discloses the return fields. This adds meaningful behavioral context beyond the annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no filler. The first sentence states the action and method; the second provides usage timing and output summary. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple with one parameter and an output schema. The description covers purpose, usage timing, approximation details, and return fields, making it complete for an agent to select and invoke correctly. The existing output schema handles return value schemas, so no additional return documentation is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers 100% of the parameter with a basic description. The tool description adds no additional detail about the input parameter (e.g., length limits or encoding), so it does not exceed the baseline for a fully covered schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it estimates token count for a text string using the cl100k_base approximation. It distinguishes itself from siblings like estimate_llm_cost by focusing on token counting rather than cost, and mentions the return fields (token estimate, character count, word count) which further clarifies its function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly instructs to call it before sending text to an LLM API to check context window fit and estimate cost. This provides clear when-to-use context, but it does not mention alternative tools (e.g., truncate_to_tokens) or exclusions, so it lacks explicit alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds useful context beyond annotations: runs entirely in-process, no API key, and the limitation that it's not real embeddings. However, the claim about 'pipe-separated pairs' in batch mode is inconsistent with the schema's array-of-objects definition, creating a minor ambiguity.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences deliver substantial information without redundancy. The core function is front-loaded, and the caveat about real embeddings is essential context. Slightly dense but all sentences earn their place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description adequately covers purpose, limitations, alternatives, and use cases. The only notable gap is the inconsistent batch-mode format description, which could confuse an agent about how to pass input.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already covers all parameters (100% coverage), so the description's mapping of 'bow', 'tfidf', 'ngram' to human-readable algorithm names adds value. The batch-mode explanation is helpful but the 'pipe-separated pairs' phrase conflicts with the schema's batch structure, slightly reducing clarity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb+resource ('Compute text similarity') and enumerates the exact local algorithms (Bag of Words, TF-IDF, Character N-grams). It explicitly distinguishes from true embeddings by clarifying 'NOT real embeddings' and pointing to run_semantic_tests, which differentiates it from sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit contrast with run_semantic_tests for embedding-based similarity, noting no API key needed and in-process execution. Also lists concrete use cases (RAG retrieval testing, semantic search evaluation, text deduplication), making when-to-use clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, and non-destructive. The description adds useful context by listing the exact supported model families and that the result is a USD cost figure. It doesn't disclose edge cases like currency rounding or pricing data freshness, but the annotation coverage lowers the burden.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core purpose and followed by a concise support list. No filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present and simple inputs, the description covers the essential context: what is calculated, the currency, and the list of supported models. It could mention the source/timing of pricing data, but it is sufficient for selecting this tool among many siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and all parameters are described with meaningful examples. The description adds only the general 'token counts' phrasing and the model list, which does not materially improve on the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Estimate') and resource ('API cost in USD') and states the inputs (model and token counts). It is clear and distinct from siblings like count_tokens, though it does not explicitly differentiate from token_budget_calculator.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is implied: call this when you need a cost estimate for a given model and token counts. However, there is no explicit guidance on when not to use it or which sibling tool might be preferable (e.g., model_info, count_tokens).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, and non-destructive behavior. The description adds behavioral context beyond annotations by specifying that it extracts the *first* valid JSON and handles markdown-fenced and inline JSON, which is important for invocation. No contradiction with annotations found.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: the first sentence states the core function, the second covers input format handling, and the third gives explicit usage context. Every sentence earns its place with no redundancy or unnecessary detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter, complete schema coverage, an output schema, and thorough annotations, the description provides ample context. It covers what the tool does, when to use it, and what input formats are acceptable, making it easy for an agent to select and invoke the tool correctly. The minor absence of edge-case behavior (e.g., no JSON found) is negligible given the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter 'input' is well-defined in the schema, with high coverage. The description enriches the parameter meaning by clarifying the input type ('chaotic LLM output', 'surrounded by markdown fences, prose, or explanatory text') and that the extraction targets the first valid JSON—details that go beyond the schema description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: extracting the first valid JSON object/array from chaotic LLM output. It specifies the resource (JSON in text) and the action (extract), and distinguishes itself from sibling tools like extract_json_path and format_json by focusing on embedded data surrounded by prose or markdown fences.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides an explicit when-to-use scenario: 'Call this whenever an LLM returns structured data mixed with explanation text instead of raw JSON.' This gives clear practical guidance. However, it does not name alternatives or explicitly state when not to use the tool, so it misses the upper boundary of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, non-destructive behavior. The description adds behavioral context such as ranking by IDF-weighted relevance and that common words do not inflate results. It also notes the optional inclusion of inputSchemas in results, which is a useful behavioral detail beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences, each earning its place: the first states the core action and timing, the second describes the return content, and the third provides the category list. There is no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema, full parameter documentation, and annotations, the description supplies the missing contextual information: when to invoke it and how results are ranked. It covers the tool's role in the larger catalogue and does so without needing to explain return structures.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so each parameter (query, category, max_results, with_schema) is already well-documented in the input schema. The description adds no additional parameter-specific meaning beyond what the schema provides. It does list the allowed categories, but this is also present in the schema description for the category parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Search' with a clear resource ('available MCP tools') and includes the scope ('by keyword or category before calling them'). It also distinguishes itself from siblings by emphasizing it is a catalogue exploration tool, not a functional tool. The return value (tool names, descriptions, optionally schemas) is stated directly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use the tool: 'Call this when you are unsure which tool to use or want to explore the catalogue.' This provides clear context for use. It does not explicitly mention when not to use it or list alternative tools, but given the meta-purpose, this is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already signal readOnly/idempotent/non-destruct, and the description adds important behavior beyond them: it is deterministic, makes no LLM call, requires no API key despite supporting LLM-based eval, and returns the YAML plus secrets and remaining steps. It also discloses the provider-API-key prerequisite for eval_contract, giving the agent useful operational context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but economical: purpose first, then gate types, then determininism, then return payload, then the companion tool. Every sentence adds distinct value, and there is no filler or repetition of schema content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 12-parametér tool, the description combined with a 100%-covered schema and an output schema is nearly complete. The one gap is that it says 'Two gate types' while the gate enum actually includes selector_drift as a third gate type plus combination values, so the prose slightly under-maps the tool's full capability.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Paraméter description coverage is 100%, so the schema already explains every parametér with defaults, enums, and nuances (e.g., app_start_command cannot be guessed). The description adds narrative around gate types but does not materially enhance parametér semantics beyond what the schema provides, which makes the baseline 3 appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence states a specific verb, resource, and outcome: generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. It also distinguishes the tool from sibling generators by naming generate_eval_yaml as complementary and by describing the exact gate types it emits.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context for when the tool applies and explicitly points to generate_eval_yaml as the companion tool for producing the contract. It does not enumerate exclusions or alternatives beyond that, so it stops short of fully routing an agent to or away from other generator siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate readOnlyHint=true and openWorldHint=true, which align with the description's mention of server-side generation with no API key needed, adding useful context. The description also discloses that it picks appropriate evaluators based on task type, which is beyond annotations, but doesn't detail side effects or limitations like model latency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and front-loaded: it states what it generates, the model used, and the output integration, all in three sentences. It avoids unnecessary detail and each sentence contributes to understanding the tool's purpose and behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity, the description covers essential aspects: generation, model, output usage, and evaluator selection. With a high schema coverage and an output schema present, the description does not need to explain return values. It misses a bit on how the tool handles ambiguous task descriptions, but overall it is fairly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already provides 100% coverage with descriptions for all parameters, including enums and hints. The description adds some value by explaining how task_type influences evaluator selection and mentions scenario_count covers happy path/edge cases/adversarial, but does not deeply elaborate beyond schema. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that the tool generates a complete .ia-eval.yaml evaluation contract from a plain-language description, which is a specific verb and resource. It differentiates from siblings like generate_test_cases and prompt_test_suite by focusing on the evaluation contract format and integration with run_eval_contract.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use it (when you need a ready-to-run evaluation contract) and mentions it pairs with run_eval_contract, but does not explicitly state when not to use it or name specific alternatives. However, the context is clear enough for an agent to decide.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false, and non-open-world, so the safety profile is covered. The description adds useful behavior beyond that: it discloses the return shape and the flexible natural-language topic resolution, though it does not discuss limits or fallback behavior when a topic isn't recognized.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and routing guidance, and every sentence except the topic enumeration adds new value. The inline topic list largely duplicates the schema enum, which is mild redundancy, but it remains readable and well structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be spelled out, yet the description still summarizes them. Combined with the routing logic and full parameter semantics, an agent has everything needed to select and invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the enum is fully documented, so the baseline is 3. The description goes further by explaining the semantics of omitting the parameter and of the 'start-here' value plus free-text resolution, adding meaning beyond the raw enum list.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Query) and resource (IA-QA methodology knowledge base), and enumerates the exact content returned (guidelines, assertion strategies, thresholds, best practices, MCP tools). This clearly distinguishes it from siblings like prompt_test_suite or multimodal_eval_guide, which execute tests rather than retrieve reference guidance.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly covers the no-topic case ('Call without a topic to list all available topics'), routes uncertain users to 'start-here', and even explains that a plain question like 'how do I test my RAG' resolves to a topic. When-to-use is fully specified with no inference required.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations include readOnlyHint true and idempotentHint true, and the description does not contradict them. It adds useful behavioral context: the alignment algorithm for sentences, the handling of negation/antonym flips, and the explicit caveat about lexical limitations ('cannot follow a paraphrase...'). This exceeds the baseline for a read-only tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is detailed yet well-structured, with a clear explanation, a concrete example, and a limitation note. It might be slightly long but each sentence earns its place; it's front-loaded with the core purpose and usage.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity, the description covers the main use case, limitations, and provides an example for clarity. The schema covers parameters and output schema exists, so completeness is good for an LLM agent to invoke correctly, though it could add a bit more about the verdict values ('contradicted', 'well_grounded') but these are implied.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and does not contradict: it describes the behavior and the strict parameter's meaning briefly, but the schema already defines all parameters. The description adds explanation about sentence alignment for 'strict' mode behavior, which is helpful, so it meets the baseline with slight bonus.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the tool performs a 'lexical' hallucination check, explicitly verifying words, numbers, and polarity against a source. It distinguishes itself from sibling tools like semantic tests by emphasizing its fast, deterministic, and lexical nature, and even names the alternative (run_semantic_tests).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly explains when to use this tool: for quick, deterministic verification of factual alignment, and when not to use it: when entailment or deeper reasoning is needed ('For entailment use run_semantic_tests'). It also mentions limitations that guide appropriate usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is clear. The description adds that the tool returns provider, endpoint, and capabilities, and that it lists models from ia-qa.com. It does not disclose pagination or rate limits, but for such a simple tool this is adequate. No contradiction exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the primary purpose and output fields. The second sentence explains filtering and usage examples. Every phrase adds value, with no fluff or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity (one optional parameter), strong annotations, and the existence of an output schema, the description is sufficiently complete. It covers the main behavior, output scope, and typical use cases without needing to explain return structures.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already documents the optional 'provider' parameter at 100% coverage, including examples. The description adds meaning by explaining the filtering effect and explicitly stating that omitting the parameter returns the full catalog, which is not apparent from the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('List'), the resource ('LLM models available on ia-qa.com'), and the output fields (provider, API endpoint, capabilities). It accurately conveys the tool's purpose but does not explicitly differentiate it from closely related sibling tools like 'model_info' or 'compare_models', which also deal with LLM models.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit use cases: 'discover which models are available before calling an LLM API, or to compare providers.' This gives clear context for when to use the tool, though it does not explicitly mention alternatives or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering the safety profile. The description adds the list of supported formats and the use case ('structured output testing'), which is helpful context. However, it does not disclose any additional behavioral traits such as error handling, return structure, or whether it simply returns a boolean or detailed results; the presence of an output schema lessens this burden, but the description itself adds no behavioral depth beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences, front-loaded with the core purpose ('Validate that an LLM output matches an expected format'). The list of formats is compact though comprehensive, and 'Essential for structured output testing' adds valuable context without bloat. Every sentence earns its place; no fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple validation tool with strong annotations, a 100% schema-covered parameter set, and an output schema, the description is nearly complete: it states the tool's purpose, the formats it supports, and the context (structured output testing). The only notable gap is that it does not clarify its relationship to similar sibling tools, but given the rich structured data, the description itself is sufficient for most use cases.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all three parameters (output, expected_format, regex_pattern) are already fully documented in the schema. The description merely repeats the enum values from expected_format and reiterates that regex_pattern is for custom regex, which does not add new meaning beyond the structured field descriptions. The baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Validate') and resource ('LLM output...expected format'), and lists the supported formats (JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, regex). This clearly states what the tool does, but it does not explicitly differentiate it from sibling tools like 'llm_output_validator' or 'regex_test', so it misses the upper bound of full distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'Essential for structured output testing' provides clear usage context, implying it should be used to verify LLM output formatting. However, it does not explain when to use this tool instead of alternatives (e.g., json_schema_validate for schema validation, regex_test for regex, or llm_output_validator for broader validation), nor does it state exclusions or prerequisites. This is implied usage rather than explicit guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes meaningfully beyond annotations by disclosing that the server supplies its own API keys, that calls are rate-limited, that only whitelisted model IDs are accepted, and which providers back which models. Annotations cover the safety profile and an output schema exists, so remaining gaps are minor.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and provider, then key operational facts. The model catalog enumeration is somewhat redundant with the schema and list_llm_models, but each sentence still carries usable information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With a full schema, an output schema, and annotations already present, the description supplies the missing operational context: auth model, rate limits, provider routing, and where to find the model catalog. Nothing needed to call it correctly is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description's model list adds little beyond the schema's own detailed model/whitelist note and is actually less precise — it names Gemma 4 27B, Llama 4 Scout and Mistral Small 24B without noting they may be BYOK-only per the schema, which could mislead model selection.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (generate text) and resource (LLM models), and names both providers it can route to. It is clearly distinguishable from siblings like list_llm_models (catalog browsing) and compare_models (multi-model comparison).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete routing cue: use list_llm_models to see the full catalog before choosing a model, and signals the no-key/rate-limited context. It does not say when to prefer this over compare_responses or build_rag_prompt, so no explicit exclusions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds behavioral insight by enumerating the validation checks performed (required fields, types, enums, nested objects, arrays), which goes beyond the structured annotation data and gives the agent a clearer expectation of tool behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences, each earning its place. The first sentence states the core purpose, the second details specific validation coverage, and the third provides contextual importance. It is front-loaded and free of unnecessary fluff or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity, the description is complete: it states the purpose, the validation details, and the primary use cases. Since an output schema exists (as indicated by context signals), the description does not need to explain return values. It adequately covers the necessary context for an agent to select and invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already provides full descriptions for both parameters (100% coverage), which sets a baseline of 3. The description adds value by explaining the validation semantics (tests required fields, types, etc.), giving the agent a better understanding of how the parameters are used in practice.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Validate') and a clearly defined resource ('LLM JSON output' against 'a JSON Schema definition'). It explicitly calls out key validation aspects (required fields, types, enums, nested objects, arrays) and differentiates itself from generic validators by focusing on LLM outputs and structured output testing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use the tool: 'Critical for function-calling and structured output testing.' While it does not explicitly name alternatives or exclusion criteria, the stated use cases are specific enough to guide an agent, especially given the sibling-tool context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description's 'Validate' is consistent. It adds a list of validation categories, but these mostly mirror the schema parameters; no additional behavioral details like auth or return behavior. No contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence with a colon-separated list, front-loaded with verb and resource. Every clause earns its place and there is no filler or repetition of schema details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 9 parameters and a detailed schema, the description provides a solid high-level scope and use case. It doesn't need to explain return values because an output schema exists. A minor gap is not explicitly relating it to more specific sibling validators, but overall it is adequately complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with detailed per-parameter descriptions, so baseline 3 applies. The description adds a high-level grouping of parameter categories (format, content, length, language, safety) that helps conceptualize the tool, but it doesn't add new semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool validates LLM responses against QA criteria, enumerating specific checks (format, content, length, language, safety). This distinguishes it from sibling tools like compare_responses or llm_format_check by positioning it as a comprehensive QA validation utility.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It says 'Essential for QA testing LLM-powered features', giving clear context for when to use it. It doesn't explicitly name alternatives or exclusions, so it stops short of full usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, and the description adds substantial behavioral context beyond that: deterministic and free in tfidf mode, no model call, PASS/FAIL/INVALID semantics, INVALID meaning base refusal/too short, relation-specific thresholds, and the require_all off-switch warning. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense and long, but every sentence carries useful information for a complex 7-parameter tool. It is front-loaded with the core concept and then covers thresholds, return values, and guardrails. Slightly less structured than ideal, but not bloated or redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (nested objects, enums, output schema, 7 params), the description is remarkably complete: it explains return values, mode requirements, relation thresholds, the require_all warning, baseline_guard behavior, and the companion tool. It leaves no major behavioral gap for an agent to discover at runtime.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds meaning well beyond the schema: it explains mode differences (tfidf vs embeddings), relation thresholds and which are advisory, the require_all 'off switch' behavior with asymmetric pass rates, and the baseline_guard first-200-chars refusal matching. This materially helps an agent choose and set parameters correctly.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Reference-free stability primitive' and immediately states the specific behavior: 'checks that an assistant's answer stays INVARIANT when the QUESTION is transformed.' This clearly distinguishes it from ground-truth comparison tools and from sibling tools like run_semantic_tests. The verb+resource+scope is precise and memorable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly frames when to use it ('instead of comparing an answer to a ground truth'), what the caller must bring ('You bring the outputs (no model is called)'), and names the companion tool: 'Use run_semantic_tests alongside it: invariance without a correctness floor is a green light for a broken assistant.' This gives clear context and an explicit alternative/complement.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive, closed-world, so the safety profile is covered. The description adds genuinely useful non-annotation context: the catalog is a fixed set of "30+ models" from eight named providers, which tells the agent the coverage is bounded and a lookup for an unlisted model may fail. It still doesn't say what happens on an unknown model name (error vs empty result).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: the first front-loads the action and enumerates the return fields, the second states catalog breadth and providers. No filler, no restatement of the tool name, and the most decision-relevant information (what you get back) comes first.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present and full annotation coverage, the description need not explain return values, and it is complete enough to call correctly: one param, clear examples, known catalog scope. The one remaining gap is unknown-model behavior, which an agent would want to know before calling with a speculative name.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

One required parameter with 100% schema description coverage, including concrete examples ("gpt-4o", "claude-3.5-sonnet"), so the schema already carries the semantics. The description adds nothing about accepted naming variants, aliases, or case sensitivity beyond the schema, which is the baseline-3 outcome when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb plus resource ("Get detailed specs for an AI model") immediately followed by the exact fields returned: context window, pricing, knowledge cutoff, provider, multimodal, reasoning, features. This clearly separates it from list/compare-style siblings in spirit, but no sibling (e.g. list_llm_models, compare_models, context_window_check) is named, so the differentiation is implicit rather than explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied by the purpose: pass a model name to retrieve its specs. There is no statement of when to prefer this over list_llm_models (to discover valid names) or compare_models (side-by-side), and no note about model-name format beyond the schema examples. For a single-param lookup tool this is workable but leaves routing ambiguous in a 40-sibling cluster.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds operational detail beyond the readOnlyHint=true annotation: it notes that no API key is needed and describes the return payload (full haystack, question, metadata). It also clarifies the generation behavior (embedding a target fact at a specified position). No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences cover purpose, behavior, use case, and operational requirements. No filler or redundant phrases.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers what the tool does, when to use it, what it returns, and auth requirements. With an output schema and full parameter schema present, there are no significant gaps in context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already provides full descriptions for all four parameters (100% coverage), including defaults and enum options. The description adds little parameter-specific detail beyond implying position and token size, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool generates a needle-in-a-haystack test by embedding a target fact into filler text, and explicitly identifies the purpose (testing LLM context window retrieval). This distinguishes it from sibling text-generation tools like lorem_ipsum or few_shot_formatter.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly recommends the tool for testing LLM context window retrieval accuracy, giving a clear context. It does not mention exclusions or alternative tools, so it stops short of full usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is covered. The description adds that it returns a test plan with a scored rubric, which is useful but not extensive behavioral disclosure. No contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose and action, no wasted words. It efficiently states what the tool does and returns.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description orients the agent on the tool's role (test suite definition) and return value (test plan with rubric). Given the output schema and 100% parameter schema coverage, the description is sufficient for selection and invocation despite numerous parameters.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description references 'system prompt, user prompt, and expected output criteria,' loosely mapping to several parameters but adds no deeper meaning beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb and resource ('Define a test suite for a prompt') and clearly distinguishes from siblings like run_semantic_tests by stating it produces an input for evaluation, not the evaluation itself.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'use this as input for manual or automated LLM evaluation' provides clear context for when to use the tool. It does not explicitly name alternatives or exclude other tools, but the intended workflow is evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds valuable context beyond annotations by disclosing the TF-IDF algorithm and the fact that it works without external resources, which is useful for understanding what the tool does under the hood.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two concise sentences. The first sentence states the action and method; the second explains the purpose and avoids unneeded details. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity, full schema coverage, output schema presence, and rich annotations, the description provides enough context. It explains what the tool does, how it does it, and when it's useful, which is complete for a read-only utility.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all parameters are already described adequately in the schema. The description does not add extra parameter-level semantics, but the baseline of 3 applies because no additional meaning is needed beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states what the tool does: 'Rank an array of text chunks by relevance to a query using TF-IDF scoring.' It also distinguishes itself from sibling tools by specifying the algorithm and noting it 'Simulates retrieval ranking for RAG testing without needing embeddings or an API.'

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context: it is for RAG testing scenarios where embeddings or an API are unavailable. It does not explicitly name alternative tools or provide exclusion criteria, but the use case is well implied and focused.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds context beyond annotations: ratings are aggregated on a public dashboard, comment truncation behavior is described, and the tool's write action is clear (no annotation contradiction).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Single efficient paragraph that front-loads purpose, then moves through scoring guidelines, dashboard note, usage rule, and example—every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple 3-param tool with output schema, the description covers what the tool does, when to use it, how to use parameters, behavioral quirks (truncation), and an example—no gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%; the description enriches each parameter with usage guidance (e.g., comment is 'the most valuable part', score scale explained, tool_name example) and provides a full invocation example.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ("Give") and resource ("usage feedback on an IA-QA MCP tool"), and clearly distinguishes this rating tool from the many sibling utilities by its unique role of collecting feedback.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit criteria for when to rate low (1-2) vs high (4-5), and a direct instruction to skip routine successes ('we want signal, not praise'), plus a concrete example showing how to call the tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite good annotations, the description adds substantial behavioral context: BYO mode is deterministic/offline, BM25 mode is a lexical baseline with no neural evaluation, the NVIDIA endpoint was retired on 2026-05-18, and the API key 'transits RAM for the single call, never stored.' This fully discloses side effects and state implications beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is structured as a numbered list with a front-loaded purpose. It is detailed but every sentence earns its place, including context like the NVIDIA retirement date and key-storage behavior. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has three distinct execution modes, CI-gating semantics, and a historical deprecation caveat, all fully explained. Output schema exists, so return-value documentation is not required here. The description is complete for the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds meaningful semantic guidance: score must be present on every passage or none, api_key is used only when no passage carries a score, and higher scores mean more relevant. These details connect the parameters to mode selection in a way the schema alone does not.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict.' It clearly differentiates three modes and is distinct from sibling tools like rag_relevance_rank or bm25_score by emphasizing ground-truth evaluation rather than mere ranking.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance for each mode: BYO scores is 'the mode to gate CI on', BM25 mode answers a specific keyword-floor question and 'never is my neural reranker good', and the live NVIDIA mode is restricted to grandfathered accounts with reranking entitlement. This is exemplary usage differentiation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive), and the description adds substantial non-obvious behavior: the 0-100 scale is computed only over criteria actually measured, total_score is null with a reason when no criteria are given, and unscored signals (question-term overlap, avg sentence length, markdown) are also returned. This is context the annotations cannot convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded in the first clause, followed by return behavior and the sibling pointer. The parenthetical rationale about a one-word answer is a slight digression, but every sentence still carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description need not document return values, yet it usefully explains the null-score edge case and the extra unscored signals. Combined with the sibling routing and scope limits, an agent has everything needed to call and interpret this tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both optional criteria parameters and the required ones are already documented in the schema, including the 'scored as coverage' and 'proportional penalty' semantics. The description reinforces but does not materially extend that meaning, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('score an LLM response') and immediately names the exact criteria it scores against (expected_keywords coverage, max_length compliance). It also explicitly differentiates itself from the run_semantic_tests sibling, so an agent can route without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly names the alternative ('For meaning against a reference answer, use run_semantic_tests') and states the condition that selects it. It also defines scope boundaries by noting that relevance and correctness cannot be measured here, which tells the agent when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/openWorld/non-idempotent/non-destructive, so the safety profile is covered. The description adds real behavioral context beyond that: it makes LLM model calls per scenario and runs scorers, implying network access, latency, and cost. It does not mention rate limits, token cost, or runtime, which keeps it out of 5 territory.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the pipeline and outcome, followed by the routing hint. No filler, no restatement of the title, and every clause carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need not be explained, yet the description still summarizes them helpfully, and annotations plus a fully-covered schema handle safety and parameters. The only minor gap is that prerequisites such as required API keys or network/cost implications are left implicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the schema already explains contract_path, inline_contract, overrides, api_keys, and validate_only in detail. The description only alludes to the contract file and 'the specified LLM model', adding essentially nothing over the structured fields, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb chain (parse, call, run, return) over a named resource (.ia-eval.yaml LLM test suite) and states the concrete output (per-scenario Pass/Fail verdicts plus Markdown summary). This is distinguishable from siblings like run_semantic_tests, run_vlm_test_suite, and generate_eval_yaml without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a clear usage context (evaluating a .ia-eval.yaml suite) and explicitly routes the agent to list_local_tests to discover test files, which is a concrete prerequisite. It stops short of saying when NOT to use it or how it differs from run_semantic_tests/run_vlm_test_suite for adjacent evaluation tasks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is clear. The description goes beyond annotations by explaining return values (verdicts) and the requirement for an API key in embeddings mode, adding valuable behavioral context without contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured, with three sentences that front-load the core purpose and then add mode details and CI relevance. Every sentence contributes useful information without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (5 params, nested objects, output schema), the description covers key aspects: modes, API key requirement, default thresholds, verdict output, and CI use case. The existing output schema handles return-value details, so the description is sufficiently complete for an agent to select and invoke the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds some semantic context by mentioning 'tfidf (default)' and 'BYOK', but these are largely redundantly covered in the schema parameter descriptions. It does not meaningfully enhance parameter understanding beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly defines the tool as a 'Semantic assertion primitive' that compares actual vs expected text pairs using cosine similarity and ROUGE-L. It distinguishes between tfidf and embeddings modes, and mentions returns of per-case and overall verdicts, which clearly differentiates it from generic similarity tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use each mode (tfidf is free/lexical, embeddings is true semantic with BYOK) and notes it is CI-ready. However, it does not explicitly name alternative sibling tools or state when not to use this tool, so it falls short of full exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnly, openWorld, non-idempotent), the description discloses the BYOK auth requirement and key formats, the external provider calls to GPT-4o/Claude/Gemini, the full return shape (per-case verdicts, pass rate, overall PASS/WARNING/FAIL, latency), an adjustable threshold, and the exact semantics of semantic_contains (TF-IDF cosine ≥ 0.4). These are material behavioral facts an agent needs before firing a billable network call.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the purpose, then inputs, then outputs, then assertion taxonomy and the BYOK caveat — no filler sentences. The em-dash chaining makes it denser than ideal, but every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be restated, yet the description still summarizes verdict structure usefully. Given 8 parameters and external API dependencies, the only meaningful gaps are failure modes (invalid key, provider rate limits, cost per run) and whether the non-idempotent latency stats vary across runs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3; the description earns extra credit by explaining the two mutually exclusive image inputs (URL or base64), the question+assertion structure of each test case, and the assertion-type vocabulary with the semantic_contains similarity floor. It doesn't add anything about threshold defaults or the 10-case cap beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a precise verb+resource: 'Run a test suite against a Vision-Language Model', and immediately specifies the inputs (image + N test cases with question/assertion) and the supported providers. This clearly separates it from text-only siblings like prompt_test_suite and run_semantic_tests, though it never names a sibling to route against explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: you'd reach for this when you want to evaluate a VLM's answers on an image against assertions. There is no statement of when NOT to use it, no mention of prompt_test_suite / compare_models / run_semantic_tests as alternatives, and no prerequisites beyond the BYOK note.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive. The description adds that no API key is required and that batch mode is supported, giving useful operational context beyond the annotations. There is no contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences front-load the purpose and metric list, then add use cases and batch support. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers purpose, use cases, metrics, auth requirements, and batch mode. The threshold parameter's behavior is in the schema, and the output schema exists, so return values are covered. It is complete for a read-only scoring tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all five parameters. The description mentions the metric names and batch mode, but doesn't add syntax or format details beyond what the schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Compute text similarity between reference and hypothesis' with a specific verb and resource, then enumerates six distinct metrics. This distinguishes it from sibling tools like embedding_similarity or levenshtein_distance by specifying the exact metric set and batch capability.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides explicit use cases: 'Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking.' It does not explicitly name alternatives or exclusion criteria, but the context is clear enough for an agent to know when to invoke this tool over similar ones.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover safety (readOnlyHint, destructiveHint=false), and the description adds substantial extra context: rate-limiting on the server key, auth via api_key+model, temperature 0, verdict thresholds (PASS ≥80/FIX ≥50/BLOCK), and an explicit data-privacy disclosure that the SKILL.md and scenarios are sent to ia-qa.com and the provider. It also explains that unreadable responses yield INCONCLUSIVE rather than a miss.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the two-stage structure and dense with actionable facts rather than filler, though the single long paragraph is heavy and could be split for readability. Nearly every clause carries operational meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists so return values need no prose, and the description still explains verdicts, step_adherence, scenarios_source, and failure semantics. Given the tool's complexity and 8 parameters, nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema carries the parameter definitions (baseline 3). The description still adds meaning: scenarios capped at 20, the should-load/near-miss split, the six built-in default neighbours, and scenario_count defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Test a SKILL.md in the open Agent Skills format') and enumerates the two stages (lint, triggering). It is instantly distinguishable from sibling linters like mcp_schema_lint or llm_format_check because it names the format and the two-stage process.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly says when to use each mode: lint_only is 'the CI gate for a SKILL.md — gate on lint.valid', and passing your own scenarios yields 'a reproducible gate' versus generated ones being 'a signal'. It does not name sibling alternatives explicitly, so it falls just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish this is a read-only, idempotent, closed-world computation with no side effects, so the safety profile is fully covered. The description adds two genuine traits beyond that: it emits a warning when the budget exceeds the model context window, and it supports 25+ models. It does not say how estimates are derived or how the total is reported, but with an output schema present that gap is minor.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the core allocation-planning purpose, followed by the warning behavior and model coverage. Nothing is redundant with the name and every sentence carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With annotations covering side-effect safety, full schema coverage, and a declared output schema, the definition covers what an agent needs to decide and call correctly. The only real gap is how the four buckets interact with the raw-text vs. token-count parameter pairs, which is left to the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all eight parameters (model, three raw-text inputs, three token-count inputs, expected output) are already documented in the schema. The description's enumeration of buckets roughly mirrors those parameters and adds no format, precedence, or mixing guidance (e.g., raw text vs. pre-computed tokens), so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb (plan) and resource (token allocation) and enumerates the four budget buckets it covers, plus an overflow-warning behavior. It is clear what the tool does, though it never explicitly distinguishes itself from close siblings like count_tokens, context_window_check, or estimate_llm_cost.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: the word 'plan' suggests pre-flight budgeting before constructing a prompt, and the warning behavior hints at validating a proposed allocation. There is no explicit when-to-use, when-not-to-use, or 'use X instead when you only need Y' routing against the many sibling token/cost tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.