Server Quality Checklist

Tool Scores

• ibm_granite_chatA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnly and openWorld hints. The description adds valuable information: payment model ($0.02 USDC via x402), that no IBM Cloud account is needed, and that token usage is returned. These details go beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, efficiently conveying the core functionality, input format, output, and pricing without redundancy. It is well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (four parameters with full schema descriptions), the description covers the essential context: input/output nature, default model, and payment. It lacks detail on response format but annotations and schema fill in gaps adequately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds context by mentioning 'role/content message pairs' and default model, which clarifies the use of the messages array and model parameter beyond schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a chat completion tool using IBM Granite models. While it doesn't explicitly differentiate from sibling tools, the names of siblings (analyze, code, embed, forecast, getting_started) imply distinct tasks, so the purpose is sufficiently clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide guidance on when to use this tool versus alternatives or when not to use it. It lacks explicit context for selection criteria among siblings or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ibm_granite_analyzeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true and openWorldHint=true, and the description adds that the tool returns structured JSON, costs $0.04 per call, and requires no account. No contradictions. It provides useful behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the main purpose, followed by cost and output format. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains return contents (entities, sentiment, etc.), compensating for lack of output schema. It also addresses pricing and authentication (no account needed). Together with sibling names, it provides sufficient context for an agent to decide to use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (all parameters have descriptions). The description reinforces the tool's capabilities but does not add new parameter-level detail beyond what the schema already provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies the verb 'extract entities, sentiment, risk signals, a concise summary, and recommended next steps', the resource 'any text', and distinguishes from siblings like ibm_granite_chat or ibm_granite_code. It is clear and specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions input types (contracts, reports, emails) and cost/payment method, providing context for when to use. However, it does not explicitly state when not to use or compare with alternatives, though the sibling list provides implicit differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ibm_granite_embedA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations (readOnlyHint, openWorldHint, idempotentHint) declare safety and idempotency. The description adds context: uses IBM Granite model with a specified default, returns float arrays per input, and notes no IBM Cloud account required. This is non-contradictory and enhances transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading the essential purpose and key constraints. Every word adds value; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple embedding tool with full schema and annotations, the description covers return type (float array), limits, cost, default model, and use cases. No output schema exists, but the description adequately explains output. Completeness is high.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters. The description adds the default model string and reiterates limits (1-64 texts, 8000 characters), which is helpful but only marginally extends schema information. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses precise language: 'Generate embedding vectors for one or more texts' and specifies the default model, output format (float array), and use cases (semantic search, RAG retrieval, similarity scoring). It clearly distinguishes from sibling tools like ibm_granite_chat or ibm_granite_code.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states the maximum number of texts (64) and the cost model ($0.005 USDC per call via x402), providing practical usage guardrails. It does not explicitly mention when not to use or alternatives, but the sibling names make the distinction clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ibm_granite_forecastA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint and openWorldHint, which are consistent with the forecasting and payment model. The description adds value by specifying the zero-shot nature, the underlying model (TTM), and the cost per call, providing behavioral context beyond annotations. No contradiction found.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences long, with the key action in the first sentence. It efficiently covers purpose, required inputs, and additional context (cost, suitability) without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of time-series forecasting and full schema coverage, the description provides sufficient context: model type, input requirements, and cost. While it mentions receiving the forecast horizon, it could elaborate on output format, but overall it is complete for an agent to use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 100% parameter description coverage. The description adds context by summarizing the required inputs (timestamps, values, cadence) and mentioning the default prediction length (typically 96 for 1h data), which aids understanding beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Zero-shot time-series forecasting' using IBM Granite TTM, clearly identifying the verb (forecast) and resource (time series). It distinguishes from sibling tools that focus on analysis, chat, code, embedding, or getting started.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions when to use (suitable for revenue, traffic, sensor, energy, financial series) and that no training or IBM Cloud account is required. It implies alternatives are not needed but does not explicitly state when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ibm_granite_getting_startedA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnly and idempotent. The description adds that it is free and returns an overview, which is consistent. It provides useful context beyond annotations (e.g., no payment needed) but does not disclose additional side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with 'FREE — start here,' and every word adds value. No redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple orientation tool with no required parameters and a clear enum, the description fully covers what the tool returns (overview, prices, payment, tools, setup, examples). No output schema is needed as the return content is described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters with descriptions, including enum values and defaults. The tool description repeats the enum values but adds no new meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns an overview of server tools, prices, payment flow, setup, and examples. It explicitly positions itself as the starting point, distinguishing from sibling paid tools by being free and introductory.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'start here' and 'Call this first to orient before invoking a paid tool,' providing clear guidance on when to use this tool. It also notes that no payment is required, reinforcing its role.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ibm_granite_codeA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses cost ($0.025 USDC per call) and account requirement (no IBM Cloud account needed) beyond annotations. Annotations already indicate read-only and open world, and description adds useful behavioral context without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences with clear front-loading of purpose, followed by usage and cost. No wasted words; every sentence adds essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers task types, prompt usage, cost, and account requirement. Lacks detailed return value specification, but the mention of 'receive code with explanation' suffices given no output schema. Could mention rate limits or max response length, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds extra value by summarizing that prompt differs per task (code for review/refactor/explain, description for generate) and stating the output type (code with explanation).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool handles code generation, review, refactoring, and explanation via IBM Granite. Lists specific task types, distinguishing it from sibling tools like ibm_granite_chat or ibm_granite_analyze.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context on when to use (for code tasks) and what to provide (task and prompt). Lacks explicit exclusions or alternatives, but the mention of 'Provide a task type and code/prompt' gives enough guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.