Server Quality Checklist

Tool Scores

• list_company_settingsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the output is formatted as Markdown and that encoded settings are decoded, which adds useful behavioral context. However, it does not mention whether pagination occurred, potential large payloads, read-only implications, or any other caveats, leaving some behavioral aspects undisclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, two short sentences, front-loaded with the action, and contains no extraneous words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description adequately covers the core purpose and return format for a simple list tool, especially given the rich schema. It states the output is Markdown with decoded values, which is enough given no output schema. However, it does not mention potential issues like large result sets or any limitations (e.g., inability to filter without optional params), but the schema offsets this. Overall, it is nearly complete for a list operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with all six parameters having clear descriptions including enums and defaults. The description adds no extra per-parameter meaning, so the schema already does the heavy lifting. The global mention of decoding does not clarify parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists company settings from the Transporeon settings API, using the specific verb 'list' and a clear resource. It also mentions the return format (Markdown with decoded values). However, it does not explicitly differentiate from sibling tools like get_company_setting or search_in_setting, though the name and 'all' hint at a broader scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention that get_company_setting should be used for a single setting or search_in_setting for searching. No exclusions, prerequisites, or alternative recommendations are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_company_settingA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the transparency burden. It adds a useful behavioral detail: 'Returns the setting with decoded value if it was encoded.' However, it does not mention error handling, authentication requirements, or how pagination parameters affect results, leaving room for more disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence with no redundant information. It efficiently conveys the core action and a key behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple get tool, the description covers the essential intent and the decoding behavior. No output schema is provided, so the return value is only vaguely described as 'the setting,' but the parameter schema handles the optional fields. It could further clarify the purpose of offset/limit for line-based settings, but overall it's reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers all 8 parameters with descriptions, so the baseline is 3. The description adds no extra parameter meaning beyond naming the key; it doesn't elaborate on optional filters like type, owner, or environment.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Get a specific company setting by key name.' It uses a specific verb ('Get'), identifies the resource ('company setting'), and narrows scope ('specific... by key name'), which distinguishes it from sibling tools like list_company_settings and search_in_setting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'specific... by key name' implies the tool is for retrieving a single known setting, but it does not explicitly mention when to use it instead of list or search alternatives. No exclusions or alternative tool references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_in_settingA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool returns matching lines with surrounding context, which is useful. However, it omits details like case sensitivity, search pattern semantics (exact vs. substring), and behavior when no matches are found. This is a moderate level of transparency for a read-like search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is direct and front-loaded with the action and target. It contains no fluff or redundant information, making it highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 8 parameters and no output schema, so the description needs to provide enough context. It states the core purpose but doesn't explain how optional filters (type, owner, childObject, environment) narrow the search, nor does it describe the output structure beyond 'matching lines with context.' The schema covers parameter details, but the description could offer more cohesive guidance for this multi-parameter search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage for all 8 parameters, so the baseline is 3. The description's mention of 'surrounding context' aligns with the contextLines parameter but does not add new meaning beyond the schema's own descriptions. No additional parameter semantics are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('search') and resource ('company setting'), and mentions returning matching lines with context. This distinguishes it from siblings like list_company_settings and get_company_setting, which have different purposes (listing vs. retrieving full settings).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when you need to find specific text within a setting's value, which is a clear use case. However, it does not explicitly say when to use this instead of alternatives, such as get_company_setting for full retrieval. The sibling names provide context, but no exclusions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_environment_tokensA
  4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral disclosure. It mentions making test requests, which implies network calls, but doesn't disclose side effects, rate limits, required permissions, or how results are returned. The behavior on invalid tokens or failure is also unspecified, leaving significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action, and every word adds value. It's efficient and well-structured without unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter diagnostic tool, the description is fairly complete: it states what it does and why to use it. However, without an output schema or annotations, it could still mention the format or meaning of verification results, leaving a small gap in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is trivially complete. The description adds no parameter information because none is needed, and the baseline for 0-param tools is 4, which is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: verifying environment tokens (pd, in, ac) by making test requests. This distinguishes it from sibling tools like list_company_settings, which target company settings, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by stating the tool 'helps diagnose authentication issues,' implying it should be used for troubleshooting auth problems. However, it doesn't explicitly mention when not to use it or compare to alternatives, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.