Server Quality Checklist

Tool Scores

• get_configC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states this is a retrieval operation, implying read-only behavior, but doesn't disclose critical details like authentication requirements, rate limits, error conditions, or response format. This is inadequate for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the purpose. It could be slightly more structured by separating usage context, but it's appropriately sized with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete. It doesn't explain what the return value looks like (e.g., JSON structure), error handling, or prerequisites. For a retrieval tool, this leaves significant gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents the 'slug' parameter. The description adds no additional parameter information beyond what's in the schema, but the baseline is 3 since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Retrieve detailed information') and resource ('specific configuration'), specifying the type of information returned (cache settings, retry policies, routing strategy). It doesn't explicitly distinguish from sibling tools like 'list_configs', which would be needed for a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention the sibling 'list_configs' tool for listing configurations or other get_* tools for different resources, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_cost_analyticsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool retrieves data 'over time' and includes specific metrics, but doesn't describe important behaviors: whether this is a read-only operation, if it requires specific permissions, how results are formatted (aggregated vs. raw), pagination, rate limits, or error conditions. For a complex analytics tool with 21 parameters, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose. Every word contributes to understanding what the tool does. However, for such a complex tool with many parameters, additional context about usage or behavior might be warranted, making it slightly too concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (21 parameters, no output schema, no annotations), the description is inadequate. It doesn't explain the return format, aggregation methods, or how the numerous filtering parameters interact. For an analytics tool that likely returns structured data, the description should provide more context about what 'detailed cost analytics data' actually includes and how it's organized.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 21 parameters thoroughly. The description adds minimal value beyond the schema - it mentions 'over time' which aligns with the time range parameters, and 'total costs and averages per request' which relates to cost and token parameters. However, it doesn't provide additional context about parameter interactions or filtering logic beyond what's in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('retrieve') and resources ('detailed cost analytics data'), including what data is returned ('total costs and averages per request'). However, it doesn't differentiate this tool from potential sibling analytics tools (none are listed as siblings, but the description doesn't mention this uniqueness).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There's no mention of prerequisites, appropriate contexts, or comparisons to other tools (like get_user_stats or get_workspace that might provide related data). The agent must infer usage from the tool name and parameters alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_user_statsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden but provides minimal behavioral insight. It mentions retrieval of analytics data but doesn't disclose pagination behavior (implied by 'page_size' parameter), rate limits, authentication requirements, or whether the operation is read-only (implied by 'retrieve'). The description adds little beyond the basic operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the core purpose without redundancy. It front-loads key information ('retrieve detailed analytics data') and uses precise terminology, with no wasted words or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 9-parameter analytics tool with no annotations and no output schema, the description is inadequate. It doesn't explain the return format, pagination behavior, error conditions, or how filters interact. The agent lacks sufficient context to use this tool effectively beyond basic parameter passing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 9 parameters. The description adds no parameter-specific details beyond implying time-range filtering, which is already covered in the schema. Baseline score of 3 is appropriate as the schema does the heavy lifting, but the description doesn't enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('retrieve') and resource ('detailed analytics data about user activity'), specifying the data types ('request counts and costs') and temporal scope ('within a specified time range'). It distinguishes from siblings like 'get_cost_analytics' by focusing on user-level metrics rather than cost aggregation, though the distinction could be more explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'get_cost_analytics' or 'list_all_users'. The description mentions a time range but doesn't specify prerequisites, exclusions, or comparative use cases with sibling tools, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_configsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions retrieving data but doesn't disclose behavioral traits like pagination, rate limits, authentication needs, or whether it's read-only. 'Retrieve' suggests a read operation, but this isn't explicitly stated. The description adds minimal behavioral context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with the core action ('Retrieve all configurations'), followed by specifics on what's included. Zero waste, appropriately sized for a simple list tool. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and a simple tool with 0 parameters, the description is incomplete. It lacks details on return format (e.g., list structure, fields), behavioral constraints, or error handling. For a tool that retrieves organizational data, more context on scope and limitations would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (though empty). The description doesn't need to add parameter details, so baseline is 4. It implicitly confirms no filtering or input is required by stating 'all configurations', aligning with the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Retrieve') and resource ('all configurations in your Portkey organization'), specifying what data is included ('their status and workspace associations'). It distinguishes from siblings like 'get_config' (singular) and 'get_workspace' (different resource), though not explicitly. Purpose is specific but could better differentiate from other list tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like 'list_all_users' or 'list_workspaces'. The description implies it's for retrieving configurations broadly, but doesn't specify prerequisites, context, or exclusions. Usage is implied by the resource name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_all_usersB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool lists users, implying a read-only operation, but doesn't disclose behavioral traits such as pagination, rate limits, authentication requirements, or what happens if no users exist. For a tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the key action ('List all users') and includes essential details ('in your Portkey organization, including their roles and account details'). There is no wasted language, and it's appropriately sized for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (0 parameters, no output schema, no annotations), the description is minimally complete. It states what the tool does but lacks behavioral context (e.g., output format, error handling). Without annotations or output schema, more detail on return values or usage constraints would improve completeness, but it's adequate for a basic list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so no parameter documentation is needed. The description doesn't add parameter semantics, but with no parameters, the baseline is 4 as it adequately covers the tool's purpose without unnecessary details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all users in your Portkey organization'), specifying what fields are included ('their roles and account details'). It distinguishes from some siblings like 'get_user_stats' (statistics vs. list) and 'invite_user' (create vs. list), but doesn't explicitly differentiate from all list-type tools like 'list_configs' or 'list_workspaces'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. It doesn't mention prerequisites, timing considerations, or comparisons with siblings like 'get_user_stats' (which might provide aggregated data) or 'invite_user' (which is for adding users). The description implies a comprehensive listing but offers no context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• invite_userB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions the action ('invite') and some permissions aspects, it doesn't disclose important behavioral traits like: whether this requires admin privileges, whether it sends an email invitation, what happens if the email is already associated with an account, rate limits, or what the response looks like. For a user invitation tool with zero annotation coverage, this leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose. Every word earns its place by specifying the action, target, and key capabilities. There's no redundancy or unnecessary elaboration, making it appropriately concise for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a user invitation tool with 6 parameters (including nested objects), no annotations, and no output schema, the description is insufficient. It doesn't explain what the tool returns, what permissions are required to use it, whether it's idempotent, or how errors are handled. The description provides basic purpose but lacks the contextual completeness needed for a mutation tool of this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents all 6 parameters thoroughly. The description adds minimal value beyond what's in the schema - it mentions 'workspace access and API key permissions' which aligns with the 'workspaces' and 'workspace_api_key_details' parameters, but doesn't provide additional context about parameter relationships, defaults, or usage patterns. The baseline of 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('invite a new user') and the target resource ('your Portkey organization'), while also specifying the scope of the invitation ('with specific workspace access and API key permissions'). It distinguishes this tool from its siblings (which are primarily read-only 'get' and 'list' operations) by being the only user creation/invitation tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool should be used when you need to add a new user to the organization with specific permissions, but it doesn't provide explicit guidance on when to use it versus alternatives. There's no mention of prerequisites (e.g., admin permissions required) or what happens if the user already exists. The context is clear but lacks explicit exclusions or alternative scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_virtual_keysB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions what information is retrieved (usage limits, rate limits, status), it doesn't describe important behavioral aspects like whether this requires authentication, how results are returned (pagination, format), rate limits on the operation itself, or what happens if no virtual keys exist. The description provides some context about what's included but misses key operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that packs substantial information about what's retrieved. It's appropriately sized for a zero-parameter tool and front-loads the core purpose immediately. There's no wasted verbiage, though it could potentially be structured to include usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list operation with no output schema, the description provides adequate but incomplete coverage. It specifies what resource is retrieved and what information is included, but lacks details about the return format, authentication requirements, error conditions, or relationship to other tools. The absence of annotations means the description should do more heavy lifting than it currently does.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters with 100% schema description coverage, so the baseline is 4. The description appropriately doesn't discuss parameters since none exist, and it focuses on what the tool retrieves rather than parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'retrieve' and the resource 'all virtual keys in your Portkey organization', with specific details about what information is included (usage limits, rate limits, status). It distinguishes itself from sibling tools like list_all_users or list_workspaces by focusing specifically on virtual keys.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. While it's clear this is for retrieving virtual keys, there's no mention of when you'd use this versus other list_* tools or how it relates to other Portkey management tools. No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_workspacesA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the scope ('all workspaces', 'configurations and metadata') but lacks critical behavioral details: pagination behavior (implied by parameters but not explained), rate limits, authentication requirements, error conditions, or response format. The description doesn't contradict annotations since none exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence efficiently conveys core purpose with zero waste. Front-loaded with main action ('Retrieve all workspaces'), followed by scope details. Every word earns its place without redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list tool with 2 documented parameters but no annotations and no output schema, the description is minimally adequate. It covers what the tool retrieves but lacks behavioral context (pagination, authentication, response format) that would be helpful given the absence of structured metadata. Completeness is borderline for a tool that presumably returns complex workspace objects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both pagination parameters. The description adds no parameter-specific information beyond what's in the schema. Baseline 3 is appropriate when schema does all parameter documentation work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Retrieve' and resource 'all workspaces in your Portkey organization', specifying scope with 'including their configurations and metadata'. It distinguishes from sibling 'get_workspace' (singular) by indicating retrieval of multiple workspaces. However, it doesn't explicitly differentiate from other list tools like 'list_all_users' or 'list_configs' beyond resource type.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving workspace collections rather than single workspaces (contrasting with 'get_workspace'), but doesn't provide explicit when-to-use guidance, alternatives, or exclusions. No mention of prerequisites, authentication needs, or comparison with other list tools is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_workspaceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes the operation as a retrieval (implying read-only) and specifies the type of information returned, but lacks details on permissions required, error handling, or response format. This is adequate for a simple read tool but misses behavioral context like rate limits or authentication needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose and includes key details without redundancy. Every word adds value, making it appropriately sized and easy to parse for an AI agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 parameter, no output schema, no annotations), the description is mostly complete: it states the purpose, scope, and return details. However, it could improve by mentioning the lack of output schema or clarifying that it returns unstructured data, which would help the agent understand response handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter (workspace_id) well-documented in the schema. The description does not add any parameter-specific details beyond what the schema provides, such as format examples or constraints. Baseline 3 is appropriate since the schema handles the parameter documentation effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('retrieve') and resource ('workspace'), specifying the scope ('detailed information about a specific workspace') and what details are included ('configuration, metadata, and user access details'). It distinguishes this from siblings like list_workspaces by focusing on a single workspace rather than listing multiple.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving details of a specific workspace, which differentiates it from list_workspaces (for listing multiple) and other siblings like get_user_stats (for user data). However, it does not explicitly state when not to use this tool or name alternatives, such as clarifying that list_workspaces should be used first to find workspace IDs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.