Server Quality Checklist

Tool Scores

• explore_repoC
  2.1/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries full burden. It only says 'Explore one configured repository with codegraph explore', giving no information about side effects, safety, permissions, rate limits, or what happens during exploration. For a tool that likely performs actions (running a command), this is critically insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and front-loaded with the core purpose. However, it is under-specified, missing essential details. It earns a 4 for efficiency but could improve by adding more context without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters, an output schema, and 4 sibling tools, the description is far too brief. It does not explain what the tool returns, how parameters affect behavior, or how it relates to siblings. The presence of an output schema does not compensate for the lack of description about the overall operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must add meaning to parameters. It does not. Parameters like repo, query, max_files, timeout_seconds are only listed by name and title, with no explanation of their purpose, format, or behavior. The description does not reference any parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states the tool explores one configured repository, but 'explore' is vague and doesn't specify if it's searching, browsing, or analyzing. The reference to 'codegraph explore' adds some context but doesn't clarify the exact function. Compared to sibling tools like ask_multi_repo, the distinction is implied but not explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus siblings. The description implies it's for a single repository (contrast with ask_multi_repo), but does not state when to use or not use it. No prerequisites or alternatives mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask_multi_repoC
  2.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Only mentions 'CodeGraph exploration' without explaining behavior (e.g., mutation, auth, rate limits). Very little transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence; overly brief for a tool with 4 parameters and multiple repos. Lacks structure; important details omitted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, description does not explain return values. With 4 parameters and no schema descriptions, the description is woefully incomplete for reliable invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. Description does not explain any parameter. While names like 'repos' and 'max_repos' hint at usage, no addition meaning provided beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool routes a question to multiple repositories and runs CodeGraph exploration. It distinguishes from sibling tools like explore_repo (single repo) and trace_across_repos (trace vs exploration). However, 'CodeGraph exploration' is not defined, so purpose is clear but not fully specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives. The agent must infer from tool names. No mention of exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trace_across_reposC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It does not disclose behavior beyond the basic action, such as whether it searches all repos, permissions needed, or response format. The output schema exists but is unmentioned.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that front-loads the purpose. It is efficient but could benefit from brief elaboration without verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (4 parameters, 1 required, no annotations), the description is incomplete. It does not explain parameter usage or provide enough context for effective invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no meaning for the parameters besides the identifier. The optional parameters repos, max_repos, and max_files_per_repo are not explained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool traces an identifier across repositories, listing examples like symbol, API route, etc. It is specific but does not explicitly distinguish from sibling tools like ask_multi_repo.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings. There are no explicit when-not or alternatives, and the context is only implied from the name and description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_reposC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden. It states the action ('refresh') but does not disclose side effects, permissions, rate limits, or whether it fetches from remote. Minimal behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with verb and resource. Concisely states the core function and optional behavior. Slightly informal but efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single parameter and an output schema (exists), the description is minimally adequate. It lacks details on what 'refresh' entails (e.g., local vs remote), but covers the optional parameter.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description adds value by explaining the optional parameter include_status ('optionally include codegraph status output'). However, it does not elaborate beyond the parameter's basic effect.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Refresh' and resource 'repository metadata', clearly distinguishing it from sibling tools like list_repos or explore_repo. However, it could be more explicit about what 'metadata' includes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives (e.g., list_repos for current state, explore_repo for details). The tool's purpose is implied but not contextualized.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_reposA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description bears full responsibility for behavioral transparency. It only states the action without mentioning side effects, permissions, pagination, or return format. A simple read operation is implied but not confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that fronts the key information. Every word is meaningful and there is no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity and the presence of an output schema, the description provides minimal context. It lacks information about scope (e.g., which repositories are listed) or prerequisites. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, so the input schema coverage is 100%. With zero parameters, the baseline is 4, and the description adds no further parameter semantics but also does not need to.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (List) and the resource (repositories configured for multi-repo CodeGraph exploration). It explicitly distinguishes the tool's purpose from sibling tools like explore_repo or ask_multi_repo, which involve different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that the tool is used to list repositories, but it does not provide explicit guidance on when to use it versus alternatives. Sibling tool names suggest distinct purposes, but no contextual notes are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.