Server Quality Checklist

Tool Scores

• searchDocsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It does not explicitly state read-only behavior, authentication needs, rate limits, or what happens on empty results. The description is too brief for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence that front-loads the purpose and result, with no wasted words. Efficient and to the point.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema exists, so description should convey return format. It mentions 'full content and sources' but does not specify structure (list, array, etc.) or pagination. Adequate but not detailed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for query, topK, and minScore. The description adds no extra semantics beyond 'returns relevant sections,' so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs semantic search over GMAT documentation and returns relevant sections with content and sources, distinguishing it from sibling tools focused on idioms, samples, and running GMAT.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for general search but does not explicitly state when to use this tool over alternatives or provide exclusions. Sibling tools suggest specific purposes, but no direct guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runGmatA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool runs 'headless' and returns a validated outcome, implying no persistent side effects. However, it does not explicitly disclose safety, resource usage, or whether the script modifies system state, leaving some ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no wasted words. It front-loads the core purpose and output shape, then clarifies the usage pattern and stage values. Every sentence is informative and earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description specifies the return structure ({ok, stage, errors, reports, raw_tail}) and defines the possible stage values. This covers the essential information for the agent to understand the tool's behavior. The tool is moderately complex with two parameters, and the description is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value by advising to increase timeout for 'multi-day low-thrust propagations', which is useful guidance beyond the schema's default value description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Run' and the resource 'GMAT mission script (text)', and explains the outcome as a validated result with specific fields. It distinguishes itself from sibling tools like getGmatIdioms or listGmatSamples by focusing on execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly frames the tool as part of a validation loop ('write, run, read, fix, repeat'), which gives context for when to use it. However, it does not explicitly state when not to use or compare with alternatives beyond the implied purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• getGmatIdiomsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears the full burden of behavioral disclosure. It fully describes the content and purpose, implying read-only behavior and no side effects. While it doesn't detail caching or access requirements, these are not critical for this simple retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with embedded examples, effectively front-loading the purpose. While slightly long, every phrase adds value and no redundancy exists.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, no output schema, and sibling tools focusing on other operations, the description is complete enough. It explains what the tool returns and why it is useful, though it could hint at return format for completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, so the description need not explain parameter semantics. The description adds value by detailing the content returned, surpassing the baseline expectation for a parameterless tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a curated knowledge base of GMAT idioms and gotchas, with specific examples. It distinguishes itself from sibling tools (getGmatSample, listGmatSamples, runGmat, searchDocs) by focusing on preventing script errors.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises reading this before writing GMAT scripts, providing clear context for use. However, it does not specify when not to use it or mention alternative tools, though the sibling tools serve different purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• getGmatSampleA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It correctly identifies the operation as a read ('Return the full text'), but does not disclose error handling, authentication requirements, or potential side effects. The 'seed a phase' hint adds mild context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, immediately front-loading the primary purpose. Every word serves a purpose; there is no redundancy or extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (single parameter, no output schema, no nested objects), the description covers the essential aspects: what it returns, how to identify the file, and a suggested use case. It feels complete for the agent's decision-making.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description adds meaningful examples showing filename format (e.g., 'Ex_SafetyEllipse.script') and supports the 'from listGmatSamples' hint, going beyond the schema's basic 'Sample file name' description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Return') and resource ('NASA sample script'), and distinguishes from sibling tools by stating it retrieves by file name from 'listGmatSamples'. This leaves no ambiguity about the tool's core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states to 'Use as a known-good template to seed a phase', providing a clear use case. It also implicitly advises that filenames come from 'listGmatSamples', but does not mention when not to use or alternatives like 'getGmatIdioms' or 'runGmat'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• listGmatSamplesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It explains the scope (NASA official plus community corpus), tagging, and naming convention. It's clear what the tool does and what the output represents. Minor omission: no mention of ordering or pagination, but for a parameterless list it's largely transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single paragraph, well-structured with clear information. Could be slightly more concise by removing the ellipsis at the end, but overall it's efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and no output schema, the description provides sufficient context: it explains what the tool lists, the source of samples, and how to use it in conjunction with getGmatSample. It is complete for the tool's purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters (schema coverage 100%), so the baseline is 4. The description adds value by explaining what the output contains (tags, community prefix), which compensates for the lack of parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists available GMAT sample scripts with tags for techniques. It distinguishes from siblings like getGmatSample and runGmat by explicitly mentioning that the list is for browsing and then using getGmatSample to read.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance: 'Scan the tags to pick the right seed for a mission, then getGmatSample to read it.' This tells when to use this tool (to browse/list and select) and when to use getGmatSample (to read the actual content).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.