Server Quality Checklist

Tool Scores

• list_asr_modelsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. While 'List' implies a read-only operation, the description doesn't specify whether this requires authentication, what format the capabilities are returned in, if there are rate limits, or if the list is static or dynamic. For a tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any fluff. It's front-loaded with the core action ('List all available ASR models') and adds necessary detail ('and their capabilities'). Every word earns its place, making it highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters) and the presence of an output schema (which should document the return values), the description is minimally adequate. However, it lacks context about when to use it relative to siblings and behavioral details not covered by annotations (which are absent). This makes it incomplete for optimal agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so there are no parameters to document. The description appropriately doesn't mention any parameters, which is correct for this case. Baseline 4 is applied as per the rules for 0 parameters, since no additional parameter semantics are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('all available ASR models and their capabilities'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'speech_to_text' which might also involve ASR models, though the distinction is somewhat implied by the listing vs. processing focus.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention whether this should be used for discovery before invoking 'speech_to_text', or if it's for administrative purposes. With sibling tools like 'diarize_speech' and 'speech_to_text' that likely use ASR models, the lack of contextual guidance is a notable gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_textA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively adds context beyond basic functionality: it warns about API costs and specifies the external service ('Whissle'), which are crucial behavioral traits. It doesn't cover rate limits, authentication needs, or error handling, but provides valuable operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It front-loads the core purpose, follows with a critical warning, and then details parameters and returns in a clear format. Every sentence earns its place, with no redundant or verbose language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no annotations, but with output schema), the description is reasonably complete. It covers purpose, cost implications, parameters, and return type. The output schema handles return values, so the description doesn't need to explain them. It could benefit from more parameter details or error scenarios, but it's largely adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It lists all three parameters (content, model_name, instruction) with brief explanations, adding meaning beyond the bare schema. However, it doesn't elaborate on parameter constraints (e.g., model options, instruction format) or provide examples, leaving some semantic gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Summarize text using an LLM model.' This specifies the verb ('summarize'), resource ('text'), and method ('LLM model'), making it immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'translate_text' or 'speech_to_text' beyond the core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage guidance with the cost warning: 'Only use when explicitly requested by the user.' This establishes a specific context for when to use the tool. However, it doesn't mention when NOT to use it (e.g., for simple text extraction) or name alternatives among sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diarize_speechA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it's a write operation (saves output files), has cost implications (API call to Whissle), and specifies default behavior (output directory defaults to $HOME/Desktop). It doesn't mention rate limits or error handling, but covers the most critical aspects for a cost-incurring tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement, cost warning, parameter explanations, and return information. Every sentence adds value, though the parameter section is somewhat lengthy. It's appropriately sized for a tool with 5 parameters and no schema descriptions, but could be slightly more concise in the Args section.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, cost implications, file output) and the absence of annotations, the description provides comprehensive context. It covers purpose, usage warnings, parameter semantics, and return values (TextContent with transcription and file path). The presence of an output schema means it doesn't need to detail return structure, and it addresses all critical aspects for the agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant semantic value beyond the input schema, which has 0% description coverage. It explains what each parameter does (e.g., 'model_name' specifies the ASR model, 'max_speakers' limits speaker identification), provides defaults, and clarifies optionality. The only gap is that 'boosted_lm_words' and 'boosted_lm_score' could be better explained, but overall it compensates well for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('convert speech to text with speaker diarization') and resources ('audio file'), and distinguishes it from sibling tools like 'speech_to_text' by emphasizing speaker diarization. It goes beyond the tool name to explain the core functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance with a cost warning ('⚠️ COST WARNING: This tool makes an API call to Whissle which may incur costs. Only use when explicitly requested by the user'), which clearly indicates when to use (only when explicitly requested) and when to avoid (due to costs). This helps the agent choose alternatives like 'speech_to_text' for non-diarized transcription.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• speech_to_textA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden of behavioral disclosure. It successfully describes several key behaviors: the API call to Whissle with cost implications, the file saving behavior with default directory logic, and the return format (TextContent with transcription and file path). It doesn't mention error handling, rate limits, or authentication requirements, but covers the essential operational behavior well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It begins with the core purpose, then provides the critical cost warning, followed by organized parameter documentation and return information. Every sentence earns its place - the warning is essential, and the parameter explanations are necessary given the lack of schema descriptions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, API calls with costs, file operations) and the absence of annotations, the description provides comprehensive coverage. It explains the purpose, usage constraints, all parameters, and the return format. With an output schema present, it doesn't need to detail return values further. The description is complete enough for an agent to understand when and how to use this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by providing detailed parameter documentation. It explains all 5 parameters with clear semantics: what 'audio_file_path' is for, default values for 'model_name' and 'output_directory', what 'timestamps' controls, and the purpose of both 'boosted_lm_words' and 'boosted_lm_score'. The description adds significant value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Convert speech to text with a given model and save the output text file to a given directory.' This specifies both the core function (speech-to-text conversion) and a secondary action (file saving). It distinguishes from siblings like 'diarize_speech' (which focuses on speaker identification) and 'list_asr_models' (which lists available models).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context with the cost warning: '⚠️ COST WARNING: This tool makes an API call to Whissle which may incur costs. Only use when explicitly requested by the user.' This gives important guidance about when to use (only when user explicitly requests) and implies when not to use (for casual exploration due to costs). However, it doesn't explicitly compare to alternatives like 'diarize_speech' for different speech processing needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• translate_textA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does so effectively. It discloses critical behavioral traits including cost implications ('makes an API call to Whissle which may incur costs') and the return format ('TextContent with the translated text'), which goes beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections (purpose, warning, args, returns). Every sentence earns its place - the warning is crucial, parameter explanations are necessary given 0% schema coverage, and return statement is valuable. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters with 0% schema coverage and no annotations, the description provides complete context. It explains purpose, usage constraints, parameter meanings with examples, and return format. The output schema exists but the description still adds value by specifying the return type.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaningful context for all three parameters with examples ('e.g., "en" for English'), explaining what each parameter represents beyond just their names. However, it doesn't specify format constraints or valid language codes beyond examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('translate text from one language to another') and identifies the resource (text). It distinguishes from sibling tools like 'summarize_text' by focusing on translation rather than summarization or speech processing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use ('only use when explicitly requested by the user') and provides a cost warning that helps determine when NOT to use it. This gives clear alternative scenarios where other tools might be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.