Server Quality Checklist

Tool Scores

• stop_monitorA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description details what happens: kills ffmpeg process, removes temp files, clears buffer. This is transparent for a cleanup tool. However, it does not mention idempotency or error handling for invalid stream_id.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: two sentences. The first states the purpose, the second lists specific actions. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, return values are not needed. The description covers the main behavior and cleanup. Minor gap: no mention of edge cases like already stopped stream.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the tool description does not explain the 'stream_id' parameter's purpose or format. The parameter is left completely undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Stop monitoring a live stream and clean up all resources.' It distinguishes from siblings like 'monitor_live_stream' which starts monitoring, and others which analyze or retrieve transcripts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. It implies it should be used after 'monitor_live_stream', but there is no explicit context or prerequisites mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_rolling_transcriptA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool 'retrieves' and 'returns' text, implying a read operation, but does not disclose prerequisites (e.g., stream must be monitored), side effects, rate limits, or error conditions. The mention of 'monitored stream' hints at a prerequisite but is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the action ('Retrieve'), and contains no redundant information. Every word contributes to understanding the tool's purpose and typical use.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema (not shown), the description does not need to explain return values. However, it omits important context such as error conditions, prerequisites (e.g., stream must be monitored via monitor_live_stream), and behavior when no transcript is available. This leaves gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only minutes_back has a description). The tool description does not add any parameter details beyond what the schema provides. For example, stream_id is not described in the tool description, and minutes_back's description in schema is minimal. The description adds no extra semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves rolling transcript from a monitored stream and returns concatenated text from the last N minutes, with a specific use case for LLM summarisation/sentiment. This distinguishes it from siblings like analyze_speaker_stress, monitor_live_stream, and stop_monitor.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage during an earnings call but provides no explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives. The context from sibling tools helps, but the description itself lacks explicit usage guidelines.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_speaker_stressA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses the analysis process (pitch extraction, jitter, hesitation detection) and the meaning of the composite score. However, it does not mention side effects, data persistence, or permissions, but for a read-only analysis this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise, with a clear summary followed by feature list and score interpretation. The final speculative paragraph is slightly verbose but does not detract significantly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists, so the description does not need to detail return format. It provides score ranges and feature explanation. Prerequisites (e.g., active stream) and error conditions are not mentioned, but overall sufficient for typical usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50% (only time_window_seconds has a description). The description adds context about the time window and pause threshold but does not explain stream_id beyond its necessity. Overall, moderate value added beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes speaker vocal stress over a recent time window, details extracted features (F0 contour, jitter, hesitation patterns), and provides a composite stress score. This differentiates it from sibling tools which handle transcription or monitoring.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for analyzing stress in a recent window of a live stream but lacks explicit guidance on when to use this tool versus siblings or prerequisites like stream activity. Score ranges provide context but no direct usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• monitor_live_streamA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and explains key behaviors: extracts audio, chunks into 15-second segments, continuous background transcription, and VAD behavior control. It could mention potential resource usage or authorization, but the provided details are substantial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is approximately 100 words, front-loaded with purpose, then usage guidance, parameter advice, and return value. Every sentence is informative without redundancy, achieving high conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters, clear purpose, sibling tools suggesting a workflow, and an output schema (not shown), the description covers the initiation process, key parameter behavior, and return value (stream_id). It is complete for an agent to understand and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50% (disable_vad has description, url does not). The description adds value by clarifying url as a live webcast link and elaborating on disable_vad beyond the schema, noting its use for low-quality audio where VAD drops muddy speech. This offsets the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Start monitoring' and the resource 'live financial webcast' with specific purpose 'for transcription and stress analysis'. It also provides examples of acceptable URLs, distinguishing it from sibling tools that handle different stages like analysis or retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use the tool (provide a URL to a live webcast) and a specific scenario (low-quality audio) for setting disable_vad. It implies use for initiation, contrasting with siblings for subsequent operations, but lacks explicit 'when not to use' or alternative tool mentions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.