Server Quality Checklist

Tool Scores

• heliograph_read_logA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The verb 'Read' signals a non-mutating operation, and the description explains the meaning of exit statuses. No annotations are present, so the description carries the behavioral burden; it does so reasonably well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is somewhat verbose and includes repeated emphasis ('Read all of it') and a metaphorical control comparison. It conveys useful context but could be tightened.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema is provided, so the description should clarify the return format. It mentions UTC timestamps and exit semantics, but does not specify the full output structure or error behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully describes both parameters ('name' and 'estate') with clear defaults and optionality. The description adds no further parameter detail, so baseline is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Read' and the resource 'captured log whole', making the primary purpose obvious. It does not explicitly differentiate from sibling tools like heliograph_logs, but the intent is still clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides interpretive guidance about reading the whole log and the meaning of a green exit, but does not explicitly state when to prefer this tool over alternatives like heliograph_logs or heliograph_status.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heliograph_statusA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden and does well: it explains the state enum, the meaning of an empty state, and that 'refused' is not a failure and that the reason names the flag that would permit it. This adds material beyond the schema and prevents misinterpreting a failure-like state.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no filler: the core purpose is front-loaded, and each sentence adds a distinct state or semantic clarification. It is easy to scan and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a low-complexity tool with one optional parameter fully documented and no output schema. The description conveys the result vocabulary (one of five states or empty) and explains ambiguous states. It does not describe the exact payload shape, but the state-based output is sufficiently implied.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single optional parameter 'estate' is fully documented in the schema, and the description adds no parameter-level detail. With 100% schema description coverage, score 3 is the appropriate baseline even though the description omits param specifics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly describes a status/state retrieval for a station and enumerates the possible state values, so an agent knows what this tool reports. It does not use an explicit verb like 'get' and does not explicitly distinguish itself from siblings, but its resource and state vocabulary make the function evident.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance about when to call this tool rather than a sibling such as heliograph_logs or heliograph_send, and no preconditions or exclusions are given. Usage context must be inferred from the word 'status' and the sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heliograph_doctorA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral disclosure burden. It explicitly states the operation is non-mutating ('change nothing') and read-only ('read access is not write access'), and explains why discovering failure later is costly. It does not describe the exact output or any network side effects, but for a read-only check this is strong coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with the main action and constraint front-loaded: 'Check whether the transport works from here, and change nothing.' The rationale in the second sentence earns its place by explaining when and why to run the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and no output schema, the description adequately covers purpose, safety, and timing. The main gap is that the exact return value or result format is not described, but 'check whether it works' makes the outcome reasonably inferable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents the single optional 'estate' parameter, so schema coverage is 100%. The description adds no parameter-specific meaning beyond the schema, placing it at the baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Check whether the transport works from here,' and adds a clear non-mutating constraint: 'change nothing.' This makes the tool readable as a connectivity diagnostic and distinguishes it from send-type tools, though it does not explicitly contrast with siblings like heliograph_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit trigger condition: 'Run this before a long step.' It also conveys an important exclusion: this is read access, not write access, so it should not be used as a substitute for a mutation. It does not name alternative tools, but the guidance is clear enough for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heliograph_estatesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The verb 'List' strongly implies a read-only operation, and the description states what information is returned: estates and their transports. However, with no annotations provided, the description does not explicitly disclose side-effect-free behavior, error conditions, or dependencies, leaving some behavioral transparency burden unmet.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. It front-loads the action and resource, then immediately adds the most important usage guidance. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless list tool without an output schema, the description covers what the tool does, what it returns at a high level, and when to call it. It could be more explicit about the exact output shape or failure behavior, but the core contextual needs for invoking this tool are met.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters and 100% schema description coverage, so there is no parameter meaning for the description to add. The baseline of 4 applies because the tool has no parameters and the description does not need to compensate for schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and names a distinct resource ('configured estates') plus the additional detail of transport each uses. The phrase 'Call this first when you do not know which estate to act on' further distinguishes it from the sibling tools, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use the tool: 'Call this first when you do not know which estate to act on.' It does not name specific sibling alternatives or exclusion conditions, so it falls short of a 5, but it provides clear and actionable usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heliograph_gapsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It reveals key behavior: returns intervals sorted longest first, attributes each gap to the line before it, and reports an error when timestamps are all identical. It also explains the diagnostic rationale. It stops short of explicitly stating read-only semantics or exact output shape, but covers the most important behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, each carrying distinct value: concise purpose statement, return behavior, usage timing with rationale, and edge-case failure mode. No filler or repetition. The most decision-relevant information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description covers the essential contract: what is returned, ordering, attribution, and error condition. It does not specify the exact structured return format, but for an analysis tool of this complexity the description is nearly complete. An agent can correctly decide when to call it and interpret its result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the 3 parameters (name, estate, min_seconds) are already fully documented in the schema. The description adds semantic context around the log/timestamp concept but does not meaningfully extend parameter understanding beyond the schema. Baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise verb+resource statement ('Where a run stalled. Returns the intervals in the log's timestamp column') and clearly distinguishes itself from log-reading by positioning itself as the pre-reading analysis: 'Do this before reading a long log.' This makes the tool's identity and scope immediately obvious and separate from siblings like heliograph_read_log.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Do this before reading a long log: a hang and slow progress are indistinguishable without it.' It also explains when the tool cannot answer the question (buffered capture with identical timestamps). It does not explicitly name alternative tools or state when not to use it, but the timing guidance is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heliograph_logsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It meaningfully discloses the sort order ('newest first') and the naming convention linking log names to request ids returned by heliograph_send. It does not mention output shape or limitations, but the core behaviors needed for correct use are present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two compact sentences carry the essential information: what the tool lists, ordering, the naming convention, and how to route to a sibling. Every sentence earns its place with no redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description tells the agent what will be returned (log names), the ordering, and how names relate to prior send requests, while pointing to the read tool for content. Since there is no output schema, this is adequate for a simple listing tool, though a note about pagination or absence of logs would make it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the single 'estate' parameter is already fully explained by the schema: 'Which estate. Optional when only one is configured.' The description adds no parameter-level meaning, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'List the captured logs, newest first.' It also clearly distinguishes this tool from its siblings by noting that it is for finding a log while heliograph_read_log is for reading one.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit routing guidance: 'Use this to find a log; use heliograph_read_log to read one.' This tells the agent exactly when to select this tool versus its most relevant sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• heliograph_sendA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the full burden. It discloses several non-obvious behaviors: the call is fire-and-forget, the station has discretion whether to execute the step, mode-less steps are refused, and state-changing steps require CONFIRM=yes plus a station started with --allow-actions. The mong these (publish success vs execution success) is a genuinely important behavioral trait an agent would otherwise discover only after a failed run.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, zero filler. The core action is front-loaded, then the async consequence, then the execution conditions. Every sentence earns its place and the most decision-relevant constraints appear last without being buried.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations and no output schema, the description is thorough: it explains the non-blocking contract, how to obtain the result (poll status, read log), and refusal conditions. The remaining gap is that it never states what the immediate response contains (e.g., a run identifier needed to correlate with heliograph_status), which an agent needs since no output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3; all four parameters (env, note, step, estate) are already documented. The description adds semantic value by explaining what the station does with the step parameter — refusal if no mode is declared, and extra authorization requirements for state-changing steps — which clarifies meaningfully how step and env interact.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence names a specific verb and resource ('Publish a step for the station to run') plus the key behavioral trait (returns immediately). It explicitly differentiates from siblings by saying it does NOT wait and directing the reader to poll heliograph_status and read the log, which distinguishes it from at least two sibling tools without needing to open their schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit post-call workflow: do not wait, poll heliograph_status until terminal, then read the log. It also states the conditions under which this tool is effective (step must declare a mode; state-changing steps need CONFIRM=yes and --allow-actions), so the agent knows when the call will have no effect. Vivid routing to named siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.