Server Quality Checklist

Tool Scores

• open_incidentB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description alone must disclose behavioral traits. It only says the action makes a break visible, but does not explain mutation effects, idempotency, permission requirements, or what happens to existing incidents. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that avoids unnecessary words. It efficiently conveys the core function and purpose without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no annotations and 0% parameter description coverage, the tool requires more context than provided. The output schema exists, so return values need not be described, but the description leaves gaps in usage guidance, parameter semantics, and behavioral transparency. Sibling tools like resolve_incident suggest a lifecycle, but the description does not position itself within it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description adds no parameter-level detail. The phrases 'on an asset' and 'the break' loosely map to dataset_urn and description, but there is no explicit explanation of what each parameter means, how they relate, or the expected format for incident_type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Open an incident') and the resource ('an asset in DataHub'), with a specific purpose ('so the break is visible to humans'). This distinguishes it from siblings like resolve_incident, which handles closing incidents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the usage context (surfacing a break for human visibility), but it does not explicitly state when to use this tool versus alternatives, nor does it mention exclusions or prerequisites. No sibling distinctions are drawn.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_incidentB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that a message explaining what was actually done is required, but it does not mention side effects (e.g., changing status), permissions, reversibility, or any other behavioral traits. This is a significant gap for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, front-loaded with the primary action, and contains no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema and only two required parameters, but the description lacks usage context, side-effect disclosure, and differentiation from siblings. It is minimally adequate but incomplete for an agent to confidently select and invoke without further schema hints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It clarifies the 'message' parameter as an explanation of what was actually done, but does not describe 'incident_urn' beyond the general reference to an incident. Thus it adds partial semantic value but leaves one parameter unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Resolve' and identifies the resource 'an incident,' and indicates a required message. This clearly distinguishes from sibling tools like 'open_incident' and 'remember_incident.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for resolving incidents but provides no explicit guidance on when to use this tool versus alternatives such as 'open_incident' or 'remember_incident.' No exclusions or alternative scenarios are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• guard_columnA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden for behavioral disclosure. It reveals that this creates a persistent assertion ('permanent guarantee for everyone downstream'), which is meaningful. However, it omits important details like whether the assertion is idempotent, what happens if the assertion already exists, or any required permissions. This is a moderate disclosure but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, with the first sentence front-loading the core action. The second sentence adds motivational context but is somewhat verbose. Overall, it is concise and well-structured, though the second sentence could be trimmed without losing essential meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three required parameters, no annotations, and zero schema parameter descriptions, the description gives a clear purpose and usage cue but lacks detail on parameter semantics and potential side effects. The presence of an output schema may cover return values, but the overall description leaves gaps that could confuse an agent on how to correctly invoke the tool with all parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero description coverage for its three required parameters. The description only indirectly explains 'column' ('watching a column') but does not clarify 'dataset_urn' or 'why'. Since the schema provides no help, the description fails to compensate for these undocumented parameters, leaving their semantics ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Leave an assertion watching a column' with a specific verb and resource. It also conveys the intended effect ('so this break cannot recur silently'), which distinguishes it from sibling tools like resolve_incident or remember_incident that focus on incident lifecycle, not preventive column guards.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Call this after a fix,' providing clear timing context. It explains the purpose—'turns one repaired incident into a permanent guarantee'—which implies it is the follow-up to a resolution. It does not explicitly mention when not to use it or alternatives, but the context is sufficient given the sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_datasets_with_failure_modeA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the exhaustive scope ('every asset in the graph') and implies a read-only query, but it does not mention potential limitations, pagination, or error behavior. It adds some context beyond the bare function but not rich behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only two short sentences. The first states the purpose clearly, the second clarifies the parameter format. Every word earns its place, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter, no nested objects) and has an output schema, so return values need not be described. The description sufficiently explains the core operation and parameter format. Minor gaps exist (e.g., what constitutes an 'asset' or 'graph'), but these are likely assumed in the domain context. Overall, it is nearly complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter with zero description coverage, so the tool description compensation is critical. It explains that failure_modes are slugs and provides concrete examples ('silent-schema-change', 'late-upstream'), giving an agent clear guidance on the expected format. This directly adds value beyond the schema's bare string type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Find every asset'), the resource ('in the graph'), and the condition ('already suffered a given failure mode'). The verb is specific and the scope is unambiguous. It differentiates from sibling tools by focusing on assets by failure mode rather than incident memory or lineage recall.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance on when to use this tool vs. alternatives. The description implies it should be used to find assets affected by a failure mode, but it does not mention exclusions or name alternative tools (e.g., recall_across_lineage or guard_column). Agents are left to infer the appropriate context from the tool name and sibling list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• remember_incidentA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It discloses that this is a persistent write operation, that it is mandatory at incident close, and that it should contain a conclusion rather than a status. It does not mention overwrite behavior, permissions, or error conditions, but the provided context is substantial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with the core action front-loaded in the first sentence. Every sentence adds meaningful context about audience, timing, and content requirements. There is no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (11 parameters, 4 required) and zero schema descriptions, the description is insufficient for guiding correct invocation. It explains the tool's purpose and when to use it but does not address how to populate key fields like failure_mode, root_cause, lineage_path, or changed_field.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides no information about any of the 11 parameters, and the schema has 0% description coverage. The parameter names such as 'dataset_urn' and 'failure_mode' are self-explanatory, but the tool description adds no value in explaining required or optional parameters, leaving many ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool writes a conclusion into the graph for humans and the next agent. It is specific about the action (write), the resource (graph), and the content (tonight's conclusion), and it distinguishes the tool from sibling recall tools as the final write step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs that this is the last thing to do on any incident and that it is not optional. It also provides a clear content guideline—write a conclusion, never a status—and explains the rationale: the next agent reads this instead of starting from nothing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recall_across_lineageA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden of disclosing behavioral traits. The verb 'Read' clearly indicates a non-mutating, safe operation, which is useful. However, it does not disclose any caveats such as how missing or invalid dataset_urns are handled, rate limits, or authentication requirements. The description is adequate for a simple read tool but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly constructed sentence that front-loads the verb and resource. Every word adds value, with no redundancy or tangential information. This is an excellent example of concise, structured documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool design (one parameter, output schema present), the description is largely complete for invocation purposes. It states what the tool does and hints at the parameter's meaning. However, it omits any guidance on error behavior or edge cases (e.g., what if some urns do not exist), which would have made it fully comprehensive. The presence of an output schema reduces the need to explain return values, so a 4 is appropriate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate for parameter explanations. It clarifies that dataset_urns refers to 'several assets' and gives a lineage path example, but the relationship between 'lineage path' and the array-of-strings schema is ambiguous. It does not explicitly explain the format or expected values of the dataset_urns entries, so it only partially compensates for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Read'), resource ('Nightshift memory'), and scope ('several assets at once'), along with a concrete example ('a lineage path'). This effectively distinguishes it from sibling tools like recall_incident_memory, which focuses on incident-specific memory.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool ('for several assets at once, e.g. a lineage path'), implying it is appropriate for batch reads across a lineage. However, it does not explicitly mention alternatives or exclusions (e.g., 'use recall_incident_memory for single incidents'), so it falls short of a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recall_incident_memoryA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals the read-only nature ('Read') and the short-circuit behavior that can skip investigation, which are valuable traits. It does not cover edge cases like empty memory, but the presence of an output schema likely covers return structure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero wasted words, front-loading the purpose and then providing a clear usage directive. It is an example of efficient, high-value communication.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with an output schema, the description is fairly complete. It explains the strategic context (first call) and the purpose, but does not explicitly address how to format dataset_urn or handle missing memory. The output schema likely compensates for return-value details, so this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not explain the dataset_urn parameter beyond its name. Since the schema provides no details and the description fails to compensate, the agent is left to infer that dataset_urn identifies the dataset. This adds minimal value over the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Read' and identifies the resource ('previous nights concluded about this dataset'), making its purpose unambiguous and distinguishing it from sibling tools like recall_across_lineage. It clearly states what the tool does and the value it provides.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('Call this FIRST, before walking any lineage') and explains the strategic benefit ('skip the investigation entirely'). However, it does not provide explicit when-not-to-use guidance or mention alternative tools, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.