Server Quality Checklist

Tool Scores

• check_orgA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the disclosure burden. It conveys the source ('from lastseen.dev'), scope, and non-advisory nature ('Lookup, not advice'), but does not mention read-only behavior, authentication needs, rate limits, or response format. This is adequate for a simple lookup but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no filler. Key scoping information is front-loaded, and every clause adds value: the resource, the source, the scope, and the lookup/advice distinction.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter lookup tool, the description plus full schema coverage is sufficient for correct invocation. It identifies the input, source, and scope. The lack of an output schema means return values are not detailed, but the description does not need to explain them per rubric. Minor missing context includes possible error conditions or assumptions about org-slug validity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the single required parameter 'name' is already fully documented as a GitHub owner/org slug. The description adds only organizational scope ('Org-level only') and does not provide additional parameter semantics, matching the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific resource ('Dated survival states across the components referenced under an ORG') and a clear action ('Lookup'). It also scopes the tool with 'Org-level only', which helps differentiate it from sibling tools like check_entity and check_manifest, though it does not name them explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Org-level only' gives clear context about when this tool applies, and 'Lookup, not advice' provides an exclusion that prevents misuse. However, it does not explicitly compare against sibling tools such as check_entity or survival_profile, leaving some routing to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_entityA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the return format ('recorded state series with dates and sources'), the non-advice nature, and the honest failure behavior for unknown/fabricated subjects ('clean not_observed, never a fabricated result'). It stops short of stating rate limits or data freshness, but for a read-only lookup these are minor omissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, all informative, with the primary subject front-loaded in the first sentence. No filler or redundancy. Each sentence adds a distinct layer: subject, return content, and failure semantics.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with only two parameters and no output schema, the description covers the essential lookup behavior, return value nature, and not-found semantics. It does not detail what a 'survival observation' looks like structurally, but the phrase 'state series with dates and sources' gives enough shape. This is well complete for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both slug and live documented in the input schema. The description adds marginal semantic value by clarifying the subject scope and the return concept, but it does not go beyond what the schema already provides. Baseline 3 is appropriate because the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Dated survival observation(s) for ONE owner/repo' from lastseen.dev. It clearly distinguishes itself from siblings by scoping to a single owner/repo and explicitly calling out its lookup nature. The example 'actions/checkout' and the return of a 'recorded state series' make the function unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: it is for one specific repo ('ONE owner/repo') and is a 'Lookup, not advice.' It also warns about unknown/fabricated subjects returning not_observed, which helps set expectations. It does not explicitly name alternatives like check_org or check_manifest, but the singular repo scope implies when this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_manifestA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does meaningful work: it states the source (lastseen.dev), defines handling for components not held ('reported as not_observed'), and explicitly frames the operation as lookup rather than advice. This is valuable behavioral disclosure beyond what the schema provides.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded, and contains no waste. Every clause adds useful information: input format, output behavior, edge-case reporting, and the 'not advice' caveat. It is a model of concise tool documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter, no-output-schema tool this is largely complete: it tells the agent what input to pass, how it will be interpreted, what results look like, and how missing components surface. It doesn't detail the exact response shape or rate limits, but no output schema exists and those are secondary for a simple lookup.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only says 'raw manifest text to parse', while the description enriches it by specifying the two accepted formats (GitHub Actions uses refs, or owner/repo lines) and the resulting interpretation per component. Since schema coverage is 100% but shallow, the description adds real semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Parse'), the resource (a pasted manifest), and the return value (dated survival state per component). It also enumerates accepted input formats (GitHub Actions uses refs or owner/repo lines), distinguishing it from siblings like check_entity and check_org. 'Lookup, not advice' further clarifies its scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly indicates this tool is for parsing manifest-size input rather than a single entity or org, which aligns with the sibling context. It doesn't explicitly name alternatives or say when not to use it, but the input-type signal is strong and the closing 'Lookup, not advice' sets appropriate expectations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• survival_profileA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses a key behavioral trait—that this tool will not perform the expected ranking and instead returns an 'honest not_supported result'—and gives pointers to alternatives. It doesn't detail the exact shape of the not_supported result, but the warning is significant and trustworthy.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The main promise is front-loaded, and the limitation plus escape hatch follow immediately. Slightly more detail on the alternatives could push it to 5, but it is already tight and useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a tool that is essentially a guard against a known limitation: the agent knows when it works, when it won't, and what to call instead. No output schema is needed because the primary outcome is the not_supported signal, which is fully explained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters. The description reinforces the purpose ('category ranking') and the intended resource, but adds no extra syntax or format details beyond what the schema provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('ranked survival profile') plus a precise resource (CATEGORY) and immediately distinguishes itself from sibling lookups. It also tells the agent what it is NOT (free HTTP API category ranking) and what it returns instead (honest not_supported result with pointers to alternates).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says when category ranking is unavailable (via free HTTP API) and where to go instead (per-entity/org/manifest lookups and CC-BY dumps). This is exactly the kind of conditional routing an agent needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.