Server Quality Checklist

Tool Scores

• commit_memA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behaviors: upsert semantics, returning an opaque version for optimistic concurrency, and idempotent behavior via `idempotency_key`. These are important beyond what annotations would provide. It doesn't mention failure modes or limitations, but the disclosed traits are significant and clearly described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, with the main purpose stated first. It uses three short paragraphs that each add value: purpose, upsert/version behavior, and idempotency. It is not overly verbose, though it could be slightly tighter by merging related thoughts. Overall, it is well-structured and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (11 parameters, 0% schema coverage, no annotations), the description is incomplete. It covers the core upsert and idempotency behavior but leaves many parameters unexplained, which could lead an agent to misuse the tool. The presence of an output schema helps with return values, but the input side is under-documented. A more comprehensive description should at least summarize the role of key parameters like `memory_type`, `priority`, and `ttl`.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It only explains `idempotency_key` in detail; the other 10 parameters (e.g., `key`, `value`, `ttl`, `priority`, `metadata`, `memory_type`) are not elaborated. The description does not clarify what `value` should be, how `memory_type` affects behavior, or the meaning of fields like `canonical` and `importance`. This is a significant gap for a tool with 11 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Write a memory entry to hosted Bilinc Cloud.' It further explains the upsert behavior ('Creates the entry if it is new and revises it if the key already exists'), which distinguishes it from siblings like `revise` and `recall`. This is a specific verb+resource with differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives context by explaining that the returned version can be passed to `revise` or `forget`, implying when those tools are appropriate. It also explains idempotency usage for retries. However, it does not explicitly state when NOT to use this tool or name alternatives for read/delete scenarios, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recallA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on the burden of explaining behavior. It discloses that higher profiles 'do more reflection and return more provenance' and are 'gated by the workspace plan,' which is valuable context. It also mentions cost implications. However, it does not explicitly state whether recall is read-only or describe other side effects, leaving some ambiguity for a no-annotation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose. Each sentence earns its place: the first states the core action, and the second explains the profile parameter's nuances and its relationship to the status tool. There is no redundant fluff or repetition of schema information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, one required), the description covers only the retrieval purpose and profile specifics. The output schema exists and can explain return values, but the description does not clarify other parameters or edge cases. It is adequate for a basic retrieval tool but feels incomplete for an agent needing to use memory_types, query_timestamp, or explain effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter meaning. It only explains the 'profile' parameter in detail, covering just one of six parameters. The description does not address query, limit, explain, memory_types, or query_timestamp, leaving the agent without sufficient guidance for these parameters despite their self-explanatory names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Retrieve memories from hosted Bilinc Cloud.' It clearly distinguishes from siblings like commit_mem, revise, and forget by focusing on retrieval. The additional note that 'Smart retrieval is this argument, not a separate tool' further disambiguates from potential alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides practical guidance: 'call `status` to see which profiles this key may use' indicates a prerequisite for using higher profiles. It also clarifies that profile-based retrieval is not a separate tool. However, it does not explicitly state when to use recall vs. other memory operations or list exclusions, so it misses some explicit alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• snapshotA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description discloses key behavior: list is free, ordering is newest-first, and neither action returns checkpoint contents. This adds meaningful context beyond the schema, though it does not cover side effects of creation or permission requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences cover purpose, usage advice, and a behavioral caveat with no redundancy. Information is front-loaded and the description is appropriately sized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema and a moderate parameter count; the description covers the main workflows. It lacks detail on parameter meanings and restore mechanics, but overall is sufficient for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains the action parameter values (create/list) but does not clarify label, limit, metadata, or idempotency_key. Schema coverage is 0%, so these parameters remain ambiguous despite self-explanatory titles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool creates or lists project checkpoints, with specific action values. Differentiates from sibling tools by focusing on checkpoint capture/list rather than memory/commit/rollback operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises using create before risky work, and notes list is free and returns newest first. Does not name alternative tools but provides context for when each action is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reviseA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the disclosure burden. It reveals important behaviors: never creates, fails with memory_not_found, supports expected_version for concurrency (version_conflict). It does not mention auth or reversibility, but for a memory tool these are less critical. Slightly incomplete but strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, each adding substantive value: purpose, alternative differentiation, failure mode, concurrency behavior. No filler or repetition, well front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters and zero schema descriptions, the description is too thin. It covers core use cases and error scenarios, but omits guidance on the optional parameters and does not describe the response shape (though an output schema exists). It is minimally adequate but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It only explains `expected_version` and implies `key` must exist, but does not clarify the meaning or usage of `value`, `reason`, `strategy`, `importance`, or `idempotency_key`. This leaves most parameters under-specified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Deliberately replace an existing memory.' It clearly distinguishes the tool from siblings by stating it is for corrections/superseding rather than creating, which aligns with the sibling context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use this instead of `commit_mem` when correcting or superseding existing knowledge, and notes that it never creates (fails with memory_not_found). This provides clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diffA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the operation is read-only and free, values are redacted unless include_values is set, and overly large diffs are refused rather than silently truncated. This gives the agent important behavioral expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with the main purpose in the first line and additional key behaviors following. Every sentence adds value, and it is not padded with fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers essential behavioral context: read-only, free, redaction behavior, and refusal of large diffs. The output schema exists, so return format need not be detailed. However, it does not mention the limit parameter or edge cases, which would make it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains the semantics of to_snapshot_id (optional, defaults to current state) and include_values (controls value redaction), but does not explain limit or from_snapshot_id in detail. This partial coverage earns a 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the primary action: 'Compare a checkpoint against another checkpoint or current state.' This uses a specific verb ('Compare') and resource ('checkpoint'), and the mention of 'current state' distinguishes it from sibling tools like snapshot or rollback.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context for when to use the tool: to compare checkpoints or see changes since a checkpoint. It also explains the key behavior of leaving to_snapshot_id empty. However, it does not explicitly mention alternatives or when not to use it, which keeps it from a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• statusA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that the tool is 'Read-only and never billed' and that 'Secrets are never returned,' which are important safety traits. It does not elaborate on caveats like rate limits, but the disclosed traits are strong for a status tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences and every sentence earns its place: the first states the purpose, the second conveys safety and usage, and the third reassures about secrecy. It is front-loaded with the most important information and contains no redundant words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters, an output schema is present, and the tool is simple (status report), the description fully covers what an agent needs to know. It states what the tool returns (workspace, plan, capabilities), when to use it, and that it is safe and read-only. The existence of the output schema eliminates the need to describe return formats.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema provides no parameter semantics. Per the rubric, a baseline of 4 is appropriate for 0-parameter tools. The description adds no parameter information because there is nothing to add.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Report the authenticated Bilinc Cloud workspace, plan, and capabilities,' which uses a specific verb and clearly identifies the resource and scope. This distinguishes it from sibling lifecycle tools like commit_mem and forget, which perform mutations. The purpose is unambiguous and immediately understood.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance: 'Use this to discover which lifecycle operations and recall profiles the current API key may use before attempting them.' This makes the intended use case clear. It does not explicitly exclude other uses or name alternatives, but the context is sufficient for an agent to know when to call this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• forgetA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It clearly discloses the destructive nature, that the entry stops influencing future recall, that `reason` goes to the audit trail, and that the deleted value is never returned. This is thorough behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured, with a bold DESTRUCTIVE warning and short paragraphs. Every sentence earns its place, front-loading the purpose and then adding necessary details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers essential aspects: destructive behavior, audit trail, and the alternative tool. An output schema exists, so return values need not be explained. It leaves out edge-case behavior (e.g., key not found) but overall is complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It adds meaning for `reason` (required, audit trail) and implicitly clarifies `key` (the memory to remove). However, it says nothing about `idempotency_key` or `expected_version` beyond their schema defaults. Partial compensation, adequate but with gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Remove a memory from active recall' with a specific verb and resource. It also distinguishes itself from the sibling tool `revise` by explicitly saying 'To correct a memory rather than remove it, use revise.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'Use this for state that is genuinely obsolete.' It also names an alternative (`revise`) and explains when not to use it (when correcting rather than removing), satisfying the when/when-not/alternatives criterion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rollbackA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the burden of behavioral disclosure. It explicitly warns 'DESTRUCTIVE in execute mode,' explains that preview changes nothing, requires a confirmation token for execute, permanently discards memories, and mentions the failure mode `state_changed_since_preview` and audit trail writing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear summary, numbered steps, and a note on failure handling. It is somewhat long but each sentence adds necessary value for a destructive two-stage tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, no annotations, output schema present), the description is complete. It covers the full workflow, danger, token requirement, failure condition, and audit trail without needing to explain return values due to the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains the semantics of mode, confirmation_token, and reason, and implies snapshot_id as the checkpoint identifier. However, it does not explicitly describe idempotency_key, though it is optional. This is a minor gap given the strong coverage of the critical parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Restore the project to a checkpoint.' This is a specific verb+resource combination that distinguishes it from siblings like snapshot (creating checkpoints) and diff (comparing states).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear procedural context: two stages must run in order (preview then execute), and it explains the requirements for each mode. It doesn't explicitly compare to alternatives or state when not to use the tool, but the workflow guidance is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.