Server Quality Checklist

Tool Scores

• memory_statsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description only mentions it returns a summary. It does not disclose whether the operation is read-only, requires authentication, or has any side effects. For a tool with no annotations, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences that are front-loaded with the action. No extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the output schema exists to describe return values, the description lacks details about the optional parameter and does not fully cover the tool's behavior. It is adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one optional parameter 'project' with 0% description coverage. The description does not explain the parameter's purpose or usage, failing to add meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies 'query AIOS knowledge base statistics' and lists specific metrics (chunk count, type distribution, recent activity). It clearly distinguishes from sibling tools like pin/unpin memory.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives, but the purpose is clear. No exclusions or usage conditions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpin_memoryA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It lacks details on side effects, error conditions, idempotency, or prerequisites (e.g., what happens if the chunk is already unpinned). The OS analogy helps but is insufficient for complete transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear purpose sentence, an illustrative analogy, and a structured Args/Returns section. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity and the presence of an output schema (not fully visible), the description covers basic usage but omits error handling and prerequisites. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context beyond the schema: it explains chunk_id as 'chunk ID to unlock' and project as 'project ID (default auto-resolves current directory)'. Schema coverage is 0%, so the description compensates well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to unpin a chunk in the current project, using an OS analogy (munlock) that reinforces the concept. It distinguishes itself from sibling tools like pin_memory by indicating it is the inverse operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage as the inverse of pin_memory, but does not explicitly state when to use it over alternatives or provide exclusions. The context of sibling tools is available but not referenced in the description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_pinnedA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the tool returns a formatted list and describes parameters, but does not disclose side effects, authentication needs, or error conditions. The read-only nature is implied by the name but not stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two short paragraphs and a clear 'Args/Returns' section. The OS analogy adds extra context but is not essential, slightly reducing conciseness. Still, it remains efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 optional params, output schema exists), the description covers core functionality, parameters, and return format. It lacks details on project resolution behavior or limitations, but is adequate for most use cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully explains both parameters: pin_type (filter options with defaults) and project (auto-resolve behavior). It adds clarity on allowed values and default behavior, which the schema lacks entirely.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: listing all pinned chunks in the current project. The verb '列出' (list) and resource 'pinned chunks' are specific and unambiguous. Although it doesn't explicitly distinguish from sibling tools, the unique resource and operation make the purpose clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (viewing pinned chunks in a project) but does not explicitly state when to use this tool versus alternatives like memory_lookup or pin_memory. No 'when not to use' guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• memory_lookupA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It describes the tool as an active retrieval operation with 'reasoning interruption / demand paging' context, implying a read-only action. However, it does not mention any side effects, permission requirements, or rate limits. The description is adequate but could be more detailed about behavioral traits such as whether results are cached or if there are size limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured: a one-line introduction, a bulleted list of usage examples, and clearly labeled Args/Returns sections. Every sentence adds value. It is concise without being overly brief; the examples make it practical. Minor redundancy: the examples could be more concise, but overall it is well-organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has an output schema (not shown but noted), the description does not need to detail return values, though it provides a brief summary. It covers the essential context: when to use, what parameters to provide, and example queries. The description is complete enough for an AI agent to understand the tool's role and usage, though it could mention that it is a read-only operation explicitly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the tool description must compensate. It explains each parameter: query (natural language or keywords), top_k (default 5), chunk_types (optional filter with examples), and project (optional, auto-inferred). This adds significant meaning beyond the schema, which only provides names and types. The examples for chunk_types are particularly helpful. A small improvement would be to clarify the format of the query string.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: active retrieval of relevant memories from the AIOS knowledge base. It provides concrete examples of queries (e.g., decisions, design constraints, performance data), making the specific resource and action unambiguous. The function is well-distinguished from siblings (list_pinned, memory_stats, pin_memory, unpin_memory) which handle different memory operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use the tool: 'when you need knowledge but are unsure' during reasoning. It gives example scenarios that guide appropriate usage. However, it lacks explicit guidance on when NOT to use it or alternatives, though siblings are distinct. A clear context is provided, but no exclusions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pin_memoryA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses all behavioral details: it prevents eviction by specific mechanisms (stale reclaim, DAMON DEAD, kswapd ZONE_MIN) and distinguishes hard (all) vs soft (except kswapd under extreme pressure). It also notes the return type.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with sections, OS analogy, and bullet points. It is informative but slightly long; however, every part adds value, so it remains efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (nuanced pinning behavior) and lack of annotations, the description covers purpose, usage, behavior, parameters, and return. It references related tools (memory_lookup) and includes an output schema, though the description already mentions a result description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description fully explains all 3 parameters: chunk_id (source from memory_lookup), pin_type (values and meaning), and project (default and auto-detection). This significantly adds value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool pins a chunk to prevent automatic eviction, with a helpful OS analogy (mlock). It distinguishes from siblings by focusing on pinning vs. listing, lookup, stats, and unpinning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains when to use 'hard' vs 'soft' pinning with concrete examples (design constraints vs. important evidence) and mentions where to get the chunk_id (memory_lookup). It does not explicitly state when not to use it, but the trade-offs are implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.