Server Quality Checklist

Tool Scores

• homescan_deviceC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While 'Get detailed information' implies a read-only operation, it doesn't specify what 'detailed information' includes, whether there are rate limits, authentication requirements, error conditions, or what happens if the IP isn't found. For a lookup tool with zero annotation coverage, this leaves significant behavioral questions unanswered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point with zero wasted words. It's appropriately sized for a simple lookup tool and front-loads the essential information about what the tool does.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete for effective tool selection. While the purpose is clear, it doesn't explain what 'detailed information' will be returned, how results are formatted, or what distinguishes this from similar sibling tools. For a tool in a family of related scanning utilities, more context about its specific role would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100% with the single 'ip' parameter well-documented in the schema itself ('IP address of the device to look up'). The description adds no additional parameter semantics beyond what the schema already provides, so it meets the baseline for high schema coverage without adding value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get detailed information') and target resource ('about a specific device by IP address'), making the purpose immediately understandable. However, it doesn't differentiate this tool from sibling tools like 'homescan_mac_lookup' or 'homescan_discover', which likely serve related but distinct purposes in the homescan system.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'homescan_mac_lookup' (presumably for MAC address lookups) and 'homescan_discover' (likely for device discovery), there's no indication of when IP-based lookup is preferred or what prerequisites might be needed for successful operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• homescan_diffB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions comparing scans and showing devices, but lacks details on what 'current' and 'last' scans refer to (e.g., stored data, real-time scans), whether this is a read-only operation, potential side effects, or output format. For a tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core functionality. It uses clear language without redundancy, making it easy to parse quickly. Every word contributes directly to understanding the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of comparing network scans and the lack of annotations and output schema, the description is incomplete. It doesn't explain how scans are stored or retrieved, what 'new/removed devices' means in practice, or the format of the results. For a tool that likely involves data analysis and comparison, more context is needed to guide effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameters need documentation. The description doesn't add parameter details, which is acceptable given the schema's completeness. A baseline of 4 is appropriate as it doesn't need to compensate for any gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Compare current network scan to the last scan and show new/removed devices.' It specifies the verb 'compare' and the resource 'network scan' with a clear outcome. However, it doesn't explicitly differentiate from sibling tools like 'homescan_report' or 'homescan_flagged', which might also involve scan analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., requiring a previous scan), exclusions, or comparisons to siblings like 'homescan_report' for broader analysis or 'homescan_discover' for initial scanning. Usage is implied but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• homescan_mac_lookupB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool performs a lookup, implying a read-only operation, but does not disclose any behavioral traits such as rate limits, authentication needs, error handling, or what happens if the MAC address is invalid. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that efficiently conveys the tool's purpose without any wasted words. It is appropriately sized and front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose but lacks details on usage, behavioral context, and output, which are needed for a more complete understanding. The schema handles parameter documentation, but overall completeness is limited.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the 'mac' parameter fully documented in the schema. The description adds no additional meaning beyond what the schema provides (e.g., it does not explain the lookup process or output format). Since the schema does the heavy lifting, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('look up') and resource ('manufacturer of a device'), and it specifies the input ('from its MAC address'). However, it does not explicitly differentiate this tool from its siblings (e.g., homescan_device, homescan_discover), which might also involve device-related operations, so it falls short of a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any context, prerequisites, or exclusions, nor does it reference sibling tools for comparison. This lack of usage instructions leaves the agent without direction on appropriate scenarios for invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• homescan_reportB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool generates a report but doesn't describe what triggers the generation, whether it's read-only or has side effects, performance characteristics, or error handling. This leaves significant gaps in understanding how the tool behaves beyond its basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the key action ('Generate a comprehensive network discovery report') and specifies format and content without unnecessary details. It could be slightly improved by structuring it into bullet points for the content elements, but it's already very concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and no output schema, the description adequately covers the basic purpose and output format. However, it lacks details on behavioral aspects (e.g., whether it's a read operation, if it triggers network scans, or how it interacts with sibling tools), which are important for a tool in a network discovery context with multiple related tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description appropriately doesn't discuss parameters, focusing instead on the tool's output and purpose. This meets the baseline for a tool with no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Generate') and resource ('network discovery report'), specifying the output format ('markdown') and content scope ('device summary, categories, and security flags'). However, it doesn't explicitly differentiate this report generation from sibling tools like homescan_export or homescan_flagged, which may also involve reporting functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, timing, or compare it to siblings like homescan_export (which might export data) or homescan_flagged (which might focus on security flags), leaving the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• homescan_flaggedA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the tool lists flagged devices but does not disclose behavioral traits such as whether this is a read-only operation, requires authentication, has rate limits, or what the output format looks like. It adds some context on the types of flags but lacks critical operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the purpose with specific details (e.g., 'unknown manufacturers, Chinese IoT devices, randomized MACs'), with no wasted words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters, no annotations, and no output schema, the description provides a clear purpose but lacks completeness in behavioral context (e.g., output format, safety profile). It is adequate for a simple list tool but could benefit from additional operational details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description does not add parameter semantics, but this is acceptable given the lack of parameters, warranting a baseline score above 3 for not omitting necessary information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'devices flagged for security review', with specific criteria (unknown manufacturers, Chinese IoT devices, randomized MACs) that distinguish it from sibling tools like 'homescan_device' or 'homescan_discover'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying the types of flagged devices, but it does not explicitly state when to use this tool versus alternatives like 'homescan_report' or 'homescan_export', nor does it provide exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• homescan_exportA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that the tool returns 'an array of files that can be saved to your vault,' which adds useful context about output behavior. However, it lacks details on permissions needed, rate limits, whether it's idempotent, or error handling, leaving gaps in behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and followed by output details. Every word earns its place with no redundancy or fluff, making it highly efficient and easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (export with format options), no annotations, and no output schema, the description does well by stating the purpose and output format. However, it could improve by mentioning dependencies (e.g., requires prior discovery) or example use cases. It's mostly complete but has minor gaps in contextual guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the 'format' parameter with enum values and default. The description doesn't add any parameter-specific details beyond what the schema provides, such as explaining the implications of choosing markdown vs. json formats. Baseline 3 is appropriate as the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Export discovered devices') and target resource ('to Obsidian inventory format'), distinguishing it from siblings like homescan_device (likely device details) or homescan_report (likely summary reports). It explicitly mentions the output format conversion, which is unique among the listed tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you want to export devices to Obsidian format, but it doesn't explicitly state when to use this vs. alternatives like homescan_report or homescan_diff. No guidance on prerequisites (e.g., must run discovery first) or exclusions is provided, leaving usage context somewhat inferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• homescan_discoverA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the action (discover devices via ARP table) and return data, but does not cover aspects like performance (e.g., speed, network impact), error handling, or prerequisites (e.g., network permissions). It adds basic context but lacks depth for a tool with potential network interactions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, consisting of two sentences that efficiently convey the tool's purpose and return data. Every sentence adds value without redundancy, making it easy to understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (network discovery with one parameter) and no annotations or output schema, the description is adequate but incomplete. It covers the core functionality and return data but lacks details on behavioral traits (e.g., network impact, error cases) and does not explain the output format beyond a high-level list, which could be insufficient for an agent to handle results effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema fully documents the single parameter 'refresh'. The description does not add parameter-specific details beyond what the schema provides, but with only one optional parameter, the baseline is high. It implicitly supports the parameter by mentioning the ARP cache and ping sweep in the context of discovery.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('discover all devices') and resources ('local network using ARP table'), and distinguishes it from siblings by specifying the discovery method and return data (IP, MAC, manufacturer). It goes beyond a tautology of the name 'homescan_discover' by explaining the action and scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for network device discovery but does not explicitly state when to use this tool versus alternatives like 'homescan_diff' or 'homescan_flagged'. It provides some context (local network, ARP-based) but lacks explicit guidance on exclusions or comparisons with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.