MCP Airlock

Tool Scores

• airlock_issue_capabilityC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Lacking annotations, the description only discloses the 'short-lived' nature (relating to ttl_seconds) but omits critical behavioral details: what authorization the lease grants, how the constraints object limits usage, side effects of issuance, or security implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single dense sentence with no filler words; information is front-loaded. However, extreme brevity becomes a liability given the complete absence of schema documentation and annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Inadequate for a security-sensitive tool with 6 parameters (including a nested constraints object). Fails to explain the capability model, what the lease authorizes, or the purpose of required fields like 'subject', creating operational risk.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description partially compensates by implying session_id, intent, and ttl_seconds via 'bound to session+intent' and 'short-lived', but leaves 'subject', 'tools', and the nested 'constraints' object completely unexplained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States the core action (Issue) and resource (capability lease) with binding context (session+intent), but uses domain jargon without explanation and fails to differentiate from sibling 'airlock_audit_tail' or explain what the lease enables.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides no guidance on when to issue capabilities versus using other tools, no security prerequisites, and no warnings about the sensitivity of granting tool access via leases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• airlock_audit_tailC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it notes events are 'recent' and 'signed', it fails to specify the time window for 'recent', result ordering, return format, pagination behavior, or the implications of 'signed' (verification requirements?). This leaves critical behavioral gaps for an audit tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at five words, immediately front-loading the verb and object. While efficient, this brevity is arguably inappropriate given the complete absence of annotations and schema descriptions, leaving the description too terse to stand alone as documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a security/audit tool handling cryptographically signed provenance data, the description is inadequate. With no output schema, no parameter descriptions, and no annotations, the description should explain what data structure is returned and what 'airlock' provenance tracks, but it provides none of this context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage for the 'limit' parameter. The description completely fails to compensate by explaining the parameter's purpose, valid ranges, or default behavior (20). The agent has no textual guidance on how to use the only available parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Read') and the specific resource ('signed provenance events'), distinguishing it from the sibling 'airlock_issue_capability' (which issues/writes) and the unrelated 'weather_hourly' and 'http_get_json' tools. However, it could better clarify what constitutes a 'provenance event' in this specific 'airlock' domain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites (e.g., specific permissions needed to read signed audit trails) or when not to use it. The agent receives no signals about appropriate usage contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• weather_hourlyA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden. It successfully indicates the geocoding behavior ('Resolve') and data source ('Open-Meteo'), but omits critical behavioral details like error handling for invalid locations, output format, units (imperial/metric), or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence with zero redundancy. Every word contributes essential information (action, scope, data source), making it appropriately sized and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple 3-parameter schema with primitive types and no output schema, the description adequately covers the core function. However, gaps remain: the 'hours' parameter is undocumented, and the absence of annotations or output schema leaves the return structure and units unspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, requiring the description to compensate. It implicitly documents the 'city' and 'state' parameters by specifying they are 'US city/state', adding geographic context not present in the raw parameter names. However, it fails to mention the 'hours' parameter or its constraints (1-168, default 24).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs ('Resolve', 'fetch') and identifies the exact resource ('hourly weather'). It clearly distinguishes the tool from siblings (airlock_audit_tail, http_get_json) by specifying the weather domain and Open-Meteo data source.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies geographic constraints by specifying 'US city/state', hinting at usage boundaries. However, it lacks explicit guidance on when to use versus alternatives (e.g., for non-US locations) or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• http_get_jsonA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It adds valuable behavioral context by specifying 'public' (implying no auth headers required) and 'JSON' (setting expectation for response parsing), but omits operational details like timeout behavior, redirect handling, or error responses for non-JSON content.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence with zero waste. It front-loads the action and precisely qualifies the target resource type and protocol without filler text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 parameters, no output schema), the description adequately covers the essential contract. The 'public' qualifier is crucial for setting correct expectations, though mentioning error handling for non-JSON responses would strengthen completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the schema fully documenting both 'url' and 'query' parameters. The description adds no additional parameter semantics beyond what the schema provides, meeting the baseline expectation for well-documented schemas.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description provides a specific verb ('Fetch'), resource type ('JSON'), and scope ('public HTTPS API endpoint'), clearly positioning this as a generic external HTTP client distinct from domain-specific siblings like weather_hourly and airlock_audit_tail.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  While it lacks explicit 'when-to-use' statements, the description implies usage through the 'public HTTPS API' scope, suggesting external/unauthenticated endpoints versus the internal/domain-specific siblings. However, it does not explicitly direct users to alternatives like weather_hourly for weather data.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.