Server Quality Checklist

Tool Scores

• fetch_job_postingB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, and the description fails to disclose behavioral traits such as read-only nature, error handling, supported job boards, or authentication requirements. The minimal text does not compensate for missing annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single 10-word sentence with no wasted words, but could include brief additional context without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and an output schema, the description covers the basic purpose but misses behavioral details like error responses and supported boards. The output schema helps define return values, but the description remains incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds meaning by stating the URL must be from a supported job board, but does not specify format, examples, or which boards are supported, providing minimal added value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fetch and parse a job posting from a supported job board URL', using specific verb and resource. It distinguishes from sibling tools like analyze_job, which implies a different operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when you have a URL to a job posting, but does not provide explicit guidance on when to use vs alternatives like analyze_job, nor states prerequisites or limitations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_visa_sponsorshipA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses the data source (USCIS Employer Hub) and the importance of name accuracy, but does not mention read-only nature, rate limits, or authentication requirements, which is a gap given no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no redundant information; the key point about the company name format is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers the main usage and a common pitfall; output schema exists so return values need not be described. Could mention prerequisites like a USCIS account, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds significant meaning beyond the schema by instructing to use the full employer name and warning about abbreviations, compensating for the 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool checks H-1B visa sponsorship history via USCIS Employer Hub, which is distinct from sibling tools focused on jobs and profiles.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear guidance to pass the full employer name exactly as in the job posting and warns about abbreviations, though no explicit when-not to use is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• setup_profileA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the persistence path (~/.config/runway-mcp/profile.json), the failure condition on existing profile, and the return type (ProfileSetupResult). Does not contradict any annotations (none provided). Could add more detail on idempotency or permissions, but current detail is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is well-structured with a clear purpose statement, prerequisites, behavior notes, and parameter/return sections. It is slightly verbose but not wasteful. Front-loaded with the core verb.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and presence of output schema, the description explains key aspects: prerequisite, failure mode, file path, and return behavior. It adequately covers the tool's usage for an AI agent to decide and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter (profile). The description does not add meaning beyond the input schema, which already includes detailed descriptions for each field and sub-object. Baseline 3 is appropriate since schema coverage is effectively high due to embedded descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool's purpose: 'Save a structured profile extracted from the user's CV.' Provides specific verb (save) and resource (profile), and distinguishes from sibling tools by noting it fails if profile already exists, pointing to update_profile as the alternative.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to read the CV before calling, and explains the failure condition and alternative tool: 'Fails if a profile already exists — use update_profile to replace it.' This provides clear context on when and when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_jobA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes key behaviors: verifies profile existence, fetches job, checks visa, returns error envelope on failures, never raises exceptions. With no annotations, this is sufficient disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with numbered steps and return list. Somewhat lengthy but efficient; front-loaded with main purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a single-parameter tool: covers flow, error handling, and return fields. Output schema exists, so detailed return specs are unnecessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter, 'url', described as 'raw job posting URL'. Schema has no description, so the description adds needed clarity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool gathers job, visa, and profile data for scoring, distinguishing it from siblings by combining multiple fetches into one call.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states the tool is the data-gathering half, and after calling it, the AI should compute the score itself. It mentions error handling via return envelope. No explicit when-not-to-use, but clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_profileA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes the error envelope behavior (never raises) and the success/failure return shape. No annotations provided, so this carries full burden and does so well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise, front-loaded with purpose, then usage, then returns. Every sentence adds value with no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter read tool with an output schema, this description is complete: purpose, usage, error behavior all covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, schema coverage 100%. Description does not need to add parameter info; baseline 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns the stored candidate profile with the verb 'Return'. It distinguishes from siblings like setup_profile or update_profile by focusing on retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use it for loading profile to score a job posting. Does not explicitly state when not to use, but context with siblings implies alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_profileA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description fully discloses behavior: overwrites existing, creates if absent, never fails (unless error). Return structure with success/error is also described, leaving no ambiguity about side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise, well-structured with Args/Returns sections. Every sentence adds essential information, no redundancy. Front-loaded with the core action and sibling comparison.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one complex parameter and no output schema provided, the description is remarkably complete: it explains the behavior relative to sibling, specifies how to prepare the parameter, and describes the return value. Enough for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'profile' lacks a description in the schema (0% coverage), but the tool description adds value by explaining it should be the structured profile extracted from the CV and instructing the agent to do extraction before calling. This compensates for the schema gap, though it doesn't detail every nested property.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool saves/overwrites a structured profile, explicitly distinguishes from setup_profile by noting it overwrites without failing and creates if none exists. Verb+resource+scope are specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit step: 'Read and extract the CV into profile before calling' and contrasts with setup_profile, telling the agent when to use this over its sibling. Both when-to-use and when-not-to-use are clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.