Server Quality Checklist

Tool Scores

• otp_validate_profileA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description only states that it returns validity status and errors. It does not disclose whether the operation is read-only, any effects on state, or potential side effects. With no annotations, the description carries full burden but is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two clear, front-loaded sentences with no unnecessary words. Every sentence adds value: purpose and parameter usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple validation tool with no output schema, the description adequately covers inputs, operation, and return value. Missing details about schema version scope or error specificity, but sufficient for the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description reinforces the mutual exclusivity of the two parameters and clarifies their semantics (absolute path vs. pre-parsed object), adding value beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a JSON document against a specific schema (OTP v0.1), with clear verb and resource. However, it does not explicitly differentiate from sibling tools like ojp_validate_job_posting, which validates a different schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides guidance on how to use the two input options (filePath or document) but does not indicate when this tool should be chosen over alternatives or mention any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ojp_introspect_job_postingA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must disclose behavioral traits. It states the tool extracts a summary, but does not explicitly confirm read-only nature, side effects, or authorization requirements. The return values are listed, but potential limitations or prerequisites are omitted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no waste. Verb-first structure ('Extract...') immediately conveys purpose. Well-suited for an AI agent's quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's scope (simple introspection of OJP documents), the description covers input options and output structure. No output schema exists, but the listed fields provide sufficient context. Sibling tools are different enough that confusion is unlikely.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already explains both parameters. The description merely restates that either filePath or document can be used, adding no deeper semantic meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Verb 'Extract' clearly identifies tool action on OJP documents. Returns structured summary with specific fields (requirements, compensation, etc.). Distinguishes from sibling tools that validate or parse.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies usage for getting a summary from an OJP document, and mentions 'Provide either filePath or document' as input guidance. However, it does not explicitly state when to use this tool over siblings like ojp_validate_job_posting or ojp_parse_job_posting.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ojp_validate_job_postingA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions that the tool returns 'validity status and a list of errors' but does not disclose whether the tool is read-only or any side effects (though validation is presumably safe). More detail on output format would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading the action, then listing input options, then output. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (2 parameters, no nested objects beyond the document object), the description covers the tool's purpose, inputs, and output sufficiently. However, since there is no output schema, more detail on the structure of the validity status and error list would enhance completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and descriptions already detail filePath as absolute path and document as pre-parsed object. The tool description adds value by clearly stating the two alternatives ('provide either filePath or document') and summarizing their roles, reinforcing the schema information effectively.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it validates a JSON document against the Open Job Protocol schema v0.1. It distinguishes itself from sibling tools like otp_validate_profile (profile validation) and ojp_parse_job_posting (parsing vs validation) by specifying the protocol and the operation (validate vs parse or profile).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides two input options (filePath or document) but does not offer guidance on when to use this tool versus alternatives like otp_validate_profile for profiles or ojp_parse_job_posting for parsing. No explicit exclusions or when-not-to-use context is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• otp_parse_resumeA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It describes the output (skeleton with annotations, confidence list, gaps list) but does not explicitly state whether the tool is read-only or has side effects. For a parsing tool, this is acceptable but could be more transparent about non-destructiveness.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one paragraph of three sentences, front-loaded with the main purpose. Every sentence is necessary and contributes useful information. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter and no output schema, the description covers the output structure (three parts) and the follow-up step. It could mention potential constraints like text length limits, but overall it is complete for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter 'text', which already describes it as 'Raw resume text (plain text, markdown, or extracted PDF text)'. The tool description adds context about processing steps but does not add new semantics for the parameter beyond alignment with the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Produce' and the resource 'structured OTP extraction template from raw resume text'. It distinguishes from siblings by specifying the output elements (document skeleton, fieldConfidence list, gaps list) and mentions the follow-up tool otp_validate_profile, which is listed as a sibling but for a different purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit context: the calling agent should fill in the skeleton after using this tool, then call otp_validate_profile. It does not explicitly state when not to use this tool, but the process is clear and no alternative tools are suggested.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ojp_parse_job_postingA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool returns a document skeleton with _EXTRACT_* annotations, a fieldConfidence list, and a gaps list. It doesn't mention destructive behavior or auth needs, which are not expected for a parse operation. The description is transparent about the output but could add safety details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (3 sentences) and front-loaded: first sentence states purpose, second lists outputs, third gives usage guidance. No unnecessary words, every sentence serves a clear function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple two-parameter schema and no output schema, the description covers the tool's function, output structure (fieldConfidence, gaps list), and workflow. It could mention error handling or input format constraints, but it is sufficiently complete for a straightforward extraction tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the description adds minimal value beyond the schema. The description only paraphrases the 'text' parameter as 'raw job posting text', which is already in the schema. No additional constraints or format details are given.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool produces a structured OJP extraction template from raw job posting text, using specific verbs and nouns. It distinguishes from siblings by mentioning the follow-up call to ojp_validate_job_posting, which is explicitly named as a sibling tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use the tool (parse raw text) and what to do next (fill skeleton then call ojp_validate_job_posting). It also mentions the output structure, aiding the agent in using the tool correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.