Server Quality Checklist

Tool Scores

• codebreaker_list_submissionsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, and the description adds useful behavioral details such as pagination (25 per page, has_next/has_previous) and filter semantics. It goes beyond the annotation without contradicting it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three tight sentences, front-loaded with the core purpose and each sentence adding value: purpose, filter usage, and pagination behavior. No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list tool, the description covers purpose, filter options, and pagination. It does not describe return fields, but no output schema is provided, and the description is adequate for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains the username and problem filters in prose, but the page parameter is not mentioned. While page is straightforward with a default, the description does not fully cover all three parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists submissions with optional filters by problem and/or user. This distinguishes it from sibling tools like get_submission (single submission) and submit (create).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context on using the username filter to see your own submissions and omitting filters for all. It also mentions using codebreaker_me for the username. However, it does not explicitly say when to prefer this over get_submission, though sibling context makes this obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_submitA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses useful behavioral details beyond the annotations: automatic spacing, the return of a submission ID, and the need to poll with wait_submission. It also notes the special case for Communication problems. No contradiction with annotations (openWorldHint=true, destructiveHint=false).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: three sentences, front-loaded with the main action, followed by necessary caveats and usage instructions. Every sentence adds value with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete enough for a submission tool: it covers the return value, how to retrieve verdicts, rate limiting, and a special parameter case. It does not describe error handling or response format, but given the simple schema and sibling tool context, this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero schema description coverage, the description compensates by explaining the optional code_b parameter and its purpose. However, it does not add meaningful detail for problem_id, language, or code (which are fairly self-explanatory), so it provides partial but not complete parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Submit a solution for evaluation,' a specific verb and resource that clearly distinguishes this tool from the sibling get/list/wait tools. It unambiguously identifies the tool's primary function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: it mentions the 2-second spacing, instructs to call codebreaker_wait_submission on the returned ID, and explains when to pass code_b for Communication problems. It lacks explicit when-not-to-use guidance, but as the only submission tool, no alternative is needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_wait_submissionA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so the read-only nature is covered. The description adds the timeout failure behavior and clarifies the return shape matches codebreaker_get_submission, which is useful beyond annotations. It does not detail error types or polling interval, but provides solid context for a polling tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the primary action, and no filler. Every word contributes either purpose or a critical behavior, making it exceptionally concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, and the description covers purpose, timeout behavior, and return shape. It does not specify behavior for an already-graded submission or polling frequency, but these are minor given the context. The description is sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description bears the burden of explaining parameters. It mentions 'timeout seconds', providing units for the timeout parameter, but does not explain submission_id or the optionality/default of timeout. Parameter names are self-explanatory, but only minimal value is added beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states a specific verb ('Poll') and resource ('a submission'), with a clear outcome ('until grading finishes, then return the full result'). It distinguishes itself from siblings like codebreaker_get_submission by focusing on the waiting/polling action, making the purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies usage when you need to wait for grading to complete, and mentions the timeout behavior. It does not explicitly exclude alternatives like codebreaker_get_submission for already-completed submissions, but the polling context is evident and sufficient for an agent to choose it appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_list_problemsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The readOnlyHint annotation already marks this as safe, but the description adds valuable context: login-dependent status display, login requirements for certain commands, and the default exclusion of Joke problems. These behaviors are not inferred from the annotation and help set expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, using a compact paragraph with semicolon-separated details. It covers commands, filters, and pagination without redundancy, though the density might make it slightly harder to parse at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a listing tool with no output schema, the description provides sufficient details about return fields, filtering, and pagination. It also notes login dependencies, which is complete enough for an agent to select and safely invoke the tool. No output schema is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 17%, but the description compensates well by explaining command values, the effect of status/tag/exclude_tags, and offset/limit pagination. It doesn't fully enumerate every enum value (e.g., status 'any' or 'unsolved' as a status), but it provides enough practical semantics for correct usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with 'List problems' — a specific verb and resource — and enumerates the returned fields (status, tags, type, AC counts). This clearly distinguishes it from sibling tools like codebreaker_get_problem, which retrieves a single problem, and codebreaker_list_submissions, which lists submissions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains the command options and their login requirements ('both of the last two require login') and filter parameters (status, tag, exclude_tags) with pagination. However, it does not explicitly state when to prefer this over alternatives or when not to use it, so it stops short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_meA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint annotation, the description discloses conditional behavior: it returns authenticated=true with username/role on success, and authenticated=false with instructions on failure. This adds useful context about the tool's response logic, though it does not describe potential edge cases or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose, and contains no fluff. Every sentence adds value: the first states what it does, the second explains the return behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no parameters, the description is the only source of return information. It fully covers both outcomes (authenticated true/false) and explains what the response contains, making it complete for this simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema is empty, so there is no parameter complexity. Per the rubric, a baseline of 4 applies, and the description does not need to add parameter details. It correctly references the session cookie as an implicit context, not a parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verb 'Report' and resource 'authenticated codebreaker.xyz user', clearly distinguishing it from sibling tools like codebreaker_get_profile or codebreaker_list_problems. It immediately states the core function without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking authentication status but does not explicitly say when to use it or mention alternatives. It lacks explicit when-to-use guidance, such as 'use this to verify the session', and does not exclude other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_get_attachmentA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that the tool downloads to a local cache and returns a path, adding behavioral context beyond the readOnlyHint annotation. It does not mention other details like auth or size limits, but the annotation covers the safety profile and the added caching detail is useful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the action and result, followed by a useful usage hint. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains what the tool does, what it returns (a path), and how to know when to use it. With no output schema, it would benefit from specifying the return type (e.g., string path), but the overall guidance is sufficient for a simple single-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only lists problem_id as a required string with no description. The description refers to 'a problem's attachment' but does not explicitly state that problem_id is the identifier of the problem whose attachment is being downloaded. Since schema coverage is 0%, the description should compensate more, though the tool name and context make the parameter fairly obvious.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Download') and resource ('a problem's attachment (a zip of input files)') and clearly states the result ('return its path'). This distinguishes it from sibling tools like codebreaker_get_problem and codebreaker_list_problems.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to check codebreaker_get_problem's has_attachment first, providing a clear precondition and relationship to a sibling tool. This tells the agent when to use this tool and when to avoid it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_get_profileA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, indicating a safe read operation. The description adds value by detailing the returned data (name, school, role, country, solved problems). It does not contradict annotations and provides useful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no redundancy. The first sentence states the action and outputs, the second provides the use case. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter tool with readOnlyHint and no output schema, the description covers the purpose, return fields, and a typical usage scenario. Minor gaps like error handling or output format details are not critical for this low-complexity tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must clarify the parameter. The description's mention of 'a user's profile' and 'peer's progress' clearly implies the username identifies whose profile to view, adding semantic meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'View a user's profile' and enumerates the specific fields (name, school, role, country, solved problems). It differentiates from siblings by focusing on any user's profile and the use case of checking peer progress.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear usage context: 'Useful to check your own solved set or a peer's progress.' It implies when to use it (for any username) but does not explicitly contrast with alternatives like codebreaker_me, which would merit a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_get_submissionA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses important behavioral details beyond the readOnlyHint annotation: the submitted code is only available for your own submissions, and the pending=true state indicates incomplete grading. These are not inferable from the annotation alone, adding genuine transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose (get full submission results) and then a concise caveat about pending submissions. Every sentence earns its place, with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool, the description covers the return structure comprehensively (score, per-subtask/per-testcase verdicts, times, memory, compile error, code), plus the ownership constraint and pending behavior. Since there is no output schema, this is fully sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not add any meaning to the submission_id parameter; it only appears in the schema as an integer with no further explanation. With 0% schema description coverage, the description should compensate but does not clarify where the ID comes from or any constraints beyond its name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Get' and clearly lists all elements of the full submission results (score, verdicts, times, memory, compile error, code). It also distinguishes itself from siblings like codebreaker_list_submissions and codebreaker_wait_submission by explicitly stating the full result scope and the pending condition.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance: when pending=true, use codebreaker_wait_submission instead of polling manually. This names the alternative tool directly and gives a clear condition for when to use it, which is excellent usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codebreaker_get_problemA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite the readOnlyHint annotation, the description adds substantial behavioral detail: two possible statement shapes, best-effort samples, pypdf extraction, a locally cached PDF path, no image extraction, and optional subtasks/editorials. This goes well beyond the annotation and helps the agent predict the tool's variations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a clear one-sentence purpose, followed by a well-organized paragraph covering variations, edge cases, and alternatives. Every sentence provides useful information, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description carries the full burden of explaining return values, and it does so thoroughly: statement_markdown vs statement_text, PDF path and URL, no images, subtasks, editorials, and the has_attachment flag. It is complete enough for an agent to select and use the tool confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, problem_id, has no schema description and the tool description does not explicitly explain where to obtain it or how it is formatted. However, the parameter name is self-explanatory given the tool's purpose, and no other parameters exist. The description offers some context about what the problem_id refers to but doesn't fully compensate for the 0% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Read a problem statement plus constraints,' which gives a specific verb and resource. It clearly distinguishes itself from sibling tools like codebreaker_list_problems and codebreaker_get_attachment by focusing on the problem statement content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit contextual guidance: it states that attachment zips should be fetched using codebreaker_get_attachment, and that PDF-only problem statements can be handed to a mineru MCP for deeper parsing. It also notes that no images are extracted, setting expectations about what this tool does not do.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.