codecalc

by The-40-Thieves

Overview Schema Related Servers Score Discussions

Python

Local

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v0.5.0
Disambiguation2/5
Many tools overlap in purpose: evaluate_expression, calc_exact, solve_expression, solve_linear, and z3_check all handle equation solving or expression evaluation with subtle differences. Similarly, bit_analysis, bitop, int_widths, and base_repr all deal with bit/representation tasks. Despite detailed descriptions, the boundaries between these clusters are not immediately clear, increasing the risk of agent misselection.
Naming Consistency4/5
Tool names consistently use snake_case and mostly follow a verb_noun pattern (e.g., list_languages, execute_code, convert_units). Minor deviations like 'bitop' (instead of bit_op) and 'z3_check' (notable for its library reference) are readable and do not break the overall pattern. The naming is predictable enough for an agent to infer functionality.
Tool Count2/5
With 52 tools, this server is far beyond the recommended 3-15 range and even the 16-25 upper bound. While each tool has a legitimate purpose, the sheer count makes it unwieldy for agents to navigate and decide among many parallel options. The surface area could benefit from consolidation or grouping to reduce cognitive load.
Completeness4/5
The toolset covers a wide domain: code execution (with sessions, streaming, background runs), math analysis (symbolic, exact, numeric, stats), bit manipulation, unit conversion, and verification utilities. There are only minor gaps—for example, no explicit update/delete for session files or a tool to list all available math operations—but the core workflows appear well-covered and no dead ends are evident.
Average 3.9/5 across 52 of 52 tools scored. Lowest: 2.4/5.
See the Tool Scores section below for per-tool breakdowns.
- 56 of 56 community issues answered or closed in the last 6 months
- 204 commits in the last 12 weeks
- Last stable release on August 22, 2026
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI is failing
This repository is licensed under Apache 2.0.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries the full behavioral burden. It names output categories but does not disclose return format, edge-case handling, whether the operation is a pure calculation, or any other behavioral details. This is too sparse for an unannotated tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely compact and has no filler words. It front-loads the main output concept and the additional rate outputs, but the terseness contributes to the lack of completeness elsewhere.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations and no output schema, the description is the only explanation of behavior. It does not specify the exact returned representation, whether rates are strings or numbers, or how the input seconds are used. For a one-parameter tool it is partially understandable but far from complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It never mentions the seconds parameter, its valid range, units interpretation, or how rates are derived. The schema title 'Seconds' and the tool name do most of the work, but the description adds no additional semantic value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose3/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description communicates that the tool produces a human-readable duration and per-day/per-30d rates, but it lacks an explicit verb and does not clearly state the operation performed on the input. It is not a tautology, but it is vague enough that it could be confused with conversion or formatting tools like convert_units.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use this tool instead of alternatives such as convert_units or epoch_time. There is no mention of use cases, prerequisites, or exclusions, leaving the agent to infer appropriateness from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It mentions that the tool produces multiple forms (simplified, factored, expanded), but it does not explain whether these are returned together, the order, error behavior for invalid expressions, or any side effects. This is insufficient for a tool with no structured annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single short phrase with no redundant words, which is concise. However, it is under-specified; it could include more useful detail without becoming verbose. It is front-loaded in the sense that the key term 'simplified' appears first, but the lack of structure (no verb, no examples) limits its value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description should explain the return value in detail. It mentions three forms but does not specify the data structure, how multiple forms are packaged, or how the input expression should be formatted. For a tool with one parameter and no annotations, this is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one parameter (expr) with 0% description coverage, and the description does not compensate by explaining what expr should contain or its expected format. The only hint is the word 'expression' in the description, which is too vague to guide the agent in constructing valid input.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies the tool's output—simplified, factored, and expanded forms—which clearly indicates a simplification tool and distinguishes it from siblings like solve_expression or evaluate_expression. However, it lacks an explicit verb (e.g., 'Returns') and could be more direct about the operation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or comparisons with sibling tools like algebraic_equiv or evaluate_expression. The intended use is only implied by the name and output description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral aspects. It mentions the CV threshold for comparing across runs, which is a useful behavioral note, but it fails to describe input expectations (e.g., must be non-empty), output format, or edge cases. The description is too sparse for a tool with no other metadata.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief, with the first sentence listing the statistics and the second providing a practical interpretation of CV. It is concise and to the point, though the second sentence could be seen as extra usage context rather than core purpose. There is no wasted wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with a single parameter and no output schema, the description should clarify what input is expected and what the return value looks like. The description only lists outputs and gives a CV interpretation, but it does not explain the input array, nor any edge cases (e.g., empty array, single element). It is incomplete for reliable usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has only one parameter (nums), and the description does not mention it at all. Schema description coverage is 0%, so the description should explain what 'nums' expects, but it does not. This is a severe gap; the parameter semantics are not clarified at all.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly lists the statistics computed (mean, median, sample stdev, CV), making the purpose evident. It does not explicitly use a verb like 'calculate', but the context is unambiguous. It distinguishes from sibling tools like percentiles by listing its specific outputs, though it does not name alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives is provided. The CV interpretation gives some practical context for interpreting results, but it does not suggest use cases or exclude other tools. There are no explicit instructions on prerequisites or typical scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It doesn't disclose whether the tool requires a symbolic math syntax (e.g., SymPy-like), whether it can throw errors for invalid expressions, or what edge cases behave like. The phrase 'evaluate or simplify' is ambiguous about what the output format is.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is short and front-loaded with the purpose and examples. It does not waste words. However, it could be slightly more structured to mention syntax or limitations.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations, no output schema, and no parameter details, the description is minimal. For a tool that performs symbolic math, the agent would need to know supported functions, syntax, and error behavior. The examples help but are insufficient for anything beyond trivial usage (e.g., how to use 'integrate' vs 'solve'). In comparison to sibling tools, it's incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% (no description for the 'expression' parameter), and there is only one parameter. The description's examples ('integrate(x**2, x)', 'sqrt(144) + 2**10') give some semantic hint about the expected syntax, but they don't fully document the language or grammar of expressions. Since there's only one param, the burden is moderate; baseline with 0% coverage is compensation but examples partially cover it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose3/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('evaluate or simplify') and resource ('a math expression'), and gives concrete examples. However, it doesn't clearly distinguish from sibling tools like calc_exact, simplify_expression, or solve_expression, all of which perform related mathematical operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use this tool versus siblings like calc_exact, simplify_expression, or solve_expression. It also doesn't specify input format requirements beyond the example, leaving the agent to guess.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It explains the binary/decimal distinction, which is helpful, but does not disclose what the tool returns (e.g., a formatted string, object) or any edge-case behavior, such as handling of negative numbers or large values.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very brief, using two sentences for the core purpose and an additional line to emphasize the common confusion. It is front-loaded, but the second sentence about the 'gap' is slightly verbose for the limited information it adds, though generally efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's apparent simplicity (single integer parameter, no output schema), the description is nearly adequate but misses key details: the meaning of 'n' and the exact output format. It clarifies the binary vs. decimal distinction but doesn't specify units or return structure, leaving room for ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has only one parameter 'n', with no description and 0% coverage. The description does not explain what 'n' represents (e.g., the number of bytes, maybe an integer value). It leaves the meaning entirely implicit, failing to add value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool computes byte sizes in both binary and decimal units, with a concise summary of the key difference. However, it does not explicitly differentiate from the sibling 'convert_units' or other related tools, which is a minor gap in clarity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool vs. alternatives like 'convert_units' or 'list_units'. It implies usage by focusing on the 1024/1000 gap, but lacks concrete 'use this when' instructions or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden for behavioral disclosure. It only states the underlying goal (finding roots/crossovers) and gives examples, but does not explain return behavior, handling of multiple roots, numeric vs symbolic solving, error cases, or output structure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with two illustrative examples. It is highly concise with no filler words, and every part contributes to understanding the tool's basic intent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, no annotations, and a nontrivial mathematical task, the description is incomplete. It lacks information about return values, expected expression syntax details, variable handling, and edge cases, making it insufficient for robust agent decision-making.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description fails to explain the 'var' parameter or the required format beyond examples. The examples imply that 'expr' is a symbolic equation string, but this is not explicitly stated, and 'var' is left entirely to the schema title and default.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool solves for a root or crossover, with concrete examples like 'x**2 - 4 = 0' and '2*x + 1 = 7'. It is specific about the mathematical task, though the term 'crossover' is ambiguous and it does not distinguish itself from sibling tools like solve_linear or calc_exact.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use this tool versus its many siblings, such as solve_linear, simplify_expression, or evaluate_expression. There are no stated trade-offs, prerequisites, or explicit alternatives, so an agent cannot reliably choose it over related tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries the full burden of behavioral disclosure. It lists the computations but does not disclose the return format, handling of edge cases (e.g., n=0, negative n, align=null), or whether results are returned as a single object or separate fields. It is minimal but not misleading.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, no fluff, directly lists operations in a compact enumeration. Every word carries information, making it highly efficient and easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no annotations, the description should explain the return value and edge cases. It does not state whether results are returned as a dictionary, tuple, or printed output, nor does it describe behavior when align is null or when n is zero/negative. This leaves uncertainty about downstream handling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explicitly explains align ('padding needed to reach an alignment boundary'), while n is implied as the integer operand in the listed operations. This is adequate but n lacks an explicit definition or constraint description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description enumerates specific bit operations (popcount, bit length, trailing zeros, power-of-two check, next power of two, padding), making its scope clear. However, it lacks a verb and does not explicitly contrast with sibling tools like bitop or int_widths, so it is clear but not fully distinguishing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as bitop, int_widths, or radix_convert. There is no mention of use cases, exclusions, or prerequisites, leaving the agent without selection criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description must disclose all behavioral traits. It mentions structural analysis but does not explain limitations (e.g., may not handle dynamic behavior, recursion, or complex algorithms accurately), nor does it mention what happens with unsupported languages or code errors.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one clear sentence, front-loaded with the purpose, and no unnecessary words. It is concise and to the point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has only 2 parameters and no output schema. The description covers the core purpose but lacks details on language support, output format, and limitations. For a simple analysis tool, it's minimally complete, but a bit more detail on what the output looks like would be helpful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for parameter meanings. It mentions 'code snippet' clearly relates to the 'code' parameter, but 'language' is not described at all, and the default 'python3' is not mentioned in the description. The description adds some meaning for 'code' but not for 'language'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool analyzes code snippets to estimate asymptotic time complexity using structural analysis, distinguishing it from sibling tools like execute_code or simplify_expression.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives, nor any exclusion criteria. The context signals indicate siblings like benchmark or compare_execution, but no comparison is made.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the transparency burden; it does disclose the notable behavior that implausible readings are suppressed. However, it does not define 'implausible', the failure behavior, or output details such as the exact ISO format/UTC marker.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single front-loaded sentence with no filler; every word contributes to the core conversion purpose or a caveat.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple and the description covers input domain and output format at a high level, but with no output schema and no annotations it leaves gaps: exact accepted input syntax, unit indication, and the meaning of 'implausible readings suppressed'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema only provides a string parameter 'n' with no description (0% coverage). The description adds that n represents epoch time in seconds/millis/micros/nanos, but it leaves unclear how the unit is specified or detected and what formats are accepted.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies the conversion from epoch seconds/millis/micros/nanos to ISO 8601 UTC, which clearly names the resource and result. It lacks an explicit action verb but the mapping is specific enough to distinguish it from sibling conversion utilities.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool versus alternatives such as convert_units or human_duration, nor any exclusions. The usage context is only implied by the conversion phrase.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden for behavioral disclosure. It only states that the tool lists files with optional cursor-based pagination; it does not disclose outputs, error behavior, path handling, recursion, permissions, or the meaning of 'bounded.'
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one concise, front-loaded sentence that delivers the core purpose and a key optional behavior without unnecessary words. It earns its place and is easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, no annotations, and four parameters, the description is under-specified. It does not explain what the returned file listing contains, how path filtering works, how cursor/page_size interact, or what a bounded cursor page means in practice.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for the four parameters. It only hints at cursor/page_size via 'bounded cursor page' and provides no explanation of session_id, path, or pagination semantics beyond that.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('List') and the resource ('workspace files'), and adds a distinguishing pagination detail ('optionally using a bounded cursor page'). This separates it from related tools like session_read_file and session_write_file.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied by the phrase 'List workspace files,' but there is no explicit guidance on when to choose this tool over siblings like session_list, session_artifacts, or session_read_file. No when-not-to-use conditions or alternative recommendations are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. It discloses language-dependent extraction accuracy, mentions the `call` parameter for non-python execution, and states the return format ('extracted program and per-input runs'). However, it omits side effects, sandbox details, error behavior, or how missing imports/helpers are handled.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief and front-loaded with the core purpose. The second sentence packs language support and the `call` parameter into one clause, making it slightly dense but still efficient with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 5 parameters, no annotations, and no output schema, the description lacks necessary context. It explains the return value and some language caveats, but fails to cover all parameters or specify edge cases, failure modes, or sandbox implications, leaving the tool under-specified for reliable invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and the description only adds meaning for `call` ('pass `call` to execute non-python'). The other parameters (code, language, function_name, test_inputs) are not explicitly explained; their semantics must be inferred from the tool name and schema titles. The description does not compensate for the low coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action: 'Extract a named function (with its imports + referenced helpers) into a standalone program and run it in the sandbox.' This uses a strong verb and resource, and the extraction+execution purpose distinguishes it from sibling execution tools like session_run or execute_code.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage guidance is implied: the tool is for extracting and running functions. It provides language-specific behavior ('python3 gets exact ast extraction; other languages best-effort') and mentions `call` for non-python execution, but it does not explicitly say when to choose this over alternatives or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It appropriately discloses that the tool flags values beyond 2^53 as unable to round-trip through JS numbers or JSON floats, and that it shows wrapped values when widths do not hold. This is helpful, though it does not detail other side effects or output formatting.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with a concrete example, front-loaded with the core purpose and a precise detail about the 2^53 threshold. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity is moderate (single parameter, no output schema), the description provides adequate details about the tool's behavior, especially the wrap and float limitation. However, it does not describe the output structure or how to interpret results beyond the wrap mention, leaving some ambiguity for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With schema description coverage at 0% and only one parameter, the description compensates with an example that clarifies the expected input type (e.g., int_widths(3000000000)) and purpose. However, it does not fully specify constraints like range limits or whether non-integer input is allowed, which could be inferred from the schema but is not described.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool determines which integer widths hold a given number and reports wrapped values, with a specific example. It distinguishes itself from sibling tools like float_repr or bit_analysis by focusing on integer width range checks and wrap behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies its usage for checking integer width suitability and JS/JSON float round-tripping limitations, but does not explicitly state when to use it over alternatives or when not to use it. Given the specialized purpose, some context is provided but exclusion guidance is absent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses the warning about p99 being the maximum for n<100, which is a key behavioral caveat, and states the calculation methods. However, it does not disclose return format, edge cases (e.g., empty array), or whether both methods are returned separately, leaving some ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core purpose, and includes a valuable caveat. Every sentence earns its place with no filler or redundancy. It is appropriately concise for a simple tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema), the description provides the key facts: which percentiles, calculation methods, and a warning. However, it lacks clarity on the output structure (e.g., does it return an object with both nearest-rank and interpolation results?) and does not address potential edge cases, making it incomplete for a fully informed agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for the 'nums' parameter. It provides no explanation of what the array should contain, constraints, or how it is used beyond the tool name. The parameter semantics are essentially left to the schema type alone, which is insufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool computes p50/p90/p95/p99 percentile values using two methods (nearest-rank and linear interpolation). It names the specific percentiles and the resource (numeric array), making it distinct from sibling tools like calc_stats which likely provide broader statistics.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when percentile values are needed, but does not provide explicit guidance on when to use this tool over alternatives like calc_stats or compare_threshold. No exclusions or alternative tool mentions are present, so it falls into 'implied usage'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds meaningful behavioral context by stating that runner internals like main.py/run.out are excluded, which is useful for an agent deciding whether to call this tool. With no annotations or output schema, though, it still leaves unspecified whether results are file names, full paths, or include metadata, and how invalid session IDs are handled.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that states the action and scope immediately, then adds a useful parenthetical exclusion. Every word contributes to the tool's understanding, with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter list operation, the description covers the core purpose and even clarifies exclusions, which is mostly adequate. However, without an output schema or sibling differentiation, it does not fully describe the return shape, edge cases, or how it relates to tools like session_files, leaving some gaps for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explicitly explain session_id beyond the phrase 'in a session.' The parameter name/title are self-evident, but the description does not compensate for the complete absence of parameter-level documentation with examples or format guidance.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List') and identifies a precise resource ('files created by executed code in a session'), while also excluding runner internals like main.py/run.out. This clearly distinguishes it from sibling tools such as session_list and session_files by defining exactly what artifact scope it covers.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'created by executed code' implies when the tool is appropriate, and the explicit exclusion of runner internals provides a scoping rule. However, the description does not name any alternative tools or state when not to use this one, leaving the choice mostly to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, and the description only provides a caveat about symbolic identity not addressing numerical concerns. It does not disclose return type, error handling, or handling of invalid inputs, leaving behavior partially transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, using two short sentences plus a caveat. No extraneous words, and the example effectively illustrates the intent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of parameter descriptions and annotations, the description offers some context (example, caveat) but lacks details on expected return values, error conditions, or variable handling. It is adequate for a simple tool but not fully comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema provides no descriptions for 'a' and 'b'. The example implies they are strings representing algebraic expressions, but the exact syntax (e.g., Python vs mathematical notation) is not specified, requiring the agent to infer format from the example.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks algebraic equivalence of two expressions, with a concrete example '(a*b)/c vs a*(b/c)' and explicitly notes it ignores numerical issues like floating-point rounding.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The caveat implies when not to use it (e.g., for numerical equality), but it does not explicitly reference sibling tools like 'compare_threshold' or 'solve_expression' for alternative use cases. Use is implied rather than explicitly guided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does disclose key behavior: per-language stdout/stderr/exit/duration and which was fastest, plus the requirement that each snippet be valid. However, it omits important execution context like sandboxing, network access, failure handling, and how stdin/timeout affect execution.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact, front-loaded with the core purpose, and includes a helpful example. Every sentence adds value, and the formatting with a code snippet makes it easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no annotations and no output schema, the description provides a reasonable overview of purpose, input, and return categories. However, it leaves gaps around stdin/timeout semantics, language availability, and failure/error behavior, which are relevant for a code-execution tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description compensates well for the primary parameter 'snippets' by explaining the language-to-code mapping and giving an example. It does not explain the 'stdin' or 'timeout' parameters at all, relying on the schema's defaults, so compensation is incomplete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action: 'Run the same code in multiple languages side by side.' It also explains the return value (stdout/stderr/exit/duration plus fastest) and gives a concrete example, making the tool's purpose unambiguous and distinguishing it from single-language execution tools like execute_code.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool: when comparing the same code across multiple languages. However, it does not explicitly mention alternatives or exclusion cases, such as 'use execute_code for a single language' or 'use benchmark for detailed performance analysis.'
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden of behavioral disclosure. It adds context about the default point (oo) and provides a working example with a return value, but doesn't disclose edge cases (e.g., divergent limits, multiple variables, error handling for undefined limits). Acceptable but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: the first defines the operation precisely, and the second provides a concrete example with a humorous but relevant closing phrase. Every word earns its place; no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple 3-parameter tool with no output schema, the description is reasonably complete: it explains inputs via example, default behavior (limit to infinity), and outputs (a value). It doesn't mention failure modes or handling of symbolic constants, but given the low complexity, this is acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description's example ('limit_expression("n*log(n)/n**2", "n")') clarifies the positional order (expr, var) and the default point. It adds meaning by showing the expression and variable usage, though 'point' remains implicit as the default oo. Good compensation for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool computes the asymptotic limit of an expression as a variable approaches a point (default infinity), with a concrete example. It distinguishes itself from siblings like solve_expression and simplify_expression, though it doesn't explicitly name them as alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage via the example ('settles complexity arguments faster than arguing') but doesn't explicitly state when to use this over alternatives like analyze_complexity or compare_execution. No exclusions or prerequisites are mentioned, making the guidance adequate but implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden of behavioral disclosure. The description states it lists all supported unit aliases, which is a read-only operation, but it does not mention any potential side effects, performance implications, or whether the list is static or dynamic. Given the simplicity of the tool, this is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that directly states the tool's function. It is front-loaded with the verb and resource, and there is no wasted wording. It is appropriately sized for a tool with no parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (no parameters, no output schema, no annotations), the description is complete enough. It clearly states what the tool does and its relationship to convert_units. However, it could mention whether the list is exhaustive or if there are any limitations, but for a listing tool, this is sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, and the schema coverage is 100% (since there are no parameters). The description adds value by explaining what the tool returns (a list of unit aliases) and its purpose (for convert_units). With no parameters, the description's role is to clarify the output, which it does effectively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'List every supported unit alias for convert_units.' It specifies the verb (list), the resource (unit aliases), and the scope (every supported alias), which distinguishes it from sibling tools like convert_units. However, it doesn't explicitly mention the output format or how the list is organized, which could be slightly ambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage: it is meant to be used to discover unit aliases for convert_units. However, it does not explicitly state when to use this tool versus alternatives, nor does it provide any exclusions or prerequisites. The context is clear but not fully elaborated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the burden. It discloses basic behavior: a lookup or list operation, and mentions 22 constants. It doesn't address error handling, return format, or any side effects, but for a simple read-only lookup this is reasonably transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single well-formed sentence that packs all essential information without fluff. It front-loads the verb and provides examples inline, making it easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a straightforward lookup tool with one optional parameter and no output schema, the description sufficiently covers the essentials: what it does, how to use it, and what to expect (values for constants). Minor gaps remain (e.g., exact return format, error cases), but it's complete enough for a simple tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has one parameter 'name' with zero description coverage (0%). The description compensates by giving example values and explaining that omitting the name yields a list of all constants. This effectively communicates the parameter's meaning and the behavior for null.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: look up a specific physical constant or list all constants. It provides concrete examples (speed_of_light, planck, etc.) and distinguishes two modes (specific lookup vs. list all). While it doesn't explicitly contrast with siblings like convert_units, the verb 'look up' is specific enough for the resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage: provide a name for a single constant, omit it to list all. It doesn't explicitly state when to prefer this over convert_units or calc_exact, but the context is clear. Lacks explicit exclusions or guidance on when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, and the description does not disclose the output format, return behavior, error handling, or any constraints (e.g., linearity, solution domain). This leaves the agent uncertain about what to expect from the tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, uses a clear structure, and includes an illustrative example without any redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the input format is well explained, the description does not mention the output shape or any expectations about the solution (e.g., unique solution, multiple solutions, no solution). Given the absence of an output schema, this omission reduces completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no descriptions for the parameters, but the description explains the delimiters (';' for system, ',' for variables) and includes a concrete example, adding meaningful context that the schema lacks.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool solves a system of equations and provides the exact input format with an example, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains how to format inputs but does not explicitly state when to use this tool over alternatives like solve_expression or simplify_expression. The name and sibling context imply it is for systems, but it lacks direct guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses key behaviors: two's complement handling, signed-overflow detection, and an example that shows how overflow is reported. However, it omits details on return format, error behavior, or edge cases like negative numbers without width.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: the first concisely states the purpose and key modifiers (width, two's complement, overflow), and the second provides a concrete example. Every word earns its place with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and only two parameters, the description captures the essential behavior but leaves gaps: it does not specify the return type (e.g., a string representation) or how errors are signaled beyond the example. For a tool with overflow detection, more detail on output structure would be helpful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description adds meaning beyond the schema: it explains that width enables two's complement and overflow detection, and the example clarifies the parameters' interaction. It does not fully define the output format for each base, but it compensates adequately for the lack of schema detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the core function—converting N to hex, octal, or binary—with explicit mention of width-based two's complement and overflow detection. It clearly distinguishes from siblings like float_repr and radix_convert by highlighting the overflow-checking capability.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when width or overflow detection is needed, but it does not explicitly state when not to use this tool or name alternative tools. The example hints at a specific scenario but lacks direct guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full transparency burden. It discloses that conversion uses sympy and lists supported dimensions, but it does not describe return format, invalid-unit behavior, case sensitivity, or precision. This is useful context but leaves notable behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core action. The supported-categories list, examples, and pointer to list_units each add value without redundant filler, and the entire description is easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is reasonably complete for a simple conversion tool, covering scope and examples. However, there is no output schema, no annotations, and no mention of return values or error behavior. Additionally, the sibling data_sizes tool may overlap with data-size conversions, but the description does not clarify the relationship.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for parameter meaning. The examples ('60','mph','km/h'), ('100','celsius','fahrenheit'), and ('1','gb','mib') illustrate the value/from_unit/to_unit ordering and alias usage, but the description does not explicitly define each parameter or mention constraints like case sensitivity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Convert a value between units (dimensional analysis via sympy).' It clearly enumerates supported unit categories and provides concrete examples, making the tool's purpose unmistakable and distinct from siblings like list_units or calc_exact.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context on what the tool handles (metric/imperial length, mass, temperature, data sizes, etc.) and directs users to list_units for the full alias table. It does not explicitly state when not to use this tool versus siblings like data_sizes, but the supported categories and examples make the intended usage clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'exact' implying no rounding, but does not describe the output format, error handling (e.g., total=0), or any side effects. This is insufficient for a tool with no structured metadata, leaving the agent guessing about the return type and edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence of eight words. It front-loads the core function and includes the key constraint (rationals accepted) without any redundant wording. Perfectly concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter calculation tool, the description is mostly adequate, but it leaves important context missing. It does not specify the output format (e.g., decimal, percentage string) or how to handle invalid inputs like zero total. These omissions could lead to incorrect invocation or misinterpretation of results, making it slightly incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema parameter descriptions are non-existent (0% coverage), so the description must compensate. It does by clearly mapping 'PART' and 'TOTAL' to the parameters and explaining their relationship via the division formula. The note about accepting rationals adds semantic detail that is not present in the schema, making the parameters more understandable.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool computes an exact share and percentage from PART/TOTAL. It uses a specific verb-implicit calculation and distinguishes from sibling 'percentiles', which deals with distribution percentiles. The inclusion of 'rationals accepted' adds precision to the input domain, making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context that this tool is used when a percentage or share of two numbers is needed. However, it does not explicitly mention alternatives or when not to use it, so it lacks explicit exclusions but provides enough contextual clarity for typical use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full responsibility for behavioral disclosure. It reveals that fractions are included, and bases that cannot represent a fraction are flagged as non-terminating. It also provides a concrete example call. It does not mention side effects (likely none) or error handling beyond flagging, but the core behavior is transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise sentences, front-loaded with the main purpose and followed by a valuable example. Every sentence adds meaningful information, and there is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (base conversion with fractions and non-terminating flagging) and the absence of an output schema, the description covers the essential behaviors and provides an example. It does not detail return values or error messages, but the conversion process is reasonably specified. The description falls short of a perfect score because it leaves parameter-specific details to the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 3 parameters with 0% description coverage, so the description must clarify each parameter. It only mentions the value and bases implicitly via the example 'radix_convert('zz', 36, 7)'. It does not explain the role of from_base vs to_base, their defaults, or that the value must be a string. The example partially illustrates parameter order but is ambiguous about which base is source vs target.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool converts values between any bases 2..36, including fractions. It uses a specific verb ('convert') and a specific resource (value between bases), and the qualification 'ANY bases 2..36, fractions included' distinguishes it from sibling tools like base_repr or float_repr. The example call further clarifies the tool's scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for base conversion but does not explicitly state when to use this tool over alternatives like base_repr. It does not mention constraints or exclusions, though the scope (2..36, fractions) is clear. There is no guidance on when not to use it, leaving the agent to infer from the description alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries some burden. It says 'List active sessions' which implies a read-only operation, but does not detail any behavioral traits beyond that, such as what 'state' means or whether the result is a full snapshot. No contradictions, but limited disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence, front-loaded with the verb and rescope. No wasted words. It perfectly balances brevity with clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters and no output schema, the description adequately communicates the tool's purpose. It doesn't explain the output format, but for a simple list operation, the description is sufficient. Could detail different states, but not critical.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the parameter semantics are trivially covered by the schema. The description adds meaning by specifying what information is returned (sessions, languages, state), which enriches the semantics beyond the empty schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List') and resource ('active sessions'), clearly stating what the tool does. It directly states the session scope which is distinct from read/run/write session tools, though it doesn't explicitly mention filtering.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the use case: checking active sessions and their languages/state. While there are many session-related sibling tools, the description clearly indicates this is for listing/metadata, but doesn't explicitly state when not to use it (e.g., for getting content use session_read_file).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the behavioral burden. It does disclose the destructive actions ('kill ... delete its workspace') and notes the conditional REPL worker, which is useful. However, it does not mention irreversibility, permissions, or side effects on running artifacts, leaving some transparency gaps for a destructive operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with the action, and every word contributes meaning. There is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter destructive tool with no output schema, the description captures the essential behavior: stopping a session and its resource cleanup. It could add notes about permanence or prerequisite lookup, but the current scope is mostly sufficient for safe invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain the session_id parameter beyond what the schema already shows. Since the parameter is a single simple string, the gap is minor, but the description fails to add any semantic depth such as format expectations or where to obtain a valid session_id.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Stop') and resource ('session'), then clarifies scope by naming the two concrete effects: killing the REPL worker (if any) and deleting the workspace. This clearly distinguishes it from sibling tools like session_start, session_run, and session_list.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intent is obvious from the imperative 'Stop a session' and the sibling context. It provides clear usage context but does not explicitly state when not to use it or mention alternatives, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral details. It mentions 'relative path, no escapes' and implies a write operation, but it doesn't state whether the file is overwritten, if directories are created, or any error conditions. It adds some context (seeding input) but lacks depth on side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no fluff. The purpose is front-loaded, and every word adds value. Efficient and clear.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple but with no annotations and no output schema, the description should cover important behavioral aspects. It explains the use case and path constraint, but omits details like overwrite behavior, error handling, and return values. For a basic write, it's adequate but incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has no descriptions (0% coverage), so the description must compensate. It clarifies that 'path' is relative and cannot escape the workspace, and 'content' is the data to write. However, it doesn't explain 'session_id' beyond its name, and no format or constraints for content are given. Some meaning added, but not comprehensive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool writes a file into a session workspace, specifying the verb 'Write' and resource, with an additional constraint 'relative path, no escapes'. It also distinguishes from sibling tools like session_read_file by focusing on writing and seeding input data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear use case: 'Use this to seed input data for executed code.' This implies when to use it, though it doesn't explicitly mention alternatives or when not to use it. The constraint on relative paths is also helpful. Slightly more explicit exclusions would make it a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral disclosure burden and does so well: it explains the output items (exact value, raw bits, ULP, neighbors, representability), gives concrete examples for both round (0.25 → EXACT) and non-round (0.1 → ...0055511151231257827) cases, and surfaces the 2^53 edge-case caveat. Minor gap: it doesn't cover special values like NaN/Infinity or state the return format explicitly, keeping it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, zero waste, and the most important information (output contents) is front-loaded. The prose is telegraphic in places ('...says EXACT.') and the final sentence is somewhat awkwardly phrased, but the density of useful information per word is high.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with no annotations and no output schema, the description covers the essential ground: what's returned, representability behavior, and a precision edge case. It could mention the output format for special float values (NaN, Infinity, negative zero) to be fully complete for a tool that inspects IEEE 754 internals, but the coverage is solid given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% with one required parameter, so the description must compensate—and it does so exceptionally. The phrase 'what binary64 actually stores for X' grounds the parameter's meaning, and the worked examples (0.1 and 0.25) clarify how x is interpreted and what output to expect. This exceeds what a typical schema description would convey.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states what the tool does—showing what binary64 stores for X, including exact value, raw bits, ULP, neighbors, and representability—which is a specific, well-scoped purpose. It's topically distinct from siblings like base_repr or int_widths, though it doesn't explicitly name them. The construction is somewhat fragmented/passive rather than a clean 'verb + resource' statement, so it doesn't earn a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The usage context is implied—if you need to inspect binary64 float internals, this is the tool—but there's no explicit guidance on when to use this versus the many math/CS sibling tools, nor any 'when not to use' advice. For a large sibling family with a 0% schema coverage and no annotations, an explicit alternative mention (e.g., base_repr for integers) would elevate this beyond the implied level.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must convey all behavioral traits. It mentions that each entry is screened per-entry to avoid RCE, similar to evaluate_expression, and that operations are computed. However, it does not specify error handling for invalid matrices (e.g., singular matrix for inverse), return type variations across operations, or any side effects. This leaves some ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and flows logically: introduction, comparison to evaluate_expression, then parameter details and example. It is somewhat verbose with repeated emphasis on 'screening' and 'structured replacement,' but each sentence contributes to the overall clarity without excessive redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description lacks completeness regarding output formats. It only illustrates the result for 'det' (a scalar -2) but does not specify the return types for inverse (matrix), eigenvalues (list), etc. It also does not address error cases (e.g., invalid dimensions, singular matrix, unsupported expressions) or the exact behavior when an entry cannot be screened. Given the non-trivial nature of matrix operations, this is a significant gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite zero schema description coverage, the tool description fully explains both parameters: 'rows' is a JSON array of arrays (row-major) with entries that can be numbers or scalar expression strings (with examples), and 'op' is one of the enumerated matrix operations. This provides complete semantic understanding, exceeding what the schema alone offers.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: performing structured matrix operations (det, inverse, eigenvalues, etc.). It explicitly contrasts this tool with evaluate_expression, explaining that it provides a safe structured alternative for matrix inputs, which distinguishes it from siblings. The example further clarifies intended usage.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states that this tool is the 'structured replacement' for evaluate_expression in the context of matrices, giving clear guidance on when to use it. It provides an example (det -> -2). However, it does not explicitly mention when not to use it (e.g., for non-matrix numerical expression evaluation), but the context implies it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses the execution model (runs code at each size, fits growth curve) and the N-from-stdin contract. It does not mention sandboxing, error handling, or side effects, but for a benchmark/execution tool the described behavior is sufficiently transparent for an agent to anticipate outcomes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a compact paragraph with a purpose statement, a contract, and an example. It is front-loaded and each sentence adds value. The mention of 'codecalc' is slightly vague but not distracting; overall it is appropriately sized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is moderately complex (executing code, fitting curves), and the description covers the core workflow and input requirements. It does not describe the exact output format (e.g., how the Big-O estimation is presented), nor does it mention failure modes, but the provided example and contract give enough context for an agent to use it correctly in most scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains the meaning of `code` (must read N from stdin) and `sizes` (comma-separated sizes), but gives no explanation for `timeout` or `language` beyond what the schema defaults imply. The example helps, but two of four parameters remain underspecified.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Empirically measure time complexity by running code at increasing input sizes.' This clearly distinguishes it from static analysis siblings like analyze_complexity, and explicitly mentions the Big-O output. The purpose is unambiguous and differentiated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on usage: it requires code that reads an integer N from stdin and does work sized by N. However, it does not explicitly mention alternatives or when not to use this tool, so it stops short of a 5. The contract is clear enough for most cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the return value ('Returns a per-input matrix plus a divergences list') and clarifies expected snippet format ('provide a correct snippet per language'), which is useful. However, it does not explain error handling, performance implications, or what happens if a language fails.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at three sentences but imparts all key information. The inclusion of the default input list is helpful, though it slightly lengthens the description. Overall, it is well-structured and front-loaded with the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 2 parameters, no annotations, and no output schema, the description covers purpose, parameter semantics, and output. It lacks information about prerequisites (e.g., available languages) and edge cases like duplicate snippets, but is still sufficiently complete for basic usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description compensates by explaining 'snippets maps language -> code' and describing the default 'inputs' list. It goes beyond the bare types by giving a concrete example of default edge-case values, though it could further clarify the structure of each snippet or the return format in more detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool 'Run the same logic in N languages on edge-case inputs and flag divergence.' It uses a specific verb ('Run') and resource ('same logic in N languages') and distinguishes itself from siblings like 'compare_execution' by focusing on edge-case inputs and divergence detection.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied through the description of default inputs ('Default inputs cover empty, zero, negative, and float-precision cases') but no explicit alternatives or exclusions are mentioned. The tool's purpose is clear, but the description does not tell the agent when not to use it or what related tools exist, making it only implicitly a comparison tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the full burden of behavioral disclosure. It explicitly warns about network use, child-process registry access, install-time hooks, non-sandboxed filesystem access, allowlist restrictions, and untrusted input. This is unusually thorough and materially helps an agent understand the risks.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well structured and front-loaded: purpose, session behavior, network behavior, then security warning. Every sentence adds meaningful information, and the formatting uses caps labels and clear sections without waste or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there are no annotations and no output schema, the description is unusually complete for a network-touching, mutating tool. It covers workspace scoping, cache behavior, network, and security implications. The only remaining gaps are exact accepted language values, package/version syntax, and expected success/error return behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no descriptions for any of the four parameters, so the description must compensate. It does explain the behavioral meaning of session_id and hints at supported language values through examples, but it does not clarify the expected format of package or version, leaving those to inference.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly names a concrete action (install) and a concrete resource (a package for a language), with helpful package-manager examples. It does not explicitly distinguish itself from sibling tools such as execute_code, but the core purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear usage context: with a session_id it installs into that session workspace; without one it installs into the shared cache. It also notes that network access is always required, but it does not explain when to choose another tool instead or when installation should be avoided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description discloses key behaviors: it runs as a fresh process in a specific directory, returns specific outputs, and handles oversized output with spill. It references execute_code's docstring for spill details, which adds context, though it doesn't detail permissions or limit specifics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core purpose, then adds relevant details. It efficiently explains workdir behavior and spill handling in two short paragraphs, though the reference to execute_code's docstring is somewhat indirect.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (multi-file execution, session state, output handling) and no annotations or output schema, the description is quite complete. It covers execution environment, import resolution, return values, and spill behavior, though it could detail parameter specifics more.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It defines entry_file and session_id implicitly through usage, but does not explain timeout, stdin, or language individually. It adds some meaning around entry_file execution context but not comprehensive parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool runs a multi-file program by executing an entry_file within a session, and explicitly notes it may import other files. It distinguishes itself from siblings like session_read_file and execute_code by specifying the session workspace execution context and the fresh process in workdir.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly indicates when to use this tool: for multi-file programs in a session where relative imports matter. It contrasts with the REPL worker and mentions spill behavior similar to execute_code, but doesn't explicitly name alternative tools for single-file or non-session execution.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full disclosure burden. It reveals the underlying formula (1 - exp(-n^2/(2*2^b))) and provides example outputs, which is transparent about the computation. However, it does not mention what the function returns (decimal, percentage, rounding), possible edge cases (items=0, bits=0), or that it's an approximation (it uses exp). This is adequate but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the formula, then provides two illustrative examples and a practical takeaway. Every sentence earns its place, and the structure is clear. Slightly more formatting (like parameter definitions) could improve scannability, but it's already quite efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool is a simple calculation with only two numeric inputs and no output schema, the description is complete. It explains the purpose, gives the mathematical formula, provides example sizes with results to build intuition, and ties it to a real-world question. No additional context is needed for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no descriptions for 'items' and 'bits' (0% coverage), so the description must compensate. It does so by explaining that items is the number of items and bits is the hash size in bits, and gives concrete examples (1e6 items into 64 bits, 1e5 into 32 bits) that illustrate their meaning. This adds meaning well beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool computes birthday-bound hash collision probability and provides the formula. It distinguishes itself from sibling tools by being the only one focused on collision probability, and gives a concrete use case ('can I truncate this to 8 hex chars?'). This is a specific verb+resource with clear purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it: when you need to assess collision risk when truncating hashes, supported by numerical examples. It does not explicitly exclude alternatives or compare to other calculation tools, but the examples and phrasing give clear contextual guidance. A slightly more explicit 'use this when...' statement would push it to 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that both sides are evaluated exactly and printed as fractions, and that the comparison cannot be reversed, which is useful behavioral context. It does not mention error handling or return format details, but the core behavior is transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, front-loaded with the core purpose, and includes a clear example. Every sentence adds value: the first states the purpose, the second explains the syntax and evaluation behavior, and the example clarifies usage.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (3 string parameters, no output schema), the description is largely complete. It covers the operation, operators, evaluation behavior, and provides an example. It could mention what the 'verdict' and 'shortfall' look like in the output, but the absence of an output schema makes this less critical.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains that 'a' and 'b' are operands compared with 'op', and gives an example ('1/25', '>', '0.05'). However, it does not specify the expected format for 'a' and 'b' (e.g., fractions, decimals) beyond the example, nor the exact set of operators beyond listing them. This is adequate but not exhaustive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool performs an exact threshold comparison with a verdict and shortfall, and specifies the supported operators. It distinguishes itself from sibling tools like compare_edge_cases by focusing on exact threshold checks with fractional evaluation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for exact comparisons where fractions must be evaluated precisely, and notes that 'a threshold comparison written out cannot be gotten backwards,' which hints at when to use it. However, it does not explicitly mention when not to use it or name alternative tools for edge cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It explains that it builds a truth table and gives examples of supported operators, but it omits important behavioral details such as output format, operator precedence, handling of parentheses, and potential limitations. This is a moderate gap given no other structured information.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that starts with the action 'Build', immediately communicates the core purpose, and includes useful examples without unnecessary fluff. Every element earns its place and is easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one string parameter, no output schema), the description provides sufficient context to understand the tool's purpose and input format. It covers supported operators and example usage. It could add details about the return representation, but for a straightforward truth table generator this is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no description for the 'expression' parameter (0% coverage), so the description compensates by specifying it takes a boolean expression and offering concrete examples. This adds meaningful context about expected syntax and supported operators, though it could be even more precise about the exact grammar.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Build' and resource 'truth table for a boolean expression', making the tool's purpose immediately clear. It also distinguishes itself from sibling tools like calc_exact or solve_expression by focusing on truth table generation rather than general evaluation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it by providing example expressions ('a and b or not c', 'p xor q', 'a implies b') that illustrate the expected input format. However, it does not explicitly state when not to use it or contrast with alternative tools such as evaluate_expression or solve_expression, so it lacks full exclusion guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden, and it does well: it discloses streaming behavior, identical result shape and ceilings for max_memory_mb, max_output_kb, and max_cpu, and the 300s vs 120s timeout difference. It stops short of describing the exact progress event shape, but the disclosed behavior materially exceeds what the schema alone would provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently structured: the purpose and key distinction are front-loaded, followed by behavior and the deliberate timeout caveat. Every sentence earns its place, and there is no redundant restating of the input schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the essential operational context: streaming, same result shape as execute_code, forwarded limits, and the timeout deviation. The main gaps are the absence of a progress-event format and sparse per-parameter semantics, but for selecting and invoking the tool the description is largely sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for 9 parameters, so the description must compensate. It adds meaning for the execution-limit parameters (max_memory_mb, max_output_kb, max_cpu, and the timeout cap) but leaves code, language, stdin, no_net, and provider undefined. This partial coverage is insufficient for a schema with zero inline parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb-resource pair ('Execute code and STREAM progress') and immediately distinguishes this from execute_code, which returns only at exit. An agent can tell exactly what this tool does and how it differs from the closest sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly names execute_code as the alternative and gives the decision condition: use this tool when you want progress and partial output before the process finishes. The wall-clock timeout difference also helps agents choose based on expected run duration.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully carries the behavioral disclosure burden. It states that no runtimes are executed, explains what 'installed', 'supported', and 'resolved' mean, and clarifies the separate tier reliability axis. This prevents an agent from over-trusting the status field and is exactly the transparency needed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but dense; the first line states the core purpose and the rest explains two subtle concepts that could easily be misinterpreted. The second paragraph about tier vs status earns its place and prevents a common misreading.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters and an existing output schema, the description covers the important semantics thoroughly. The only gap is a small internal inconsistency: it references promoting a runtime to 'available' while the status enumeration only mentions 'installed' and 'supported'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the schema leaves nothing unspecified. The description appropriately uses the space to explain output semantics instead, which is the right trade-off.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence clearly identifies the verb ('List'), the resource ('every language codecalc can execute'), and the fields returned (extension, compile flag, machine resolution). It is unambiguous, but it does not explicitly distinguish itself from sibling tools such as runtimes_status, so it narrowly misses 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly warns that status_basis is 'resolved' and that nothing was executed to check, and it directs the user to codecalc doctor --deep for actual verification. This provides a clear when-not-to-rely-on-this-tool signal, but it never names an alternative MCP sibling tool for related tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full behavioral disclosure. It thoroughly explains the process: runs both versions against same inputs, times at increasing sizes, and reveals that a rejection indicates which gate failed and by how much. It also clarifies grading (cross_checked vs ungraded) and the semantic implication that correctness alone doesn't earn a grade for the optimization claim.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is moderately lengthy but each sentence contributes: it states the purpose, explains the process, defines acceptance, and describes outcomes. It is well-structured and front-loaded with the core purpose, avoiding redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is quite complete for a complex tool with no annotations or output schema. It explains the process, acceptance gates, rejection feedback, and grading outcomes. It references 'grade_basis' without detailing it, but overall it provides sufficient context for an agent to understand the tool's function and likely outcomes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It explicitly mentions min_speedup (with default) and implies test_inputs and sizes ('same inputs', 'increasing sizes'), but does not define original, candidate, or language beyond context. The parameters are self-explanatory from the tool's purpose, but the description doesn't add detail for all six parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'PROVE an optimisation: same outputs, and measurably faster.' It specifies the acceptance criteria (equivalent AND at least min_speedup faster) and distinguishes it from sibling tools like benchmark or compare_execution by tying correctness and performance verification together.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use the tool ('You write the optimised version') and details the rejection criteria, including that speed is never measured if correctness fails. It does not explicitly name alternatives, but the context implies this is for proving optimizations rather than general benchmarking or comparison.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It clearly states the action (runs both programs), the output (reports per input: matched, diverged, inconclusive), and handles edge cases (missing runtime or both failing = inconclusive, never a pass). It also explains the grading: a pass is 'cross_checked', a non-pass is 'ungraded'. This is transparent about the tool's behavior and consequences. It does not mention side effects (e.g., whether it modifies state), but for a verification tool, the primary behavior is execution, which is disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is structured into three clear paragraphs: purpose, usage/context, and grading semantics. Each sentence adds value—the first paragraph defines the core action, the second gives usage guidance, and the third clarifies the grading. It is slightly longer than minimal but remains efficient, with no fluff. The information is front-loaded with the core purpose, making it easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (5 parameters, no output schema, no annotations), the description covers the essential aspects: what it does, how it reports results, edge cases (inconclusive), grading semantics, and complementary tool. It does not fully specify the exact return format or what happens when test_inputs is null, but it provides sufficient understanding for an agent to use the tool effectively. The cross-reference to 'grade_basis' is a minor gap but not critical.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It implicitly defines the core parameters: source_code and target_code as 'your source and your port', test_inputs as 'the same inputs' to run both programs, and source_language/target_language through the examples 'python3 -> go, node -> rust' and the mention of 'runtimes'. It explains the behavior of test_inputs (per input reporting) and the importance of languages for runtime selection. While it doesn't explicitly map each parameter in a checklist, it provides enough semantic context for an agent to infer usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a bold, specific statement: 'PROVE that a port is equivalent: run both programs, compare their output.' It clearly identifies the verb (verify/prove), the resource (a port/translation), and the method (running and comparing). This distinguishes it from siblings like compare_edge_cases (finding edge cases) and compare_execution (comparing execution metrics) by focusing on equivalence validation. The phrasing is unambiguous and not a tautology.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool: 'Use it after porting anything: python3 -> go, node -> rust, a rewritten function against the original.' It also suggests a complementary tool: 'Pair with compare_edge_cases to find the inputs worth testing.' This provides clear use context and hints at an alternative for finding test inputs. It does not explicitly mention when NOT to use it or compare with all alternatives, but the guidance is strong enough for an agent to decide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral burden. It explains the grading semantics thoroughly: unsat is solver_proven, sat and unknown are ungraded, and why a counterexample cannot wear a proof grade. This goes beyond basic expectations for a checker tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the main purpose and example, then adds useful grading context. The grading semantics are somewhat detailed but earn their place because they clarify the proof-related meaning of result values.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple: one string parameter, no output schema, no annotations. The description explains the output categories (sat/unsat/unknown plus a model) and the grading distinctions, which is sufficient for this complexity level. It references grade_basis for engine version and timeout details, adding useful context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides only a string parameter named smt2 with no description. The tool description compensates by clarifying the parameter must be an SMT-LIB2 formula and supplies a full example showing acceptable syntax, including declare-const, assert, and check-sat.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks an SMT-LIB2 formula with Z3 and returns sat/unsat/unknown plus a model. It includes a concrete formula example and is distinct from sibling tools like solve_expression or execute_code.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: use this tool when you need to check an SMT-LIB2 formula with Z3 specifically. It does not explicitly name alternatives or exclusions, but the specialized scope is evident from the description and tool name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully carries the burden and does so thoroughly: it discloses output formats (unsigned, signed, hex, octal, binary), distinguishes logical vs arithmetic shift, and specifies overflow behavior. The concrete 0x80 example clarifies how results are computed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: the first sentence states scope, the second covers output representation, and the third explains critical edge cases. Every sentence adds useful information with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the central complexity: operations, widths, output representations, shift distinctions, and overflow behavior. Since there is no output schema, the return-format description is especially valuable. It falls just short of full completeness because it does not document error behavior for invalid op values or clarify operand requirements for all operations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It lists valid operation names, width options, and uses the '0x80 shr 1' example to imply operand roles for a and b. However, it does not explicitly state which operations require b or how b is used in shifts, leaving a small gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool as providing bitwise operations, listing specific operations ('and or xor nand nor xnor not shl shr sar rol ror') and supported widths (8/16/32/64). It distinguishes itself from siblings like base_repr or bit_analysis by focusing specifically on performing bitwise computations with multi-base output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The tool's intended usage is implied by the operation list and examples, but there is no explicit 'use this when...' guidance or comparison to alternative tools. The shift semantics example ('shr is logical; sar is arithmetic') gives helpful context but does not address when to prefer this tool over sibling utilities.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It thoroughly explains the returned state shapes, the fact that reading is non-destructive ('nothing is consumed by reading it'), and that provider resources are released on first terminal read (RunSupervisor.cleanup()). It also discloses memory growth and references the durable state machine scope. This is exemplary transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is detailed and covers many important aspects (state shapes, retention, cleanup, edge cases). Every sentence adds value, but it is somewhat verbose for a simple polling operation. It could be condensed without losing critical information. However, given the complexity of the behavior (retention, resource cleanup, cancellation nuances), the length is justified.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no annotations, the description must fully explain the return values and behavioral nuances. It does so comprehensively: describes both running and terminal states, merges with execute_code result shape, handles cancellation caveats, explains retention and cleanup semantics, and notes memory implications. This is complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has only one parameter, run_id, with 0% description coverage. The description does not explicitly say that run_id is the identifier returned by run_submit, though it is implied by 'Poll a background run started with run_submit.' This is adequate but leaves room for explicit clarification (e.g., 'run_id as returned by run_submit'). The description adds minimal parameter-specific detail beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Poll a background run started with run_submit', which clearly states the verb (poll) and the resource (a background run), and immediately distinguishes it from the sibling run_submit (which starts runs) and run_cancel (which stops them). This is a specific, action-oriented description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly states when to use the tool (to poll a background run) and provides guidance on interpreting terminal results, including checking 'ok' and 'verdict'. It also references run_cancel's docstring for cancellation semantics. However, it does not explicitly state when NOT to use it (e.g., if you need to start a new run or cancel one), though the context makes that clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It explains that text files return content, image files return inline images, and how as_image forces image mode. It does not mention error scenarios or max_bytes behavior, but covers the primary behavior adequately.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise paragraphs, front-loaded with the core purpose, followed by behavioral details and cross-references. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a file-read tool with 4 parameters and no output schema, the description covers the main return types (text vs image) and directs users to discovery tools. It lacks details on max_bytes limits and error handling, but for a simple read operation, it is reasonably complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It adds meaning for as_image (forces image output) and hints at path via session_files. However, max_bytes and session_id are not explained; session_id is self-evident but max_bytes behavior (truncation, error?) is missing. The description provides some semantic value but not comprehensive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool reads a file from a session workspace, with specific details about text vs. image handling. It distinguishes itself from siblings like session_write_file and session_files by explicitly mentioning them as alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit guidance on when to use this tool (to read files), and directs the user to session_files for discovering paths and session_artifacts for executed code output. It also explains when to set as_image=True, giving clear usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of disclosing side effects. It explains that python3/node get a stateful REPL with persistence, other languages get a workspace, and a session_id is returned. It does not cover lifecycle aspects like whether sessions need to be explicitly stopped, but it is transparent about the core behavior without contradicting any annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, front-loaded with the primary purpose 'Start a persistent session,' and every clause adds value—explaining language-specific behavior and the return value. There is no redundancy or unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one optional parameter and no output schema, the description adequately covers purpose, language-specific behavior, and return value. It does not mention session_id usage in subsequent calls or error handling, but the sibling tool names (session_run, session_stop) imply the workflow, making it reasonably complete without over-explaining.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has only one optional parameter 'language' with 0% coverage and no enums, so the description compensates by explaining the semantic difference: python3/node yield a stateful REPL, while other languages yield a persistent workspace. This adds meaningful meaning beyond the raw schema, though it does not enumerate all possible language values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Start a persistent session' with a specific verb and resource. It further distinguishes behavior for python3/node versus other languages, making it distinct from sibling tools like execute_code that likely run one-off code. The return value (session_id) is also specified, adding to clarity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context: use this to start a persistent session where variables/imports persist or a workspace directory is created. However, it does not explicitly mention when not to use it or name alternatives like execute_code for one-off execution, so it stops short of full 'when/when-not' guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It transparently explains what the `ready`/`strict` fields mean (resolution facts, not execution-tested) and what they do not mean, which is crucial for interpreting the output. It does not explicitly state that the operation is read-only or describe potential side effects, but the listing nature implies no mutation, and the focus on output semantics is a strong, relevant disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is longer than a single sentence but each part earns its place. It front-loads the purpose, then adds a clarifying paragraph that prevents a common misinterpretation. The structure is logical and the detail is justified by the domain's ambiguity, though it could be trimmed slightly without losing critical information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has an output schema (though not provided here), the description correctly avoids repeating return structures and instead explains the semantic meaning of key fields. It also references sibling tools for complementary information, closing the loop on what the user should look at for language reliability. Nothing essential is missing for a zero-parameter listing tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the description adds no extra meaning to parameter values—there are none to document. Per the baseline, this earns a 4. The description focuses entirely on output interpretation, which is appropriate for a parameterless listing tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a precise verb+resource: 'List execution providers (execution BACKENDS — local subprocess, gVisor-strict, remote) and their machine-readable capabilities.' It immediately clarifies the core concept and sets it apart from language concerns. The explicit contrast with 'not which LANGUAGE' and the references to list_languages/runtimes_status make the tool's unique position unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance: for backend capabilities, not language reliability. It explicitly says per-language reliability is 'a separate axis reported by list_languages/runtimes_status/codecalc doctor as tier' and warns that 'a ready provider says nothing about whether a specific language running through it has ever been execution-tested.' This tells an agent exactly when to pick this tool over its siblings and what it must not use it for.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full responsibility for behavioral disclosure, and it does so well: it states NETWORK: yes, clarifies that non-mutating applies to this machine's runtimes rather than network traffic, and warns that `tier` is orthogonal to update availability. It could add failure or latency behavior, but the core side-effect and network profile is clearly disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose and safety label, then adds only necessary detail about output fields, the parameter format, tier semantics, and the network caveat. Every paragraph earns its place; the non-mutating clarification prevents a likely misinterpretation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete for a one-parameter status tool with no output schema: it explains what the report contains, how to narrow it, the meaning of tier, and the network behavior. The pointer to list_languages for tier definitions closes the remaining semantic gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: `languages` is defined as optional, comma-separated, and illustrated with 'python3,node,rust'. It does not enumerate valid language names, but the example and context make the parameter usable.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Check every language runtime for available updates' and emphasizes '(NON-MUTATING)', which clearly separates status checking from update/install actions. It names concrete outputs — current vs latest, package manager, command — and the NON-MUTATING label distinguishes it from update_runtimes without requiring schema inspection.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It tells the agent how to scope the call with 'Optional `languages` = comma-separated subset' and gives a concrete example. The NON-MUTATING label and 'exact command that would run' imply this is the dry-run/status counterpart to update_runtimes, but it never explicitly names that alternative or states a when-not-to-use condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that all numbers are exact rationals, integers are arbitrary precision, and lists supported operations and math functions. It also notes the difference from plain Python, which is key behavioral context. However, it doesn't mention error handling or edge cases like division by zero, but the core behavior is well covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured. It starts with a clear statement of the tool's purpose, then explains the underlying model, lists supported operations, and provides usage guidance and examples. Every sentence adds value, and the examples are illustrative without being verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (supports many operations and functions) and the lack of annotations or output schema, the description is quite complete. It covers the main behavioral aspects, usage context, and examples. However, it could be more complete by mentioning potential error conditions or the format of the result (e.g., returns a string or number), but overall it provides sufficient information for an agent to use the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has only one parameter 'expr' with no description, and schema description coverage is 0%. The description compensates by explaining what the expression should contain, providing examples, and listing supported operations. This gives the agent a clear understanding of how to construct the expression string, though it doesn't specify syntax details like operator precedence or function call format.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool performs exact arithmetic with rational numbers, distinguishing it from plain Python float arithmetic. It explicitly lists supported operations and gives examples, making the purpose unambiguous and distinct from siblings like calc_stats or evaluate_expression.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says to use this tool BEFORE asserting any computed number, and provides specific use cases (thresholds, ratios, overflows, percentages). It contrasts with plain Python behavior, giving clear guidance on when this tool is appropriate over alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full transparency burden. It discloses background execution, immediate run_id return, standalone process behavior, timeout semantics, the run cap, and the resource_exhausted error path. Even retention is explicitly referenced. This goes beyond surface-level behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is clear and well-organized: it states the core behavior, contrasts with its alternative, explains the timeout contract, and briefly covers admission control and retention. Every sentence contributes meaningful information without excess.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations and no output schema, the description covers the key lifecycle and error behavior very well. It is slightly incomplete for fully understanding resource-limit parameters, and it defers some semantics to execute_code's shape rather than stating them directly, but overall it is sufficiently actionable for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema gives no parameter descriptions (0% coverage), so this description must supply meaning. It helpfully explains timeout semantics and says the request shape matches execute_code minus session_id. However, individual parameters like no_net, max_cpu, provider, and max_memory_mb are not directly described, making the description only moderately informative for many parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Submit code for BACKGROUND execution; returns a run_id immediately.' It clearly distinguishes this from execute_code by highlighting background vs. synchronous execution and the standalone-process nature of a run.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says 'Use this instead of execute_code when you would rather not hold an MCP call open for the whole computation.' It also tells the agent to poll with run_inspect(run_id) and cancel with run_cancel(run_id), covering both selection and follow-up effectively.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description bears the full burden — and it carries it superbly. It discloses return payload, resource ceilings, verdict enum, security enforcement details (seccomp-bpf vs best-effort symbol shim, surfaced via `unenforced`), the 64 KiB/4 MiB caps, spill semantics, and trivially-important edge cases like 'compact never drops `unenforced`'. This goes far beyond a typical tool description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense paragraphs, well organized: return values front-loaded, then a bulleted parameter glossary, then the nuanced spill/session behavior. It is long, but the length is earned — every clause carries a behavioral fact and there is no filler or restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a tool this complex, with no annotations and no output_schema to lean on. It covers return format, verification verdicts, resource limits, network security guarantees and their fallbacks, spill truncation edge cases, session vs session-less differences, and a pointer to SECURITY.md. Nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description must compensate — and it does, explaining the subtle parameters in depth: session_id (workspace/state persistence), max_memory_mb/max_cpu (per-call ceilings), no_net (egress blocking + platform-specific enforcement), container leakage of default vs explicit max_output_kb, and compact (which fields drop vs never drop). Six of the trickiest params get real semantic meaning beyond the schema's title/default.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line states a specific verb and resource — 'Execute `code` in `language` in a sandbox' — and immediately lists concrete return values (stdout, stderr, exit_code, duration_ms, CPU/memory stats, verdict). This is unmistakably distinct from siblings like execute_code_stream and session_run. The first sentence alone fully disambiguates it.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit conditions for session-based use ('With `session_id` set and `max_output_kb` left at its default... output ... is instead SPILLED') and clarifies stateful interpreter persistence. It references alternatives (session_start, session_read_file, execute_code_stream) and explains the session-less fallback behavior. However, it never directly says 'use execute_code_stream instead when X', so the contrast with the closest sibling is left implicit. Clear context, no explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description is highly transparent about behavioral outcomes: it discloses that the local provider does not actually stop runs and that the tool honestly reports this, while provider with 'cancel: true' will terminate the process tree. It also clarifies the idempotent response for already-completed runs, leaving no ambiguity about side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is overly verbose, repeating phrases like 'matching execute_code's own rule' and 'that is reported honestly' in a way that adds redundancy. While every sentence contributes meaning, the same information could be conveyed more succinctly. The structure is logical but could be tightened.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers all relevant contexts: the tool's relation to 'run_submit', idempotency behavior, provider-specific cancel capabilities, and the recommended fallback (setting 'timeout' on submission). It even explains the underlying rationale for the local provider's behavior, making it complete for any user deciding whether to use this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, 'run_id', is well-named and its purpose is obvious from the tool name and description. Although the schema provides no description, the tool description implicitly defines it as the identifier of the background run to cancel. A brief explicit mention would elevate clarity, but it is not necessary.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the tool cancels a background run started with 'run_submit', which is a clear and specific verb-resource pairing. It distinguishes this tool from the sibling 'run_inspect' by focusing on cancellation rather than inspection.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides thorough usage guidance, explaining idempotency (calling on already-finished runs reports state instead of erroring), provider-dependent behavior, and the fallback of using 'timeout' with 'run_submit' for the local provider. It also references 'list_execution_providers' for capability details, making the tool's applicability explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Although there are no annotations, the description details crucial behavioral aspects: the dry-run semantics that minimize changes on disk, the network usage on both dry-run and apply paths, the sudo/gating behavior for apt, and the `elevated` flag per entry. This far exceeds typical disclosure and provides agent-critical safety information.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections separated by blank lines, starting with a bold SAFE BY DEFAULT callout, then the privileges and network details. It is slightly verbose with repeated parenthetical lists and explanations, but each sentence adds necessary context; could be tightened without losing value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (multiple package managers, privileged operations, conditional gating) and the absence of annotations, the description covers all critical aspects: the dry-run safety, the exact commands, the gating mechanism, the network implications, and the per-entry flags. With no output schema, it also explains what entries report (ok: false, elevated flag) but could further specify the success outcome structure, yet this is sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has three parameters with defaults but no descriptions; the tool description compensates by explaining `apply` (dry run vs. apply) and `languages` (comma-separated subset, empty = all). However, `timeout` is not discussed beyond its existence, though its default (600s) and type (integer) are implicitly clear, so the compensation is strong but not complete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool updates language runtimes, enumerates the specific commands that would be executed, and clarifies the dry-run versus apply behavior. It distinguishes itself from sibling tools like runtimes_status or list_languages by focusing on updates, and from other tools via the unique verb 'update'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit guidance on when to pass apply=False as a safe dry run and when to pass apply=True to actually execute changes, including the privileged apt-get path with its gating requirement. It also explains the `languages` parameter for subsetting, though it doesn't mention alternatives among siblings, but the distinctions are clear enough given the tool's purpose.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![codecalc MCP server](https://glama.ai/mcp/servers/The-40-Thieves/codecalc/badges/card.svg)](https://glama.ai/mcp/servers/The-40-Thieves/codecalc)

Score Badge

Copy to your README.md:

[![codecalc MCP server](https://glama.ai/mcp/servers/The-40-Thieves/codecalc/badges/score.svg)](https://glama.ai/mcp/servers/The-40-Thieves/codecalc)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/The-40-Thieves/codecalc'

If you have feedback or need assistance with the MCP directory API, please join our Discord server