Skip to main content
Glama

Server Details

Verifier-grounded AI promotion gates, disposable report cards, and signed PASS/HOLD/BLOCK receipts.

Status
Healthy
Last Tested
Transport
Streamable HTTP
URL
Repository
CarlSR9001/whetstone
GitHub Stars
0

Glama MCP Gateway

Connect through Glama MCP Gateway for full control over tool access and complete visibility into every call.

MCP client
Glama
MCP server

Full call logging

Every tool call is logged with complete inputs and outputs, so you can debug issues and audit what your agents are doing.

Tool access control

Enable or disable individual tools per connector, so you decide what your agents can and cannot do.

Managed credentials

Glama handles OAuth flows, token storage, and automatic rotation, so credentials never expire on your clients.

Usage analytics

See which tools your agents call, how often, and when, so you can understand usage patterns and catch anomalies.

100% free. Your data is private.
Tool DescriptionsB

Average 3.7/5 across 14 of 14 tools scored. Lowest: 2.9/5.

Server CoherenceA
Disambiguation4/5

Most tools have clearly distinct purposes: start/submit pairs are separated by track (open_bench vs report_card), and analysis tools focus on different aspects (audit, health, promotion). Some overlap exists among promotion_gate, inspect_promotion, and bank_health, but their descriptions clarify different workflows. Overall, an agent can reliably select the right tool with careful reading.

Naming Consistency3/5

Tool names follow no single convention: some are verb-first (audit_leakage, inspect_promotion), some are noun-first (bank_health, promotion_gate), and some are compound with verb at the end (open_bench_start, report_card_submit). While all are snake_case and readable, the pattern is not predictable, making it harder to guess tool names for similar operations.

Tool Count5/5

14 tools is well-scoped for a platform covering benchmarking, promotion gates, diagnostics, and patching. Each tool offers a distinct capability without redundancy, aligning with the complexity of the domain. This count is within the ideal 3-15 range and feels neither sparse nor bloated.

Completeness4/5

The tool surface covers the full lifecycle for the two main tracks (start, submit, leaderboard) plus supporting analytics and utilities. Minor gaps exist, such as no way to fetch a single open benchmark receipt or list report-card sessions, but these are non-critical given the disposable session design. Overall, agents can accomplish the intended workflows without dead ends.

Available Tools

14 tools
about_whetstoneAInspect

What this service is: the tool catalog, the tier boundaries, and where the source lives.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It indicates the tool provides information (catalog, tiers, source), implying a read-only operation with no side effects. However, it does not describe output format, scope limits, or any other behavioral details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that directly conveys the tool's purpose. No wasted words; every element contributes to understanding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple informational tool with no parameters and no output schema, the description adequately covers the core content. Minor ambiguities like 'tier boundaries' and 'source' could be more explicit, but overall it is sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, and the schema coverage is 100% (none needed). The description does not need to explain parameters, and the baseline of 4 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's scope: explaining what the service is, including the tool catalog, tier boundaries, and source location. It distinguishes itself from sibling tools by being informational rather than action-oriented, though it lacks an explicit verb like 'provides' or 'describes'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives. The description implies it is for understanding the service, but it does not state when to choose it over other tools or mention any exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

audit_leakageBInspect

Exact declared-exposure audit over your exam rows: row identity, behavioral fingerprints for graph-DSL expressions, text-similarity review flags, and a clean exam export. Full example: GET /api/examples key 'leakage'.

ParametersJSON Schema
NameRequiredDescriptionDefault
examYesExam rows. Each row needs item_id (or id) plus prompt/content/input/task/question/expression.
exposureNoDeclared exposure rows carrying identity/content fields and an optional source or path.
fingerprint_max_nNo
similarity_thresholdNo
enable_text_similarityNo
enable_behavioral_fingerprintNo
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the burden. It discloses outputs and mentions computational aspects (behavioral fingerprints, text-similarity) but does not indicate whether the operation is read-only, any side effects, permissions required, or behavior on edge cases. It also lacks any details about the 'clean exam export' format. This is insufficient for a tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact: a single purpose statement with a list of outputs and an example reference. All information is relevant and front-loaded. There is no redundant text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 6 parameters, no output schema, and no annotations. The description provides a high-level overview but lacks details on return values, interpretation of results, and parameter constraints. The example reference is helpful but not a substitute for explaining the audit process. The description is too sparse to fully guide an agent in invoking and interpreting the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers only 33% of parameters (exam and exposure). The description adds context by naming 'behavioral fingerprints for graph-DSL expressions' and 'text-similarity review flags', which maps to enable_behavioral_fingerprint and enable_text_similarity. However, it does not explain fingerprint_max_n or similarity_threshold semantics, nor does it clarify the exact fields expected in exam/exposure rows beyond what the schema states. Partial compensation for low schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Exact declared-exposure audit over your exam rows') with a clear resource ('exam rows') and lists the outputs (row identity, behavioral fingerprints, text-similarity flags, clean exam export). This distinguishes it from siblings like counterexample_hunt or memory_relevance and makes the tool's purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit guidance on when to use this tool vs alternatives. The description mentions an example endpoint but does not state conditions for use, prerequisites, or exclusions. Sibling tools suggest a variety of audit/analysis functions, but no differentiation is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

bank_healthBInspect

Item-lifecycle diagnostics over your grading history: discriminators, saturated and flaky items, frontier gaps. Full example: GET /api/examples key 'health'.

ParametersJSON Schema
NameRequiredDescriptionDefault
itemsNoOptional item definitions.
historyYesObserved item/system outcomes.
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It does not mention whether the operation is read-only, side-effect-free, or what the response format looks like. The 'Full example' reference is ambiguous and doesn't disclose behavioral traits. It does add value by naming diagnostic categories but stops short of transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is brief and front-loaded with the core purpose. However, the 'Full example: GET /api/examples key 'health'' clause is cryptic and could be clearer, though it adds a useful pointer. Overall, it earns its place with minimal waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has no output schema, so the description should explain return structure or behavior. It lists expected diagnostic categories but not how they are presented (e.g., metrics, flags, counts). The example reference is vague and doesn't compensate for missing output details. For a one-required-param tool, it's serviceable but has gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already provides descriptions for both parameters (items and history), achieving 100% coverage. The description's mention of 'grading history' aligns with the required `history` parameter but does not add additional syntax, constraints, or meaning beyond the schema. Baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as performing item-lifecycle diagnostics over grading history, listing specific diagnostic outputs (discriminators, saturated/flaky items, frontier gaps), which distinguishes it from sibling tools focused on audits, promotions, or submissions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool vs alternatives is provided. It only states what it does, leaving the agent to infer context from the name and sibling tools. No exclusions or alternative recommendations are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

counterexample_huntAInspect

Bounded simulated-annealing search for a graph counterexample inside a DSL predicate class, with an exact certificate when found. CPU-bounded and strictly rate-limited. Full example: GET /api/examples key 'counterexample'.

ParametersJSON Schema
NameRequiredDescriptionDefault
nsNoGraph sizes searched.
seedNo
stepsNo
restartsNo
expressionYesGraph predicate in the Whetstone DSL, for example: is_connected and is_triangle_free and not is_bipartite
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description must disclose behavior. It adds useful constraints: CPU-bounded, strictly rate-limited, and produces an exact certificate when found. However, it does not describe failure modes, return format, or side effects, leaving gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with purpose, then constraints, then example pointer. No redundant wording; every sentence contributes.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has no output schema and five parameters, yet the description gives only a generic 'exact certificate' mention and defers to a full example. It covers constraints and high-level behavior but stops short of explaining return values or parameter interactions. Adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is only 40%; only ns and expression have descriptions. The description does not elaborate on seed, steps, or restarts beyond the algorithm hint 'simulated-annealing', so the agent must infer their roles. The expression example is helpful but not sufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('search for a graph counterexample') with a clear method (bounded simulated-annealing) and resource (DSL predicate class). This distinguishes it from sibling tools like bank_health or open_bench_start, which address unrelated domains.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use or alternative tool guidance is provided. The phrase 'Full example: GET /api/examples key counterexample' implies usage but does not explain when to choose this over siblings or any prerequisites. Usage is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

inspect_promotionBInspect

Quarantine declared exposure, compare paired baseline/candidate outcomes on the clean remainder, and issue a promotion receipt. Bring your own exam rows, exposure records, and per-item results. Full example: GET /api/examples key 'inspector'.

ParametersJSON Schema
NameRequiredDescriptionDefault
examYesExam rows. Each row needs item_id (or id) plus prompt/content/input/task/question/expression.
policyNoExplicit promotion policy. Omitted fields use the documented defaults.
domainsNoOptional item_id -> domain label mapping.
baselineYesitem_id -> boolean pass/fail result for the baseline system.
exposureNoDeclared exposure rows carrying identity/content fields and an optional source or path.
candidateYesitem_id -> boolean pass/fail result for the candidate system.
baseline_nameNobaseline
candidate_nameNocandidate
retained_probeNoOptional retained-capability result checked alongside the paired cohort.
fingerprint_max_nNo
similarity_thresholdNo
enable_text_similarityNo
enable_behavioral_fingerprintNo
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the main behavioral pipeline: quarantine declared exposure, compare on the clean remainder, and issue a receipt. It does not mention side effects (e.g., whether data is mutated), output format, failure conditions, or statistical methods. With no annotations provided, the description carries the burden but only partially covers behavioral expectations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense sentence followed by a brief example pointer. It is front-loaded with the core actions and avoids fluff. Every sentence earns its place, making it appropriately concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 13 parameters, nested objects, no output schema, and no annotations. The description gives a high-level process but omits critical details such as what the promotion receipt contains, how the policy parameters are used, and what happens if exposure is not declared. The example pointer helps but does not provide enough context for correct invocation in complex scenarios.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is only 54%, leaving several parameters (e.g., baseline_name, fingerprint_max_n, similarity_threshold, enable_text_similarity) without descriptions. The tool description only mentions 'exam rows, exposure records, and per-item results,' mapping to exam, exposure, baseline, and candidate, but does not compensate for the uncovered parameters. The pointer to a full example ('GET /api/examples key inspector') is helpful but does not directly explain parameter semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: quarantine exposure, compare baseline/candidate outcomes on the clean remainder, and issue a promotion receipt. The specific verb phrases ('quarantine,' 'compare,' 'issue') and the resource ('baseline/candidate outcomes') make the purpose clear. It does not explicitly differentiate from sibling 'promotion_gate,' but the name and description imply an inspection/audit role.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage: 'Bring your own exam rows, exposure records, and per-item results' suggests a self-service analysis tool where the user supplies data. However, there is no explicit guidance on when to use this tool versus siblings like 'promotion_gate' or 'audit_leakage,' nor any stated exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

memory_relevanceCInspect

Compare query-free salience against objective-conditioned relevance for a set of memories under a token budget. Full example: GET /api/examples key 'memory'.

ParametersJSON Schema
NameRequiredDescriptionDefault
memoriesYesMemories to rank.
objectiveYes
current_stepNo
token_budgetNo
question_kindNogeneric
context_entitiesNoEntities already active in context.
objective_entitiesNoOptional explicit entities when the objective text is not self-describing.
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose behavioral traits. It mentions comparing under a token budget, implying resource constraints, but does not state whether the operation is read-only, what outputs to expect, or any side effects. The transparency is insufficient for a tool without annotation support.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: one sentence stating the core purpose plus a pointer to an example. It is front-loaded and efficient, though the example pointer ('GET /api/examples key memory') could be clearer. Every word adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 7 parameters, no output schema, and no annotations, the description is under-specified. It provides only a high-level purpose and an example pointer, lacking details on parameter interactions, return values, or operational context. This is inadequate for a tool of this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds semantic context by explaining that 'objective' conditions relevance and 'token_budget' limits the comparison, which helps interpret these parameters. However, with only 43% schema coverage, several parameters like current_step, question_kind, and context_entities remain unexplained. The description partially compensates but leaves gaps.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: comparing query-free salience against objective-conditioned relevance for memories. It identifies the resource (memories) and the operation (compare), which is specific and distinct from the sibling tools about benchmarks and audits. However, it does not explicitly differentiate from siblings, though the focus on memory ranking is unique.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It merely hints at an example location but lacks context such as prerequisites, typical use cases, or when not to use it. The schema description adds purpose but not usage direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

open_bench_leaderboardAInspect

TIER 2: list the self-attested public Open Promotion Bench receipts. Entries contain manifests, verdicts, item-level transitions, and commitments but never task contents or submitted answers.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses that the tool lists public, self-attested receipts and never includes task contents or submitted answers, implying a read-only, privacy-conscious operation. It does not mention pagination or other details, but the core behavioral traits are covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that communicates the action, scope, and content without any unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter list tool with no output schema or annotations, the description covers the core purpose, scope, included fields, and exclusions. It omits optional details like pagination ordering, but remains largely complete for its simplicity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has zero parameters, so the baseline is 4. The description correctly implies no inputs are required, and there is nothing more to explain.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists 'self-attested public Open Promotion Bench receipts' with a specific verb and resource. It differentiates from sibling tools by focusing on listing receipts rather than starting/submitting/inspecting promotions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied: it lists receipts. However, there is no explicit guidance on when to use this tool versus alternatives like inspect_promotion or open_bench_submit, nor any mention of exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

open_bench_startAInspect

TIER 2: start a one-shot Open Promotion Bench session. Returns six fresh virtual-repository scope-integrity tasks. Run a baseline and candidate independently on the same cohort, then submit both answer maps with open_bench_submit. This is an open, procedural, self-attested track rather than a private-bank credential.

ParametersJSON Schema
NameRequiredDescriptionDefault
challengeNoCaller nonce bound into the signed receipt for replay detection.
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the behavioral burden. It discloses that the session is one-shot, returns six tasks, and is self-attested, but does not mention session state management, whether repeated starts are allowed, or other side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, front-loaded with the purpose, and every sentence contributes to explaining the tool's function and workflow. The 'TIER 2' prefix is a minor label but does not distract.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and no annotations, the description provides the essential context: what it returns (six tasks), the one-shot nature, and the required follow-up submission. It could elaborate on task format or the role of the challenge parameter, but is sufficient for a simple one-parameter tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers 100% of parameters (challenge) with a clear description as a nonce for replay detection. The tool description adds no additional parameter semantics beyond what the schema already provides, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool starts an Open Promotion Bench session and returns six fresh tasks, using a specific verb and resource. It distinguishes itself from open_bench_submit and the private-bank credential track.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains the workflow: run baseline and candidate independently on the same cohort, then submit answer maps with open_bench_submit. It also provides an implicit exclusion by noting this is an open procedural track rather than a private-bank credential.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

open_bench_submitAInspect

TIER 2: grade paired baseline and candidate patches, count gains/regressions/ties, and issue PASS/HOLD/BLOCK. Set publish=true plus attestation=true to append only the safe manifests and sanitized receipt to the public board; tasks and answers are never persisted.

ParametersJSON Schema
NameRequiredDescriptionDefault
publishNo
session_idYes
attestationNo
baseline_answersYes
baseline_manifestYes
candidate_answersYes
candidate_manifestYes
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses key behaviors: grading, counting, issuing verdicts, and the conditional publish behavior with attestation. It also explicitly states a data persistence guarantee ('tasks and answers are never persisted'). While it does not address all edge cases (e.g., behavior if attestation is false), it offers substantial transparency for an unannotated tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is exactly two sentences, front-loaded with the primary action, and every clause adds useful information. It avoids redundancy and is well-structured for AI consumption.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 params, nested objects, no output schema, no annotations), the description covers the core purpose and persistence behavior but omits details about the return format, how the verdicts are communicated, and what constitutes 'safe manifests.' It is adequate but leaves gaps that an agent might need to infer.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, and the description only explains publish and attestation booleans. It does not explain the structure or requirements for baseline_answers, candidate_answers, session_id, or the manifest objects beyond what their names imply. This is a significant gap for a tool with 7 parameters, especially since the answers objects are opaque in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'grade paired baseline and candidate patches, count gains/regressions/ties, and issue PASS/HOLD/BLOCK.' This specific verb-resource pairing ('grade patches') distinguishes it from siblings like 'open_bench_start' (starting a run) and 'open_bench_leaderboard' (viewing results). The mention of publish behavior adds further distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides concrete usage context: it explains when to set publish=true and attestation=true for appending to the public board, and clarifies that tasks and answers are never persisted. However, it does not explicitly state alternatives or when NOT to use this tool, though the context of siblings implies this is for final grading/submission.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

promotion_gateAInspect

PASS, HOLD, or BLOCK from paired per-item results: gains, regressions, exact McNemar p-value, per-domain breakdown. Full example: GET /api/examples key 'gate'.

ParametersJSON Schema
NameRequiredDescriptionDefault
policyNoExplicit promotion policy. Omitted fields use the documented defaults.
domainsNoOptional item_id -> domain label mapping.
baselineYesitem_id -> boolean pass/fail result for the baseline system.
candidateYesitem_id -> boolean pass/fail result for the candidate system.
baseline_nameNobaseline
candidate_nameNocandidate
retained_probeNoOptional retained-capability result checked alongside the paired cohort.
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the output (decision, p-value, breakdown) but does not disclose whether the tool has side effects (e.g., writes anything), how it handles mismatched item sets, what happens on insufficient data, or any performance implications. For a statistical computation tool, this missing transparency is a notable gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very concise: two sentences, front-loaded with the primary action and output. The reference to an example endpoint adds practical value without redundancy. No filler or wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description gives a high-level overview of outputs (gains, regressions, p-value, domain breakdown) but does not fully suffice for a tool with 7 parameters, nested objects, and no output schema. It lacks explanation of the policy object semantics, the retained probe, or how the decision is derived. The example hint helps but does not replace a fuller description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 71%, so the schema already documents many parameters (baseline, candidate, policy, etc.). The description only vaguely references 'paired per-item results' which maps to the input schema. It does not add meaning beyond the schema descriptions, so a baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the tool's function: it processes paired per-item results and returns a decision ('PASS, HOLD, or BLOCK') with supporting statistics (gains, regressions, p-value, domain breakdown). This clearly distinguishes it from siblings like 'inspect_promotion' which likely focuses on deeper inspection rather than a gate decision.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is used when you have paired baseline/candidate item-level pass/fail results and need a statistical gate decision. However, it does not explicitly state when to choose this tool over alternatives, nor does it provide exclusions (e.g., when not to use it, or when to prefer 'inspect_promotion' for detailed analysis).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

replay_traceAInspect

Turn reasoning-emulator control events into checkpoints, rewinds, notes, and a timeline. Full example: GET /api/examples key 'replay'.

ParametersJSON Schema
NameRequiredDescriptionDefault
notesNoOptional analyst notes.
eventsYesOrdered reasoning-emulator events.
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It explains the transformation but does not mention side effects, read-only status, required permissions, or how the output is delivered. The provided example reference is helpful but insufficient to understand runtime behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, with the first stating the core purpose and the second providing an example access point. No redundancy or filler; every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a complex event schema and no output schema, yet the description only offers a high-level transformation statement and an example key. It does not explain the output structure, expected input constraints beyond schema, or potential failure modes, leaving notable gaps for an agent to use it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds context that events are 'reasoning-emulator control events', but the schema already details event kinds and notes. It does not provide additional parameter-specific semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Turn') and resource ('reasoning-emulator control events') to define outputs (checkpoints, rewinds, notes, timeline). This clearly distinguishes it from sibling tools that focus on benchmarks, audits, or memory.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage is for replaying control events into a timeline, but it does not explicitly state when to use it vs alternatives or when not to use it. Sibling tools are unrelated, so some implicit guidance exists, but it lacks direct exclusions or comparative context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

report_card_startAInspect

TIER 1: start a disposable report-card session. Returns exam items (graph-repair prompts minted from the repository's public frontier) for THIS agent to answer. Answer every item, then call report_card_submit exactly once. Sessions are one-shot, expire in 15 minutes, and are strictly rate-limited. This demonstrates the promotion-gate mechanism on disposable items; it is not a private-bank credential.

ParametersJSON Schema
NameRequiredDescriptionDefault
challengeNoCaller nonce bound into the signed receipt for replay detection.
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral transparency burden. It discloses disposable/one-shot nature, 15-minute expiry, strict rate limiting, and explicitly clarifies it is not a private-bank credential. This is solid context, though it doesn't cover failure modes or concurrent session behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact at three sentences and front-loads the core purpose. The 'TIER 1' prefix adds a small amount of context, and the final sentence about demonstrating the promotion-gate mechanism, while useful, is slightly tangential. Overall, it is appropriately sized with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description communicates the return value (exam items as graph-repair prompts), the lifecycle (one-shot, 15-minute expiry), and the expected workflow. Without an output schema, it gives enough context for an agent to invoke and follow up. It does not specify the exact structure of returned exam items, but that is not necessary for a session-starting tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers the single optional parameter 'challenge' with a description ('Caller nonce bound into the signed receipt for replay detection') and 100% schema coverage. The tool description adds no additional parameter semantics, so the baseline of 3 applies since the schema fully documents the parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'start a disposable report-card session' and specifies that it returns exam items for the agent to answer. It distinguishes itself from siblings by naming the follow-up tool (report_card_submit) and framing the session's purpose within the promotion-gate mechanism.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives concrete usage context: answer every item, then call report_card_submit exactly once. It also notes the session is one-shot, expires in 15 minutes, and is rate-limited. However, it does not explicitly mention when to prefer this tool over alternatives or when not to use it, though the workflow is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

report_card_submitAInspect

TIER 1: submit answers for a report-card session and receive the graded report (per-item verdicts, per-domain totals, SHA-256 commitments). Grading is by checker spec: verified strict refinements are reported separately, and promotion grade requires at least 5% clean-support retention. No answer key exists. The session is destroyed by this call.

ParametersJSON Schema
NameRequiredDescriptionDefault
answersYesitem_id -> answer (a DSL predicate, or the JSON reply the prompt asked for)
session_idYes
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden and discloses critical behavioral traits: the session is destroyed by this call, grading follows a checker spec, strict refinements are reported separately, promotion requires 5% clean-support retention, and no answer key exists. This is rich, non-obvious context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences with the action and output front-loaded. Every sentence provides distinct value: action/result, grading details, destruction warning. No redundancy or fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description explains the return value structure ('per-item verdicts, per-domain totals, SHA-256 commitments'), grading behavior, and destructive nature. It lacks explicit prerequisites (e.g., must have an active session from report_card_start) and error conditions, but is otherwise complete given no output schema or annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 50% (answers has a description, session_id does not). The description does not add parameter-level details, so it does not compensate for the gap. Baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action: 'submit answers for a report-card session and receive the graded report.' It names the specific resource (report-card session) and output (graded report), distinguishing it from siblings like report_card_start.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage after a report-card session has been started, but it does not explicitly say 'Use after report_card_start' or provide exclusions versus other submission tools like open_bench_submit. Usage guidance is only implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

safe_patchAInspect

Apply a section-scoped Markdown patch under conservation checks (untouched sections stay byte-identical; protected tokens preserved). Full example: GET /api/examples key 'safepatch'.

ParametersJSON Schema
NameRequiredDescriptionDefault
reasonNo
documentYesComplete Markdown document to patch.
operationsYes
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so well by disclosing core behavior: section-scoping, conservation checks, byte-identical untouched sections, and token preservation. It stops short of explaining return values or error conditions, but the constraints that affect caller expectations are clearly stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the primary purpose, followed by a concise reference to an example. Every phrase adds value and there is no redundant jargon.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has moderate complexity (multi-operation array, token protection) and no output schema or annotations. The description clarifies the patch mechanism but does not explain what the tool returns, how errors are surfaced, or how the example endpoint is used. This leaves some contextual gaps for an agent to fully understand the invocation contract.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has only 33% parameter coverage (document is described; reason and operations are not). The description adds meaning for target_heading ('section-scoped') and allow_token_changes ('protected tokens preserved'), but does not elaborate on 'reason', 'find', or 'replace'. It partially compensates for the low schema coverage but leaves gaps.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Apply') and clearly identifies the resource ('section-scoped Markdown patch'). It also enumerates key behavioral guarantees ('untouched sections stay byte-identical; protected tokens preserved'), making it unmistakable what the tool does and distinguishing it from any sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use the tool: for patching Markdown documents with conservation guarantees. It also provides a pointer to a full example ('GET /api/examples key 'safepatch''), which is a form of usage guidance. No explicit alternatives are mentioned, but none are needed given the unrelated sibling list.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Discussions

No comments yet. Be the first to start the discussion!

Related MCP Servers

  • A
    license
    A
    quality
    A
    maintenance
    Post-quantum, tamper-evident receipts for consequential agent actions. Provides tools for auditing, gating decisions, and egress classification with quantum-hardened security.
    7
    Apache 2.0
  • A
    license
    -
    quality
    D
    maintenance
    Provides cryptographic governance receipts for AI agents, enabling pre-execution evaluation and signed verdicts (EXECUTE/BLOCK/REVIEW/SHADOW) with offline-verifiable audit trails.
    MIT
  • A
    license
    -
    quality
    B
    maintenance
    Verifiable action receipts for AI agents — agents sign claims locally, an independent witness countersigns and timestamps, anyone can verify offline.
    15
    MIT
  • A
    license
    A
    quality
    A
    maintenance
    The accountability layer for AI agents — a named human's signed yes before an agent does anything irreversible (payment, record change, deploy), then an offline-verifiable Trust Receipt. Apache-2.0, formally verified.
    17
    810
    Apache 2.0

View all MCP Servers

Try in Browser

Your Connectors

Sign in to create a connector for this server.