Skip to main content
Glama

Xzar MCP server

Crash-test your AI agent on real professional decisions. Xzar gives your agent real cases, cut at the moment of decision, and grades its answer against what the professional actually decided. Most cases were decided after current models were trained, so they can't have been memorized; fresh ones are added monthly.

Live today:

  • Tariff classification: real U.S. classification decisions, graded exactly at 2, 4, 6, 8 and 10 digits, with a consistency check (the same case asked twice). Testing a tariff classifier? See https://xzar.si/scorecard

  • Patent examination: first action, response and appeal.

More fields are on the way; ask for yours at https://xzar.si.

Setup

  1. Get an API key (with $5 of free credit) at https://xzar.si/signup.html

  2. Add the server to your MCP client:

{
  "mcpServers": {
    "xzar": {
      "command": "uvx",
      "args": ["xzar-mcp"],
      "env": { "XZAR_API_KEY": "your-key" }
    }
  }
}

Related MCP server: agentviz

Hosted connector (no install)

In Claude or any MCP client that supports remote servers: add https://api.xzar.si/mcp and sign in with your Xzar API key.

Claude plugin

The plugin/ folder is the Xzar plugin for Claude. It bundles the hosted connector with two skills: crash-test, which runs a task end to end and explains the score, and xzar-request, for custom tests, pilots, quotes and training access. In Claude Code:

/plugin install xzar --marketplace xzar-si/xzar-mcp

Tools

Tool

What it does

start_task

Start a task: the case, the instructions and the tool budget

search_library

Search the case's reference library

read_publication

Read one library item

search_rules

Search the professional rulebook for the task

read_rule

Read one rulebook section

submit_answer

Submit the decision; returns the exact score and the grade receipt

request_service

Ask for a custom test for your field, a pilot, a quote, or training access; get a reference and next steps

get_scorecard

Your agent's scorecard across tasks

get_usage

Credit balance and usage

Answer keys never leave Xzar's servers. This package holds no tasks; it calls the Xzar API with your key.

Pricing

Prepaid credits, charged per task. See https://xzar.si/docs.html. Pilots and ongoing testing: https://xzar.si.

Available Tools

9 tools
get_scorecardGet your scorecardA
Read-onlyIdempotent
Inspect

Get your agent's scorecard across all finished episodes: scores by skill and the failure types it hits most often. Read-only; no credit used.

NameRequiredDescriptionDefault

No parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive, and closed-world, so the safety profile is covered. The description still adds value beyond them by disclosing the cost signal ('no credit used') and the scope limitation to finished episodes — meaningful extras for an agent deciding whether to call it.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded with the verb and resource, followed by the cost caveat. Every clause carries information an agent would use.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no input params and no output schema, the description must convey what comes back, and it does: per-skill scores and dominant failure types. It stops short of describing the structure or granularity of those scores, but for a read-only, zero-argument tool this is close to sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is nothing for the description to disambiguate; the baseline for a parameterless tool applies. No parameter-related gaps exist.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Get') and resource ('your agent's scorecard') and enumerates the content returned: scores by skill plus most-frequent failure types across finished episodes. That scope ('all finished episodes') also separates it from get_usage and the task-oriented siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'across all finished episodes' implies the right moment to call it, but there is no explicit when-to-use guidance, no mention of prerequisites such as having completed episodes, and no named alternative (e.g. get_usage) to route against.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_usageGet your credit balance and usageA
Read-onlyIdempotent
Inspect

Get your credit balance and your most recent charges and top-ups. Read-only; no credit used.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, non-destructive, so the safety profile is covered; the description still adds a non-obvious fact beyond them — that the call consumes no credits, which matters for an agent operating under a budget. It does not describe units, currency, or how many 'recent' entries are returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, zero filler, with the capability stated first and the no-cost guarantee second — exactly the information an agent needs, front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter, read-only tool with no output schema, the description adequately previews the return content (balance, recent charges, top-ups). It stops short of stating response shape, currency, or recency window, which an agent might want given there is no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, which is the baseline-4 case; there is nothing for the description to disambiguate. Schema coverage is 100% and the empty object confirms no arguments are needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (get credit balance) and scopes the result set (most recent charges and top-ups), which is more precise than the title alone. No sibling tool covers billing/usage, so no differentiation is needed, but it also doesn't explicitly claim that territory against the rest of the catalog.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Read-only; no credit used' implicitly tells the agent this is always safe to call, but there is no explicit when-to-use or when-not guidance, nor any mention of alternatives (none of the siblings overlap, so the gap is small). Usage is inferred rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_publicationRead a library publicationA
Read-onlyIdempotent
Inspect

Read the full text of one publication from the case's library, by the id search_library returned. Use it to check whether a reference really discloses what the claim requires. Counts against the tool budget.

NameRequiredDescriptionDefault
episode_idYesThe episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e
publicationYesPublication id from search_library results, e.g. US20190012345A1

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the read-only, idempotent, non-destructive, closed-world profile, so the description doesn't need to repeat safety facts. It adds genuinely new behavioral information by warning that the call 'counts against the tool budget', which is not encoded anywhere in the annotations. It stops short of noting result size or truncation behavior for full texts.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with what the tool returns, followed by why to use it and the budget cost. No filler and nothing buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter read tool with full annotation coverage, the description covers purpose, input provenance, motivation, and cost. It leaves the return value underspecified beyond 'full text' (length, truncation, not-found behavior), which is a minor gap given there is no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with both episode_id and publication documented with format examples, so the baseline is 3. The description only restates the publication id's provenance from search_library — which the schema already says — and adds nothing about episode_id or id format.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (read) and resource (full text of one publication), and scopes it to 'one publication from the case's library', which cleanly separates it from the sibling search_library. An agent can tell exactly what this retrieves without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit usage context: obtain the id from search_library, then use it to verify whether a reference actually discloses what a claim requires. It names the sibling that produces the input but doesn't spell out exclusions (e.g., don't call it just to preview a title), so it stops short of a full when/when-not rubric.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_ruleRead a rulebook sectionA
Read-onlyIdempotent
Inspect

Read one rulebook section by its number, as returned by search_rules. Returns the section text.

NameRequiredDescriptionDefault
sectionYesSection number, e.g. 2143.01
episode_idYesThe episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, non-destructive, and closed-world, so the safety profile is fully covered. The description adds that the output is the section text (useful since there is no output schema) and where section numbers originate, but says nothing about error behavior or missing sections. Modest added value.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded with the core action, with zero redundant text. Every clause adds information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter read tool with rich annotations, the definition covers the action, the identifier source, and the return value. A brief note on what happens when the section does not exist would complete it, but an agent has enough to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with both section and episode_id documented including an example format. The description reinforces that 'section' is a section number sourced from search_rules, but adds no format or eligibility detail beyond the schema. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: read one rulebook section by number, returning the section text. It also ties the identifier source to search_rules, which helps place it among siblings. It does not explicitly distinguish itself from read_publication, so differentiation is only partial.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"As returned by search_rules" gives a clear usage context: obtain a section number via search_rules, then read it here. No explicit when-not-to-use or failure conditions are given, but the workflow routing is unambiguous enough for an agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

request_serviceRequest a custom test, pilot, quote or training accessAInspect

Ask Xzar for something not available self-serve: a custom test for a field, a pilot, a quote, or training access. Returns a reference, next steps and pricing; Xzar replies by email within one business day. Does not use credit.

NameRequiredDescriptionDefault
nameNoYour name
typeYesWhat you are asking for
emailYesWork email for the reply
fieldNoThe professional field, e.g. insurance claims
notesNoAnything else Xzar should know
volumeNoRough number of cases or tasks you need
companyNoYour company or team
deadlineNoWhen you need it, e.g. end of Q1
decisionNoThe decision your AI makes, e.g. approve or deny a claim

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare a non-read-only, non-destructive, non-idempotent, closed-world tool. The description adds substantial context: a human replies by email within one business day, a reference and pricing are returned, and no credit is consumed. Minor gap: it does not say whether repeated calls create duplicate requests, which matters given idempotentHint is false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the purpose and followed by response expectations and cost. No sentence is redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an intake/form-submission tool, the definition covers purpose, response contents, turnaround time, and cost – exactly what an agent needs before calling. The parameters are fully documented in the schema and no output schema exists, so nothing material is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already explains all nine parameters, including the enum and email reply address. The description adds only a restatement of the request types, not field-level guidance like what makes a useful `field`, `volume` or `decision` value. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource, 'Ask Xzar for something not available self-serve,' and enumerates the request types (custom test, pilot, quote, training access) that match the `type` enum. It is clearly distinguishable from self-serve siblings such as start_task or get_scorecard.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Ask Xzar for something not available self-serve' gives the condition that selects this tool and implicitly excludes anything covered by self-serve flows. It does not name which sibling to use instead for self-serve requests, leaving that inference to the agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_librarySearch the case's reference libraryA
Read-onlyIdempotent
Inspect

Search the case's reference library (patent cases: prior-art publications dated before the filing date). Use it to find evidence before deciding. Returns matching publication ids with titles and snippets; read one with read_publication. Counts against the tool budget.

NameRequiredDescriptionDefault
queryYesKeywords or a short phrase describing the technology, e.g. 'battery cell cooling plate'
episode_idYesThe episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive, and closed-world behavior. The description adds the non-obvious cost signal ('Counts against the tool budget') and describes the shape of results (publication ids, titles, snippets) even though there is no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Compact and front-loaded: domain definition, usage intent, return shape and next step, and cost warning. 'Use it to find evidence before deciding' is slightly generic but still orients the agent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, yet the description explains return values and next steps, and covers the budget cost. It does not mention result limits or pagination, which would complete the picture for repeated searching.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters are already well documented in the schema. The description adds no extra syntax, constraints, or interpretation beyond what the schema provides, which is the baseline when the schema does the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource and immediately clarifies what the reference library contains for patent cases (prior-art publications dated before the filing date). It also distinguishes itself from the read_publication sibling by name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly frames the tool as an evidence-gathering step before deciding, and routes the agent to read_publication for reading a hit. It does not clarify its boundary versus search_rules, which is the main adjacent alternative.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_rulesSearch the rulebookA
Read-onlyIdempotent
Inspect

Search the professional rulebook for the case (patent cases: the examination manual examiners follow). Use it to find the legal test that applies. Returns section numbers and titles; read one with read_rule.

NameRequiredDescriptionDefault
queryYesThe rule or test you need, e.g. 'obviousness combining references'
episode_idYesThe episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover safety (readOnlyHint, idempotentHint, closed-world), so the description's added value is in disclosing that results are shallow (section numbers and titles only), which prevents an agent from assuming it can answer directly from search results. It omits result-count limits and ranking behavior, but that is a minor gap given the annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, front-loaded with the core action, with the domain clarification and the output/next-step facts each earning their place. No redundant restatement of the name or annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description compensates by stating what is returned (section numbers and titles) and how to continue (read_rule). It could be stronger by noting whether results are limited or ranked, but an agent has everything needed to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters (including the natural-language query example and the episode_id provenance) are already documented in the schema. The description adds only the implicit framing that the rulebook is case-scoped, which is marginal beyond what structured data provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Search the professional rulebook for the case") and even disambiguates the domain with the parenthetical about the patent examination manual. It also clarifies the output form (section numbers and titles, not full text) and names the sibling read_rule as the follow-up, so an agent can tell it apart from search_library/read_publication.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Use it to find the legal test that applies" gives a clear usage condition, and "read one with read_rule" directs the agent to the correct next step instead of expecting full text here. It does not explicitly contrast with the sibling search_library, so the alternative-selection guidance is incomplete rather than absent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

start_taskStart an evaluation taskAInspect

Start a new evaluation task (one episode) and get the case. Call this first. Returns the episode id (needed by every other episode tool), the case instructions, the answer format and the tool budget. Uses one task of credit; a start that fails on Xzar's side is refunded.

NameRequiredDescriptionDefault
task_idNoA specific task id to retry; omit to get the next unserved case
tasksetNoWhich kind of case to work. Default patents-first-action. tariffs-classification asks for the tariff code of a real product.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations by disclosing cost behavior ('Uses one task of credit') and the refund edge case on server-side failure, plus the critical dependency that every other episode tool needs the returned episode id. This is exactly the kind of non-obvious operational detail annotations cannot convey, and it is consistent with idempotentHint=false and readOnlyHint=false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, front-loaded with the action and the 'call this first' directive, and no sentence is padding. The return-value list is slightly dense but each item is used by downstream calls, so it earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly takes on the burden of enumerating the returned fields (episode id, case instructions, answer format, tool budget) and flags the credit/refund behavior. Nothing needed to invoke it correctly or interpret its result is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the taskset enum is self-documenting, so the schema already carries the parameter semantics. The description adds no parameter-level detail beyond what is in the schema, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start a new evaluation task (one episode) and get the case') and immediately distinguishes it in the sibling set with 'Call this first'. Enumerates what it returns (episode id, case instructions, answer format, tool budget), so an agent knows this is the entry point rather than a sibling like submit_answer or search_library.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Call this first' gives an explicit ordering constraint relative to the other episode tools, which is the key usage decision here. It does not state when-not to call it (e.g., reusing an existing episode), but the schema's task_id 'retry' note covers the re-entry path.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_answerSubmit the agent's answer for gradingAInspect

Submit the final decision for the episode, as a JSON object in the format start_task gave (tariff cases: {"code": "8471300100"}, 10 digits, 8 accepted). One answer per episode; it ends the episode. Returns the exact score against what the professional actually decided, plus a grade receipt. The answer key itself is never shown.

NameRequiredDescriptionDefault
answerYesThe decision, in the answer format from start_task
episode_idYesThe episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds meaningful context beyond the annotations: only one answer is allowed per episode, the episode terminates on submission, and the response contains the exact score and a grade receipt. It also discloses that the answer key is never shown. Annotations already cover read-only and idempotency traits, so this is strong but not exhaustive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action and format, followed by the terminal consequence and return behavior. Every sentence adds useful information: the format example, one-answer constraint, episode termination, return contents, and answer-key visibility.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a terminal submission tool with no output schema, the description explains the expected answer format, the one-shot nature of the call, the episode-ending consequence, and what the response will contain. Nothing critical for correct invocation appears missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so episode_id is fully documented in the schema. The description adds useful meaning for the answer parameter by specifying the JSON object format, a tariff example, and digit constraints (10 digits, 8 accepted) that go beyond the schema's generic reference to start_task.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: submit the final decision for the episode. It distinguishes this terminal grading action from siblings like get_scorecard by noting that it ends the episode and returns the exact score plus a receipt.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly establishes when to use it: submit the final decision once per episode, and doing so ends the episode. It references start_task for the required answer format, though it does not explicitly name alternatives or exclusions among siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 7 tool updatesv0.1.3
    • Changedread_publication2 fields changed
      • addedInput schema / properties / episode_id / description
        Added value: +"The episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e"
      • addedInput schema / properties / publication / description
        Added value: +"Publication id from search_library results, e.g. US20190012345A1"
    • Changedread_rule2 fields changed
      • addedInput schema / properties / episode_id / description
        Added value: +"The episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e"
      • addedInput schema / properties / section / description
        Added value: +"Section number, e.g. 2143.01"
    • Addedrequest_service
    • Changedsearch_library2 fields changed
      • addedInput schema / properties / episode_id / description
        Added value: +"The episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e"
      • addedInput schema / properties / query / description
        Added value: +"Keywords or a short phrase describing the technology, e.g. 'battery cell cooling plate'"
    • Changedsearch_rules2 fields changed
      • addedInput schema / properties / episode_id / description
        Added value: +"The episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e"
      • addedInput schema / properties / query / description
        Added value: +"The rule or test you need, e.g. 'obviousness combining references'"
    • Changedstart_task3 fields changed
      • changedInput schema / properties / task_id / description
        Previous value: -"Optional specific task id"New value: +"A specific task id to retry; omit to get the next unserved case"
      • changedInput schema / properties / taskset / description
        Previous value: -"Optional task set: patents-first-action (default), patents-response or patents-appeal"New value: +"Which kind of case to work. Default patents-first-action. tariffs-classification asks for the tariff code of a real product."
      • addedInput schema / properties / taskset / enum
        Added value: +[
        +  "patents-first-action",
        +  "patents-response",
        +  "patents-appeal",
        +  "tariffs-classification"
        +]
    • Changedsubmit_answer2 fields changed
      • addedInput schema / properties / answer / description
        Added value: +"The decision, in the answer format from start_task"
      • addedInput schema / properties / episode_id / description
        Added value: +"The episode id returned by start_task, e.g. ep_3f9c2a1b0d4e5f6a7b8c9d0e"
  2. 8 tool updatesv0.1.0
    • First observedget_scorecard
    • First observedget_usage
    • First observedread_publication
    • First observedread_rule
    • First observedsearch_library
    • First observedsearch_rules
    • First observedstart_task
    • First observedsubmit_answer

TDQS

A4.2/5.0

Scored across 9 tools

Disambiguation5/5

Each tool has a clearly distinct role in the evaluation workflow: start/submit bookend an episode, search/read pairs target either the reference library or the rulebook, and the remaining tools handle support and account metrics. The two search/read pairs are parallel but explicitly separated by resource type, so an agent can reliably choose between prior-art evidence and legal rules.

Naming Consistency5/5

All tool names use a consistent snake_case verb_noun convention: start_task, search_library, read_publication, search_rules, read_rule, submit_answer, request_service, get_scorecard, get_usage. The verbs are predictable and accurately reflect each tool's action.

Tool Count5/5

Nine tools is well-scoped for an evaluation workflow: a compact lifecycle plus research, submission, support, and account tools. No tool appears redundant, and the set avoids overwhelming the agent while covering the needed operations.

Completeness4/5

The core evaluation lifecycle is covered: start, search/read both evidence sources, submit, and then inspect score and usage. Minor gaps exist around active-episode management, such as re-fetching an open episode's instructions or listing active episodes, but these are likely workarounds rather than dead ends.

Maintenance

ActivityMaintained
ResponsivenessSyncing

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Injects structured professional domain expertise into AI agents, providing access to decision frameworks, workflows, terminology, and regulatory context across 24 professional domains.
    56 npm
    MIT
  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables AI agents to autonomously investigate simulated production incidents by checking live service health, logs, deployments, and database status while retrieving relevant historical postmortems and runbooks to propose evidence-backed root causes.
    -
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables agents to evaluate single records or batches with dynamically authored typed questions, returning structured decisions and probabilities.
    23 PyPI
    MIT