Skip to main content
Glama

Create custom benchmark

create_custom_benchmark

Use when someone wants to keep watching the prices of products they choose (e.g. five named writing tools). Creates a PRIVATE custom benchmark (a saved, calculated peer cohort) over Agentery's data — no account needed. Two modes: (A) explicit members: pass members (a list of exact handles; product names/domains resolve where unambiguous). (B) fork a market: pass base_niche (its slug) plus optional remove/add. Returns a one-time secret benchmark_id (cb_… token) — store it; it's your only key. Use it later in get/update/delete and in market_report/get_price_index/get_price_index_history. Ambiguous names are returned as candidates, never silently resolved; unresolved inputs block creation unless allow_partial:true. All prices/history are computed from Agentery's immutable observations; canonical market data is never changed.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
addNomembers to add
nameNoOptional private label
removeNoMode B: members to drop from the forked market
membersNoMode A: exact handles (preferred), product names or domains
base_nicheNoMode B: slug of the canonical market to fork
allow_partialNoCreate with only the resolved members when some inputs don't resolve (default false)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changed
    • changedInput schema / properties / base_niche / description
      Previous value: -"Mode B: canonical niche slug to fork"New value: +"Mode B: slug of the canonical market to fork"
    • changedInput schema / properties / remove / description
      Previous value: -"Mode B: members to drop from the forked niche"New value: +"Mode B: members to drop from the forked market"
  2. Added

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations leave gaps that the description fills well: no account needed, the result is private, the return is a one-time secret cb_ token that must be stored as the only key, ambiguous names come back as candidates and never silently resolve, unresolved inputs block creation unless allow_partial:true, and canonical market data is never mutated. That is unusually rich behavioral context for a create tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the trigger condition and mode breakdown, and every sentence carries information (modes, token storage, resolution policy). It is dense and abbreviation-heavy, but nothing is redundant padding; slightly tighter phrasing would earn a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists yet the description still states the return value (the one-time cb_ benchmark_id token) and its downstream uses, covers both creation modes, and disambiguates the failure/partial-resolution behavior. An agent has everything needed to invoke it correctly on the first try.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema coverage the baseline is 3, but the description adds real meaning the terse schema strings omit: it frames members as Mode A versus base_niche as Mode B, specifies that members accept exact handles (preferred) with names/domains resolving where unambiguous, and explains what allow_partial actually changes (create with only resolved members). This goes meaningfully beyond the schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Creates) and resource (a PRIVATE custom benchmark / saved calculated peer cohort), and immediately differentiates itself from siblings by naming get/update/delete and the read-side tools that consume its output. An agent can tell what this tool produces and where it fits in the lifecycle without opening any other schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly opens with 'Use when someone wants to keep watching the prices of products they choose' and then enumerates the two selection modes (A explicit members, B fork a market) with the exact parameters that select each. It also routes the agent to the follow-up tools the returned id feeds into, so both when-to-use and next-step are covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources