Skip to main content
Glama

Create custom benchmark

create_custom_benchmark

Use when someone wants to keep watching the prices of products they choose (e.g. five named writing tools). Creates a PRIVATE custom benchmark (a saved, calculated peer cohort) over Agentery's data — no account needed. Two modes: (A) explicit members: pass members (a list of exact handles; product names/domains resolve where unambiguous). (B) fork a market: pass base_niche (its slug) plus optional remove/add. Returns a one-time secret benchmark_id (cb_… token) — store it; it's your only key. Use it later in get/update/delete and in market_report/get_price_index/get_price_index_history. Ambiguous names are returned as candidates, never silently resolved; unresolved inputs block creation unless allow_partial:true. All prices/history are computed from Agentery's immutable observations; canonical market data is never changed.

Input Schema

NameRequiredDescriptionDefault
addNomembers to add
nameNoOptional private label
removeNoMode B: members to drop from the forked market
membersNoMode A: exact handles (preferred), product names or domains
base_nicheNoMode B: slug of the canonical market to fork
allow_partialNoCreate with only the resolved members when some inputs don't resolve (default false)

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "description": "JSON result; identical to the text content block.",
      +  "type": "object"
      +}
  2. Changed2 schema fields changed
    • changedInput schema / properties / base_niche / description
      Previous value: -"Mode B: canonical niche slug to fork"New value: +"Mode B: slug of the canonical market to fork"
    • changedInput schema / properties / remove / description
      Previous value: -"Mode B: members to drop from the forked niche"New value: +"Mode B: members to drop from the forked market"
  3. Added

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare non-readonly, non-idempotent, non-destructive, closed-world behavior, but the description adds crucial context the agent would otherwise lack: the benchmark is PRIVATE, it returns a one-time secret benchmark_id token that is the sole key for later operations, ambiguous names are returned as candidates rather than auto-resolved, and unresolved inputs block creation unless allow_partial:true. It also reassures that canonical market data is never changed, which matters for a create operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but front-loaded: the primary purpose and trigger condition come first, followed by the two modes and then the return-value/error-handling details. Every sentence carries useful information, though the single-paragraph form is long enough that mode details and secret-token guidance compete for attention; light structuring (e.g., explicit 'Mode A / Mode B' labels, which are present) keeps it readable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given six optional parameters, an output schema that covers return values, and rich annotations, the description fills the remaining gaps an agent needs: the two creation modes, ambiguity handling, the one-time secret benchmark_id and its downstream uses, and the allow_partial behavior. Nothing needed to call this tool correctly or interpret its result is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already explains each parameter. The description nonetheless adds mode-level semantics the schema cannot convey: which parameters belong to Mode A vs Mode B, that product names/domains resolve only when unambiguous, and the precise effect of allow_partial (creating with resolved members only). This cross-parameter logic materially reduces invocation error risk.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (creates a private custom benchmark / saved peer cohort) and immediately distinguishes it from siblings like get_custom_benchmark, update_custom_benchmark, and delete_custom_benchmark by naming where the returned benchmark_id is later used. The framing 'Use when someone wants to keep watching the prices of products they choose' makes the intent unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly defines two modes with the exact parameter that selects each (members for Mode A, base_niche plus optional remove/add for Mode B) and states no account is needed. It also specifies downstream usage (get/update/delete, market_report, get_price_index) and the allow_partial fallback when inputs don't resolve, leaving no ambiguity about when to use this tool versus its siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources