Skip to main content
Glama

ZeroWidth Caliper

by ai.zerowidth
OverviewDiscussionsAdmin

Create a Caliper dataset

caliper_datasets_create

Creates a dataset of test items — THE FIRST STEP of setting up evaluation for a flow. Four item shapes: Q&A (input + optional expectedOutput, the golden answer); CHAT (messages: the conversation so far as role/content user and assistant messages you write, ending on a user message, + expectedResponse for the next reply; the flow writes only that reply, so every earlier assistant turn stays fixed; THE shape for an agent's conversation, built up by depth: a greeting alone, then two messages, then four, each item testing one more reply with the earlier ones baked in); SEQUENCE (turns: 2-20 scripted user messages the model answers one at a time with its own earlier replies in front of it, + expectedResponse for the final reply, optional expectedBehavior for the whole conversation); SIMULATED (goal + optional persona/disposition/strategy/maxTurns + expectedBehavior; a platform flow plays a person adaptively, Caliper-run evals only; disposition is a preset id like genuine, pressure, confused, impatient, or free text). Use sequences and simulated items for the slow attacks and for real customers with real needs: a model that holds on message one often folds on message ten. Write good inputs from real usage: the Compass pages the flow was built from are the best source of realistic scenarios.

Input Schema

NameRequiredDescriptionDefault
itemsYesTest items (1-100).
titleYesDataset name (2-120 chars).
workspaceNoWorkspace slug. Personal tokens with no default workspace MUST pass this; tokens with a default can override per call. Ignored for workspace API keys.
approvalIdNoApproval id from a prior needs_confirmation response. Omit on the first call.
visibilityNoWho can see it: PRIVATE (only the user), WORKSPACE (every member, the default), or SHARED (specific people, granted afterwards). Say 'make it private' → PRIVATE.
descriptionNoWhat this dataset covers.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changed
    • changedInput schema / properties / items / items / properties / expectedResponse / description
      Previous value: -"Sequence: the standard for the FINAL reply."New value: +"Sequence: the standard for the FINAL reply. Chat: the standard for the next reply."
    • addedInput schema / properties / items / items / properties / messages
      Added value: +{
+  "description": "Chat: the conversation so far as role/content messages (user and assistant alternating, an optional system first), 1-50. Every assistant message here is FIXED, written by you, not the model; the flow under test writes only the NEXT reply. End on a user message and put the standard for that reply in expectedResponse. This is the shape for testing the 5th reply of a conversation with the first four baked in.",
+  "items": {
+    "properties": {
+      "content": {
+        "type": "string"
+      },
+      "role": {
+        "enum": [
+          "system",
+          "user",
+          "assistant"
+        ],
+        "type": "string"
+      }
+    },
+    "required": [
+      "role",
+      "content"
+    ],
+    "type": "object"
+  },
+  "type": "array"
+}
  2. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly=false, destructive=false, openWorld=false, so the safety profile is covered; the description adds real behavioral context beyond them, notably that in CHAT items every assistant message is fixed and the flow writes only the next reply, and that SIMULATED items are Caliper-run evals only. It omits the approval/confirmation flow implied by the approvalId parameter.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the purpose well, but the body is a dense wall of text heavy with capitalized emphasis and restatement of the shape semantics already present in the schema descriptions. Much of the length is informative yet redundant, so it is not tightly economical.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description never says what a successful create returns (e.g., a dataset id needed for follow-up calls) nor explains the approvalId/needs_confirmation workflow. It is thorough on item shapes but leaves operational gaps for a mutation tool that seeds a downstream workflow.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description goes beyond the per-field schema blurbs by explaining how parameters interrelate for each shape (messages built up by depth, turns answered one at a time with prior replies in front, disposition presets versus free text). That conceptual framing adds genuine value over the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Creates a dataset of test items') and frames it as 'THE FIRST STEP of setting up evaluation for a flow', which positions it ahead of add_items/generate siblings. It never names an alternative explicitly, so the distinction from caliper_datasets_add_items or caliper_datasets_generate_items is only implied.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context on when each item shape applies ('Use sequences and simulated items for the slow attacks and for real customers with real needs') and where to source inputs (Compass pages the flow was built from). It stops short of stating when to use this tool versus the generate/generate_items alternatives or when not to use it at all.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All ZeroWidth Caliper toolsInspect
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources