Skip to main content
Glama

Ai Model Experiments

Experiment Create

experiment_create
Read-onlyIdempotent

Model Lab: run one prompt across many AI models at once and compare their outputs, cost, and latency side by side. Create and start an experiment: run each prompt against each model (× reps), collecting output, tokens, latency, and billed cost per cell. PREPAID: requires Pipeworx credit balance ≥ max_spend_usd (top up via experiment_topup); bills actual provider cost × 1.5 with a $0.10 minimum per experiment. ASYNC: returns experiment_id immediately — execution starts within ~1 minute; poll experiment_status until complete, then call experiment_results. Do NOT wait synchronously. Set summary:false to skip the AI-written model-comparison summary. Example: experiment_create({ name: "tone test", prompts: ["Rewrite formally: ..."], models: ["anthropic/claude-haiku-4.5", "openai/gpt-5-mini"], reps: 2, max_spend_usd: 2 })

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
nameNoShort experiment name
repsNoRepetitions per prompt×model for variance, 1-5 (default 1)
modelsYesModel ids from experiment_models (max 12)
paramsNoOptional {system, temperature, max_tokens (default 512)}
promptsYesPrompts to test (max 20)
summaryNoAI-written comparison of the models' outputs when the run completes (default true)
max_spend_usdYesREQUIRED hard spend cap in USD for this experiment (max 100). Execution stops when reached.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description declares a mutating, money-spending operation ('Create and start an experiment', 'bills actual provider cost × 1.5', requires prepaid balance), yet annotations claim readOnlyHint=true and idempotentHint=true. This directly contradicts the structured safety profile an agent relies on, which is a serious inconsistency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action, then billing, async behavior, and an example in a logical order. It is dense and long, but nearly every sentence carries operational weight (prepaid requirement, async polling, summary toggle, example).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description still explains the return (experiment_id immediately, execution starts within ~1 minute) and the full downstream workflow via experiment_status and experiment_results. Nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning beyond the schema: it explains reps as prompt×model repetitions, clarifies summary:false skips the AI-written comparison, and ties max_spend_usd to the prepaid billing model. The worked example shows how the parameters combine.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (create and start an experiment running each prompt against each model × reps) and names the observable outputs (output, tokens, latency, billed cost per cell). It is clearly distinguishable from siblings like experiment_status, experiment_results, and experiment_models without opening their schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent: poll experiment_status until complete, then call experiment_results; top up via experiment_topup; and a hard 'Do NOT wait synchronously' exclusion. It names alternatives and the conditions that select them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.