Skip to main content
Glama
deepset-ai

deepset-mcp

Official
by deepset-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

create_experiment

Creates an experiment to compare a candidate pipeline against a baseline, scoring both with one metric group over a dataset's test cases.

Instructions

Creates an Experiment that compares a candidate pipeline version against a baseline.

Both sides are scored by the same Metric Group over the test cases of one Dataset. The baseline is either another pipeline version (give baseline_pipeline_name and baseline_pipeline_version_id together) or Recorded (leave both unset): a Recorded baseline reuses the Sessions each test case recorded, with nothing replayed. The candidate is always a pipeline version that will be replayed on each test case.

Replaying runs the pipeline on the customer's models, so get a yes before creating when the cost matters. sessions_per_test_case sets how many Sessions each side scores per test case (1..50); existing Sessions on that version are reused and only the shortfall is replayed. Find ids with list_metric_groups (metric groups) and list_experiments (an existing Experiment's dataset_id). Read the running or finished Experiment with get_experiment_results. :param name: Human-readable name for the Experiment. :param dataset_id: The Dataset's id (the UI's Dataset dropdown). :param metric_group_id: The Metric Group id to score both sides with. :param candidate_pipeline_name: Name of the candidate pipeline. :param candidate_pipeline_version_id: The candidate pipeline version to replay. :param baseline_pipeline_name: Baseline pipeline name; omit for a Recorded baseline. :param baseline_pipeline_version_id: Baseline version id; required when baseline_pipeline_name is given. :param sessions_per_test_case: Repetitions per test case (1..50, default 1). :param replay_mode: FIRST_USER_MESSAGE replays the first turn only; ALL_USER_MESSAGES replays every turn. :returns: The created Experiment, or an error message.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
nameYes
dataset_idYes
replay_modeNoFIRST_USER_MESSAGE
metric_group_idYes
baseline_pipeline_nameNo
sessions_per_test_caseNo
candidate_pipeline_nameYes
baseline_pipeline_version_idNo
candidate_pipeline_version_idYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.1.33

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it warns that replaying runs on customer models and requires a "yes" before creating when cost matters. It also explains the Recorded-vs-replayed asymmetry. It does not cover error behavior or rate limits, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well structured and front-loaded with purpose, then baseline semantics, cost warning, and parameter docs. It is somewhat long due to the embedded :param list, but every paragraph adds real information and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 9-parameter tool with no output schema and no annotations, the description covers the creation semantics, baseline selection, cost implications, and id-discovery routing. It omits what the Experiment response contains beyond a brief :returns line, but overall it is complete enough to call correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: every parameter is explained, including the baseline/candidate pairing rules, the 1..50 range and default for sessions_per_test_case, and the replay_mode enum values. The only gap is that enum semantics for replay_mode are brief.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Creates an Experiment") and immediately scopes it as comparing a candidate pipeline version against a baseline. It explains what the tool is for in one sentence, which distinguishes it from siblings like list_experiments and get_experiment_results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear conditions for the baseline mode (set both baseline fields together vs. leave unset for Recorded) and names sibling tools for finding ids and reading results. It does not explicitly state when not to use the tool, but the routing information is strong.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools