Skip to main content
Glama
Sign In
deepset-mcp

deepset-mcp

Official
by deepset-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

create_experiment

Creates an experiment comparing a candidate pipeline version against a baseline, scored by one Metric Group over Dataset test cases; preview shows reuses, replays, and cells before running.

Instructions

Creates an Experiment that compares a candidate pipeline version against a baseline.

Both sides are scored by the same Metric Group over the test cases of one Dataset. The baseline is either another pipeline version (give baseline_pipeline_name and baseline_pipeline_version_id together) or Recorded (leave both unset): a Recorded baseline reuses the Sessions each test case recorded, with nothing replayed. The candidate is always a pipeline version that will be replayed on each test case.

Creating costs money: it replays the pipeline on the customer's models, and then every Session is judged by the Metric Group's Evaluators, which are billed when they use an LLM. Call it first with preview=True: that starts nothing and returns, per side, how many Sessions are reused_sessions and how many replays_needed, plus cells (Sessions times Evaluators) against max_cells. Show that to the user and get a yes before creating for real. sessions_per_test_case sets how many Sessions each side scores per test case (1..50); existing Sessions on that version are reused and only the shortfall is replayed. Set reuse_sessions=False to replay every Session instead of reusing any, for example after a change outside the pipeline's configuration such as a secret; it costs more replays, so preview it. A Recorded baseline still scores the Dataset's own Sessions. Find ids with list_metric_groups (metric groups) and list_experiments (an existing Experiment's dataset_id). Read the running or finished Experiment with get_experiment_results; follow its progress with get_experiment. :param name: Human-readable name for the Experiment. :param dataset_id: The Dataset's id (the UI's Dataset dropdown). :param metric_group_id: The Metric Group id to score both sides with. :param candidate_pipeline_name: Name of the candidate pipeline. :param candidate_pipeline_version_id: The candidate pipeline version to replay. :param baseline_pipeline_name: Baseline pipeline name; omit for a Recorded baseline. :param baseline_pipeline_version_id: Baseline version id; required when baseline_pipeline_name is given. :param sessions_per_test_case: Repetitions per test case (1..50, default 1). :param replay_mode: FIRST_USER_MESSAGE replays the first turn only; ALL_USER_MESSAGES replays every turn. :param reuse_sessions: False replays every Session each side scores and reuses none (default True). :param preview: True returns the plan (what would be reused and replayed) and creates nothing. :returns: The created Experiment, or with preview the plan, or an error message.

Input Schema

NameRequiredDescriptionDefault
nameYes
previewNo
dataset_idYes
replay_modeNoFIRST_USER_MESSAGE
reuse_sessionsNo
metric_group_idYes
baseline_pipeline_nameNo
sessions_per_test_caseNo
candidate_pipeline_nameYes
baseline_pipeline_version_idNo
candidate_pipeline_version_idYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.1.44
    • addedInput schema / properties / preview
      Added value: +{
+  "default": false,
+  "title": "Preview",
+  "type": "boolean"
+}
    • addedInput schema / properties / reuse_sessions
      Added value: +{
+  "default": true,
+  "title": "Reuse Sessions",
+  "type": "boolean"
+}
  2. Addedv0.1.33

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does: it discloses that creating costs money (pipeline replays on customer models plus billed LLM evaluators), what preview returns (reused_sessions, replays_needed, cells vs max_cells), and that preview starts nothing. It also explains the side effects of reuse_sessions=False and that a Recorded baseline reuses existing Sessions with nothing replayed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the cost warning and the preview-first rule, which is the most important thing an agent must know. It is long but dense; the only waste is that preview, reuse_sessions, and sessions_per_test_case are each explained twice, once in prose and again in the parameter list.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers everything needed for an 11-parameter, high-cost mutation with no annotations: cost model, preview protocol, baseline pairing rules, session reuse semantics, and a :returns: line standing in for the absent output schema. Nothing material is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must supply meaning, and the :param: block does — baseline_pipeline_name/version_id must be given together or omitted for a Recorded baseline, sessions_per_test_case is 1..50 default 1, replay_mode enum values are interpreted, reuse_sessions and preview defaults are stated. A few entries (name, dataset_id) merely restate the parameter, keeping it short of a 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — creates an Experiment comparing a candidate pipeline version against a baseline — and immediately explains the two sides, the dataset/metric-group scoping, and the two baseline flavors. An agent can distinguish it from get_experiment, list_experiments, and start_experiment_run without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit workflow guidance: call first with preview=True, show the plan to the user and get a yes before creating for real. It also routes id discovery to list_metric_groups and list_experiments, and points to get_experiment_results / get_experiment for reading and following the run.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools