Skip to main content
Glama
Sign In
deepset-mcp

deepset-mcp

Official
by deepset-ai
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

start_experiment_run

Scores selected or time-windowed pipeline Sessions with a Metric Group outside an experiment; preview the judged cells, then queue and poll the run.

Instructions

Scores a pipeline's Sessions with a Metric Group, outside any Experiment. Costs money.

Every Evaluator of the Metric Group judges every chosen Session, so the run has Sessions times Evaluators cells, and an Evaluator that judges with a model is billed per judged turn on the customer's provider. Name the Sessions in session_ids (find them with list_sessions), or give a window of time: window_from and window_to together, optionally narrowed by origin, deployment_id, pipeline_version_id and revision_id, judge every Session whose last activity falls in it. The server resolves the window to its Sessions when the run starts. Give session_ids or a window, never both. A window run is always SAMPLED and cannot take focus; it is refused when it has more Sessions or cells than the ceiling.

Preview first: preview=True with a window starts nothing and returns sessions, evaluators and cells (the judge calls it would make) against the ceiling. Show that to the user and get a yes before starting for real. There is no preview for named Sessions: their cells are the number of Sessions times the Metric Group's Evaluators.

The run is queued, not finished: poll it with get_experiment_run (counts, then the grid), and find earlier runs with list_experiment_runs. reuse REUSE_SUCCEEDED judges only what this Metric Group has not already judged successfully, which makes re-running after a partial failure cheap; RERUN_ALL judges everything again. selection HAND_PICKED marks Sessions chosen because they looked wrong, which keeps the run out of trends and exports; use it for named Sessions that are not a sample of the traffic. A focus makes it HAND_PICKED. :param pipeline_name: Name of the pipeline whose Sessions are judged. :param metric_group_id: The Metric Group to judge with (see list_metric_groups). :param session_ids: The Sessions to judge: search_session_id values, or a single-turn run's query_id. :param window_from: Window start, ISO 8601 with a timezone: last activity at or after it. :param window_to: Window end, ISO 8601 with a timezone, not in the future: last activity at or before it. :param origin: Window filter: only Sessions from SERVICE, PIPELINE_RUN or PLAYGROUND. :param deployment_id: Window filter: only Sessions this Service served. :param pipeline_version_id: Window filter: only Sessions this pipeline version served. :param revision_id: Window filter: only Sessions this Service revision served. :param reuse: RERUN_ALL (default) or REUSE_SUCCEEDED. :param selection: SAMPLED (default) or HAND_PICKED. :param focus: Per Session id, the turns to judge by query_id; a Session left out is judged on every turn. :param preview: True counts what the window would judge and starts nothing. :returns: The queued run (pipeline_experiment_run_id, cells), or with preview the counts, or an error.

Input Schema

NameRequiredDescriptionDefault
focusNo
reuseNoRERUN_ALL
originNo
previewNo
selectionNoSAMPLED
window_toNo
revision_idNo
session_idsNo
window_fromNo
deployment_idNo
pipeline_nameYes
metric_group_idYes
pipeline_version_idNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.1.44

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does it well: it discloses that the run costs money, bills per judged turn on the customer's provider, may be refused at a ceiling, runs asynchronously ('queued, not finished'), and that ``selection`` HAND_PICKED keeps runs out of trends and exports. These are exactly the non-obvious traits an agent must know before mutating state with financial impact.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the critical cost warning and organized logically (costs, targeting, preview, lifecycle, semantics, then params). Slightly long, and the 'Sessions times Evaluators cells' arithmetic is stated twice (in prose and again in the preview paragraph), which is minor redundancy rather than waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 13-parameter, high-cost, asynchronous mutation with no annotations and no output schema, the description covers cost model, targeting modes, preview/confirmation flow, ceiling rejection, polling follow-up, and the return payload fields (``pipeline_experiment_run_id``, ``cells``). Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 13 parameters, yet the description documents each one: the ISO 8601-with-timezone requirement and non-future constraint for window bounds, the meaning of each window filter (``deployment_id``, ``pipeline_version_id``, ``revision_id``), the ``reuse`` and ``selection`` enum effects, and that ``focus`` maps Session ids to turn ``query_id`` lists. It fully compensates for the empty schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource: 'Scores a pipeline's Sessions with a Metric Group, outside any Experiment.' The 'outside any Experiment' clause explicitly distinguishes it from the experiment-creation siblings, and the agent can tell immediately what it does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit routing: find Sessions with ``list_sessions``, give ``session_ids`` or a window but never both, preview with ``preview=True`` and get user confirmation before starting, poll with ``get_experiment_run``, find prior runs with ``list_experiment_runs``. Alternatives and sequencing are spelled out rather than implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools