Skip to main content
Glama
thenavidm

Gumloop MCP Server

by thenavidm
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Hybrid

Run evaluation on sessions

run_organization_evaluation
Destructive

Grades up to 200 existing sessions with an organization evaluation, queues results asynchronously, and skips ineligible or in-flight sessions. Use dry_run to preview cost without queuing.

Instructions

Grades up to 200 existing sessions with this evaluation. Grading is asynchronous: each accepted session gets a result with status: queued; poll it with GET /evaluations/{evaluation_id}/results/{result_id} until it is completed or failed.

Sessions are skipped, not rejected, when they are not completed sessions of an agent the evaluation covers (ineligible) or already have a queued or running result for this evaluation (in_flight, with the existing result_id). The caller is charged one credit per queued session. Set dry_run: true to see the cost and skips without queuing anything. Explicit confirmation is required for this exact account operation. Runs can spend credits or trigger downstream actions; never resubmit unknown outcomes automatically.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
accountNoNamed private Gumloop account; selects private credentials and user/team identity.
confirmNoSet true only when the user asked for exactly this action.
dry_runNoReport cost and skipped sessions without queuing.
payloadNoComplete JSON request body instead of body flags. Preserves current endpoint fields and values.
session_idsNoSessions to grade. Duplicates are rejected.
payload_fileNoRegular local JSON body file, at most 5 MB. Cannot be mixed with body flags or payload.
evaluation_idYesID of the organization evaluation.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv3.0.0
    • changedInput schema / properties / confirm / description
      Previous value: -"Must be true for this exact requested account change, agent/flow execution, upload or deletion."New value: +"Set true only when the user asked for exactly this action."
  2. First observedv2.0.1

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Substantially exceeds the annotations: it discloses asynchronous grading, the queued status and the exact polling endpoint, skip-vs-reject semantics with the ineligible and in_flight reasons, the one-credit-per-queued-session charge, and a warning never to auto-resubmit unknown outcomes. This is consistent with destructiveHint=true and adds real operational context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core action is front-loaded and the async/polling detail follows logically. Three short paragraphs with little filler, though the credit and confirmation warnings are slightly repetitive with schema descriptions.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex, nested, async, credit-spending tool with no output schema, the description supplies everything an agent needs: batching limit, async flow, polling path, skip conditions, cost model, dry-run preview and confirmation requirement. Nothing material is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents account, confirm, dry_run, session_ids, payload and payload_file. The description restates the dry_run effect and the 200-session cap but adds no syntax or semantics beyond the structured fields, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence gives a specific verb and resource ('Grades up to 200 existing sessions with this evaluation'), so an agent immediately knows what the tool does. It does not, however, explicitly distinguish itself from the sibling run_evaluations, which a 5 would require.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear situational guidance: use dry_run:true to preview cost and skips without queuing, and confirmation is required for this exact account operation. It explains skip conditions (ineligible, in_flight) but never names an alternative sibling such as run_evaluations, so it stops short of full when/when-not/alternative coverage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools