Skip to main content
Glama

Evaluation dataset builder

submit_eval_dataset_builder
Idempotent

Turn a table of task examples into a JSONL evaluation set and a scoring rubric. When you have labelled examples and need an eval set and rubric. Do not choose this when you need synthetic examples generated, which needs a model. Example requests: Build an eval dataset from these examples. | Turn this table of task examples into a JSONL eval set. | Make an evaluation set and rubric for this task.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "inputs": "<inputs>",
      "authorized_by": "<authorized_by>"
    },
    "name": "submit_eval_dataset_builder"
  }
}

Input Schema

NameRequiredDescriptionDefault
inputsYes
quote_idNoThe quote_id returned by POST /v1/jobs. Required to authorize a priced capability; the service rejects a priced job that does not name the quote it issued.
authorized_byYesIdentifier of the human or agent authorizing this job.
accepted_priceNoThe amount accepted, exactly as quoted (for example "5.00" or 5). Checked against the stored quote, so a price that moved between quote and authorize is refused rather than silently absorbed. Optional, but sent to make the acceptance explicit.

Output Schema

NameRequiredDescriptionDefault
quoteNo
stateYes
job_idYes
artifactsNo
access_tokenNoPer-job credential. Present it as the X-Job-Token header to read the job or list it. Keep it; it is the only credential for this job.
verificationNoIndependent verification. verification.ok = false means the job failed, not a partial success.
capability_idNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, idempotentHint=true, destructiveHint=false and openWorldHint=false, so the safety profile is covered. The description adds no behavioral context beyond that: it does not say this is a submitted, priced job, that a quote_id from a prior quote step is required, or how the JSONL/rubric output is delivered. Given the sibling set includes preflight_job, submit_job and quote-based authorization, that omission is a real gap for a non-read-only tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The first two sentences are appropriately front-loaded and earn their place. However, the trailing list of three example requests restates the same capability three times and adds no routing information beyond the opening sentence, which is avoidable bloat.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained. Still, this is a multi-step priced capability with a nested input object, and the description never acknowledges the quote-then-authorize flow implied by quote_id and accepted_price in the schema, nor any async/job semantics. Adequate on inputs, incomplete on the operational contract.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 75% and the nested inputs object documents both task_description and examples well, including format and byte limits. The description adds nothing about the authorization parameters (quote_id, accepted_price, authorized_by) or about the shape of the examples CSV. With the schema doing most of the work, baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence gives a specific verb and dual resource: turn a table of task examples into a JSONL eval set and a scoring rubric. That is far more concrete than the title alone, and it implicitly separates the tool from the many other submit_* converters. It stops short of explicitly distinguishing itself from near neighbors like submit_fine_tune_data_formatter or submit_regression_suite_generation, so it is clear but not fully differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states the selecting condition ('When you have labelled examples and need an eval set and rubric') and an explicit exclusion ('Do not choose this when you need synthetic examples generated, which needs a model'). That is the full when/when-not pattern, so an agent can route correctly without inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources