Skip to main content
Glama

start_evaluation

Start token-weighted next-token loss and perplexity evaluation for local models; poll job status to monitor progress.

Instructions

Launch token-weighted next-token loss and perplexity evaluation; poll job_status.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "request": "<request>"
    },
    "name": "start_evaluation"
  }
}

Input Schema

NameRequiredDescriptionDefault
requestYes

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.1

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, destructiveHint=false, and openWorldHint=false, so the agent knows this creates local state without destroying anything. The description adds genuinely new behavioral information by implying an asynchronous job whose progress is tracked via job_status. It omits cost/duration expectations, device resource implications, and whether duplicate concurrent jobs are permitted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with the launching action first and the polling follow-up second; every clause carries information and nothing is redundant. Ideal size for a job-launch tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be described, and the async pattern is conveyed. Still, for a launch tool with eight nested configuration parameters and zero schema descriptions, the definition leaves the agent without prerequisites or parameter intent, which is a meaningful gap for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for a single 'request' object wrapping eight properties (model, dataset, dtype, device, adapter, batch_size, max_length, max_examples), and the description explains none of them. In particular it never clarifies whether 'model' and 'dataset' are registered asset identifiers matching the name pattern, nor what an 'adapter' does in the eval context, leaving the agent to infer from titles and patterns alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Launch') plus a precise resource ('token-weighted next-token loss and perplexity evaluation'), which is clearly distinguishable from siblings like start_generation or start_fine_tune. It stops short of an explicit contrast statement, but the metric definition alone tells an agent what this compute job produces.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The clause 'poll job_status' supplies a concrete post-invocation workflow pointer to a named sibling, which is real guidance. However, there is no statement of preconditions (model/dataset must already exist or be registered), no guidance on when to prefer evaluation over generation, and no note on whether re-launching an identical job is acceptable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.