Skip to main content
Glama

atlas_start_custom_eval_batch

Start scoring several candidates with a custom evaluation model (5 credits per candidate). Returns batch_id: check atlas_get_custom_eval_batch_status(batch_id), then atlas_get_custom_eval_batch_results(batch_id). Runs in the background: usually a few minutes, longer for large batches.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "custom_model_id": "<custom_model_id>",
      "candidate_ids": "<candidate_ids>",
      "context_id": "<context_id>"
    },
    "name": "atlas_start_custom_eval_batch"
  }
}

Input Schema

NameRequiredDescriptionDefault
context_idYesFrom atlas_list_contexts
template_idNo
detail_levelNostandard
candidate_idsYesFrom atlas_list_candidates
custom_model_idYesFrom atlas_list_custom_eval_models

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed3 schema fields changed
    • changedInput schema / properties / candidate_ids / description
      Previous value: -"Candidate IDs from atlas_list_candidates"New value: +"From atlas_list_candidates"
    • changedInput schema / properties / context_id / description
      Previous value: -"Context ID from atlas_create_context or atlas_list_contexts"New value: +"From atlas_list_contexts"
    • changedInput schema / properties / custom_model_id / description
      Previous value: -"Custom eval model ID (from Atlas dashboard)"New value: +"From atlas_list_custom_eval_models"
  2. First observed

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the mutation/async profile (readOnlyHint=false, openWorldHint=true, idempotentHint=false, destructiveHint=false), so the description's job is to add context. It does add real value with the per-candidate credit cost, the async background execution model, expected duration, and the returned batch_id. It omits failure behavior, whether credits are charged on partial failure, and whether re-invocation duplicates work.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four tightly packed sentences with the action and cost front-loaded, followed by the return value and the follow-up chain, then the async timing. No filler and no repetition of the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly supplies the return contract (batch_id) and the downstream calls, plus timing expectations, which covers the essentials for a launch-a-job tool. It falls short only on the undocumented parameters and on what happens when a batch fails or partially completes.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 60% and the description names no parameters at all, so it adds no meaning beyond the schema. Two parameters (template_id, detail_level) are undocumented in both places, and the description does not explain the tradeoff of the brief/standard/deep enum, which directly affects scoring quality.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ("Start scoring several candidates with a custom evaluation model") and implicitly scopes it away from the related inference/batch tools by naming the custom-eval batch lifecycle. An agent can tell this is the batch-launch entry point of the custom eval chain without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear operational follow-up guidance: call atlas_get_custom_eval_batch_status(batch_id) then atlas_get_custom_eval_batch_results(batch_id), which tells the agent this is the first step of a workflow. It does not state when to prefer this over atlas_start_custom_eval_inference, so the alternative-selection condition is left implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources