Skip to main content
Glama

Create an eval config

modelfinder_create_eval_config
Read-onlyIdempotent

Builds and validates an evaluation config from candidate models, examples, checks, and settings; screens examples for personal data and secrets and returns JSON ready to save and run.

Instructions

Build an eval config from candidate models, examples with checks, and settings, then validate it and screen the examples for personal data and secrets. Use once examples and checks are agreed; to check a config the user already has, use modelfinder_validate_eval_config instead. Read-only and local: no network calls, no files written. Returns the config for the user to save as JSON and run with eval_models.py, plus a valid flag, the API call count and the next step. Keys are referenced by environment variable name only, never by value.

Input Schema

NameRequiredDescriptionDefault
taskNoOne-line description of what the model must do.
repeatsNoRuns per example (3+ for noisy tasks).
examplesYesInputs with the checks each reply must pass.
settingsNomax_output_tokens (default 1024), timeout_s (default 60), temperature, system (system prompt).
candidatesYes1-10 models to compare.
pass_thresholdNoShare of calls that must pass every check (0-1).
response_formatNomarkdown (default) for a readable summary, json for the full machine-readable result.markdown

Output Schema

NameRequiredDescriptionDefault
validYes
configYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv1.3.5

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so the safety profile is covered; the description still adds real behavior beyond them by disclosing that examples are screened for personal data and secrets and by stating exactly what comes back (config, valid flag, API call count, next step). The 'no network calls, no files written' line largely restates the annotations rather than adding new information.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Five compact sentences, front-loaded with the action and the alternative, then behavior, then return values, then the key-handling caveat. Nothing is padding and there is no repetition of the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 7 parameters, nested objects and enums, the description covers what is built, what side effects do not occur, what is returned and what to do next. With an output schema present it does not need to enumerate return fields, and the screening and file-writing behavior are exactly the non-obvious facts an agent needs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter is already documented in the schema, including the env-var-only key convention and the checks type list. The description restates the top-level shape (candidates, examples with checks, settings) and reinforces the env-var rule, but adds no syntax, constraints, or examples the schema does not already carry; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and composite resource: 'Build an eval config from candidate models, examples with checks, and settings,' then adds the validation and personal-data/secret screening steps. It also names the sibling it is not (modelfinder_validate_eval_config), so an agent can separate the two without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states the trigger condition ('Use once examples and checks are agreed') and explicitly routes the alternate case to modelfinder_validate_eval_config for configs the user already has. That is a concrete when/when-not pair with a named alternative, not implied usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.