Create an eval config
modelfinder_create_eval_configAssemble an eval config from candidate models, test examples with checks, and settings. Validate the config and screen examples for personal data and secrets before saving as JSON.
Instructions
Assemble an eval config from candidate models, examples with checks, and settings; validate it and screen the examples for personal data and secrets. Returns the config to save as JSON for eval_models.py. Keys are referenced by environment variable name only.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| task | No | One-line description of what the model must do. | |
| repeats | No | Runs per example (3+ for noisy tasks). | |
| examples | Yes | Inputs with the checks each reply must pass. | |
| settings | No | max_output_tokens (default 1024), timeout_s (default 60), temperature, system (system prompt). | |
| candidates | Yes | 1-10 models to compare. | |
| pass_threshold | No | Share of calls that must pass every check (0-1). | |
| response_format | No | markdown (default) for a readable summary, json for the full machine-readable result. | markdown |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| valid | Yes | ||
| config | Yes |