Create an eval config
modelfinder_create_eval_configBuilds and validates an evaluation config from candidate models, examples, checks, and settings; screens examples for personal data and secrets and returns JSON ready to save and run.
Instructions
Build an eval config from candidate models, examples with checks, and settings, then validate it and screen the examples for personal data and secrets. Use once examples and checks are agreed; to check a config the user already has, use modelfinder_validate_eval_config instead. Read-only and local: no network calls, no files written. Returns the config for the user to save as JSON and run with eval_models.py, plus a valid flag, the API call count and the next step. Keys are referenced by environment variable name only, never by value.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| task | No | One-line description of what the model must do. | |
| repeats | No | Runs per example (3+ for noisy tasks). | |
| examples | Yes | Inputs with the checks each reply must pass. | |
| settings | No | max_output_tokens (default 1024), timeout_s (default 60), temperature, system (system prompt). | |
| candidates | Yes | 1-10 models to compare. | |
| pass_threshold | No | Share of calls that must pass every check (0-1). | |
| response_format | No | markdown (default) for a readable summary, json for the full machine-readable result. | markdown |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| valid | Yes | ||
| config | Yes |