ModelLens
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| DATA | Yes | Path to the history file passed via `--data` (e.g. "/path/to/history.json"). Stores the newest 200 probes per model so rankings improve over time. | |
| CONFIG | Yes | Path to the JSON config file passed via `--config` (e.g. "/path/to/config.json"). Defines the `endpoints` array, where each entry has `name`, `base_url`, an optional `api_key` (may be empty for a localhost gateway), an optional `models` filter, and an optional `enabled` flag. |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
ModelLens: discover, probe, rank and recommend models across OpenAI-compatible endpoints.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| list_modelsC | List the models each configured endpoint advertises. |
| probeC | Probe models with real requests: latency, success, streaming, tool calls. |
| rankingC | Quality ranking built from recorded probe history. |
| recommendC | Best model for a task profile. |
| reportC | The whole picture as markdown. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 5 tools
Each tool maps to a distinct pipeline stage (list → probe → rank → recommend → report), so purposes are mostly clear. The only soft overlap is between 'ranking' (quality ordering from history) and 'recommend' (best model for a task profile), which could be confused by an agent wanting a single 'which is best' answer.
Only list_models follows a verb_noun pattern; the rest mix bare verbs (probe, recommend) with bare nouns (ranking, report). The names are readable and short, but the convention is not predictable.
Five tools is well-scoped for a model evaluation/benchmarking server, and each maps to a necessary stage of the workflow. No tool feels redundant or padded.
The surface covers listing, probing, ranking, recommending, and reporting, which is a full analysis lifecycle. However, endpoints are referenced as 'configured' yet there is no tool to add, edit, or remove endpoints, and no way to clear or inspect probe history beyond the derived ranking.