Skip to main content
Glama
Sign In
ModelLens

ModelLens

by Greatbeing
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

ModelLens

See which of your models actually works.

An MCP server that turns any OpenAI-compatible gateway into an observable, rankable set of models. It discovers what a provider advertises, probes what works right now — latency, success, cancellation, streaming, tool calling — keeps the history, and answers "which model should I use?"

Zero third-party dependencies. Python 3.9+ standard library only.

Why

If you run more than one model provider, you already have this problem:

  • The gateway says it serves 89 models. How many actually answer?

  • Which one is fast today? Yesterday's answer is already stale.

  • Your router picks the first entry in a list — and the first entry is slow.

  • Failures are invisible until a user complains.

ModelLens makes that visible. It is the difference between trusting a model list and measuring it.

Real example from a live gateway (89 models, 16 probed):

magpie/kilo/nvidia/nemotron-3.5-super-vl:free     97.6  100.0%    0.0%      628ms   healthy
magpie/auto/beam-free                             96.6  100.0%    0.0%      909ms   healthy
magpie/longcat/LongCat-2.5                        88.4  100.0%    0.0%     3104ms   mixed
magpie/group/daily                                70.0  100.0%    0.0%    77345ms   mixed
magpie/radeon-cloud/DeepSeek-V4.1-Flash           70.0  100.0%    0.0%    87388ms   mixed
magpie/nvidia/deepseek-ai/deepseek-v4.1-flash     0.0    0.0%  100.0%         -    unreliable
magpie/modelscope/Qwen/Qwen3.8-27B                 0.0    0.0%    0.0%         -    quota_blocked

The default routing group was putting an 87-second model first and a 0.9-second model third. ModelLens found that in one pass.

Related MCP server: llmprobe

Install

# from PyPI
pip install modellens
modellens --config config.json --data history.json --once ranking

or straight from source — no dependencies, no install step:

curl -fsSL https://raw.githubusercontent.com/Greatbeing/modellens/main/modellens.py -o modellens.py
python3 modellens.py --config config.json --data history.json --once ranking

Configure

{
  "endpoints": [
    {"name": "magpie",  "base_url": "http://127.0.0.1:3425/v1", "api_key": ""},
    {"name": "openai",  "base_url": "https://api.openai.com/v1", "api_key": "sk-..."},
    {"name": "ollama",  "base_url": "http://127.0.0.1:11434/v1"}
  ]
}

api_key may be empty for a localhost gateway. models optionally filters the advertised list; enabled: false disables an endpoint without deleting it.

Run as an MCP server

Point any MCP client at it:

{
  "mcpServers": {
    "modellens": {
      "command": "python3",
      "args": ["/path/to/modellens.py",
               "--config", "/path/to/config.json",
               "--data",   "/path/to/history.json"]
    }
  }
}

Works with Claude Desktop, Cursor, Windsurf, VS Code, and any other MCP client.

DeepSeek Harness

Register it in the profile's cordis.patch.yml (note: a new entry needs - insert:, not a bare - id: — a bare one means "modify an existing entry"):

- insert:
    - id: modellens
      name: "@deepseek-ai/dsh-mcp-client"
      config:
        serverName: "modellens"
        transport: "stdio"
        command: "python3"
        args:
          - "/data/workspace/modellens/modellens.py"
          - "--config"
          - "/data/workspace/modellens/config.json"
          - "--data"
          - "/data/workspace/modellens/history.json"

Then restart dsh web. Verified: the server runs as a child process of dsh-web (pgrep -P $(systemctl --user show dsh-web -p MainPID)).

Tools

Tool

What it does

list_models

What each endpoint advertises (optional substring filter)

probe

Real requests: latency, success, streaming, tool calls

ranking

Quality ranking from recorded history

recommend

Best model for a profile: fast | cheap | capable

report

The whole picture as markdown

Example

probe(limit=20, check_tools=true)
ranking(limit=10)
recommend(profile="fast")

How it scores

Every probe is a real request, so the numbers are evidence, not estimates.

Signal

Weight

Notes

Success rate

50%

0% success ⇒ score 0, regardless of anything else

Median latency

30%

Linear to 0 at 8 s

Cancellation rate

20%

Timeouts and client cancellations

Verdicts: healthy (≥95% success, <3 s median) · mixed · unreliable (≥30% cancelled) · quota_blocked (all 429) · dead (all 404) · untested.

History is kept per model (newest 200 probes) in history.json, so rankings improve as you use it.

CLI

The server doubles as a command-line tool:

python3 modellens.py --config config.json --data history.json --once probe --limit 20
python3 modellens.py --config config.json --data history.json --once ranking

Use cases

  • Before you pick a model — recommend(profile="fast")

  • When a provider goes quiet — ranking shows it dropping before your users do

  • When your router misbehaves — compare group members, find the slow one

  • Cost control — spot the expensive model that is never chosen

  • Onboarding a new provider — probe once, see what actually works

Status and roadmap

Working today: discovery, probing, streaming/tool-call checks, scoring, ranking, recommendation, markdown report, MCP stdio transport, CLI.

Roadmap: OpenAI-compatible /v1/models only for now; Anthropic and Gemini transports, per-endpoint cost accounting, alerting webhooks, and a hosted multi-team version.

License

MIT. Use it, ship it, charge for your own hosted version.

Honest notes

  • Probing costs tokens. Each probe is a real completion. Keep limit sensible and don't probe a hundred models every minute.

  • A single probe is weak evidence. The score is meaningful after several probes; the report says n=1 so you can see the sample size.

  • Latency depends on you. Your network, not just the provider, is in the measurement. Run it from where your agents run.

Available Tools

5 tools
list_modelsC

List the models each configured endpoint advertises.

NameRequiredDescriptionDefault
limitNooptional max per endpoint
containsNooptional substring filter

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full disclosure burden, yet it says nothing about contacting remote endpoints, latency, or what happens when an endpoint is unreachable or advertises nothing. "Advertises" hints at live network probing, a significant behavioral trait that is left implicit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single well-formed sentence with the resource and scope front-loaded and no filler. It is efficiently sized, though its brevity contributes to the missing usage and return-shape context rather than compensating for it.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple (zero required parameters), and the description conveys the essential operation, which is adequate at a minimum-viable level. With no output schema and no annotations, however, the agent still cannot tell what the result looks like (model IDs grouped per endpoint?) or how failures are surfaced.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for both optional parameters ("optional max per endpoint", "optional substring filter"), so the schema already does the work and the baseline of 3 applies. The description adds nothing about whether the substring matches model names only, is case-sensitive, or whether it is applied per endpoint or globally.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("List the models") and scopes it to "each configured endpoint advertises," which tells an agent this is a discovery/inventory call rather than an inference call. It does not name or contrast with the abstract siblings (probe, ranking, recommend, report), so differentiation is inferred rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when this tool should be chosen over probe, ranking, or recommend, and no prerequisites (e.g. that endpoints must already be configured). The agent must guess that this is the pre-flight inventory step before invoking an inference-related sibling.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

probeC

Probe models with real requests: latency, success, streaming, tool calls.

NameRequiredDescriptionDefault
limitNomax models (default 20)
modelNoexact model id
containsNosubstring filter
endpointNooptional endpoint name
check_toolsNoalso test function calling

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It discloses that real requests are made and lists measured outcomes, but it omits important operational context such as cost, rate limits, permissions, latency expectations, and whether probing has side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single front-loaded sentence with no filler. It immediately communicates the core action and the measured dimensions. Every element earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool that makes real requests and has no annotations or output schema, the description is too thin. It hints at return dimensions but does not explain result format, cost implications, or operational constraints. The schema covers parameters, but behavioral completeness is weak.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the input schema already documents all five optional parameters. The description adds no parameter-specific meaning beyond the schema and does not clarify ambiguous types such as string-valued limit or check_tools. A baseline 3 is appropriate because the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: probing models with real requests. It also enumerates what is measured: latency, success, streaming, and tool calls. It does not explicitly distinguish itself from siblings like list_models or ranking, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies that the tool is used for live model probing rather than metadata listing, but it gives no explicit when-to-use guidance. It does not mention alternatives, exclusions, or conditions under which probing is preferred over ranking or recommending.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

rankingC

Quality ranking built from recorded probe history.

NameRequiredDescriptionDefault
limitNorows to show (default 30)

TDQS

C2.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full behavioral burden. The phrase 'recorded probe history' hints that results come from previously stored probe data rather than triggering new probes, which is the only behavioral signal; nothing is said about ordering direction, cost, freshness, caching, or whether calling it has any side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is a single short sentence with no filler and the key concept (quality ranking from probe history) is front-loaded. The terseness is close to under-specification rather than verbosity, but nothing in it is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no annotations and no output schema, the description should say more: whether the ranking is ordered best-to-worst, what a row contains, whether it is stale relative to new probes, and how it relates to 'recommend'. As written, an agent can call it but cannot predict what it returns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is a single optional parameter with 100% schema description coverage ('rows to show, default 30'), so the schema already does the work and the baseline of 3 applies. The description adds nothing about the parameter, and notably does not clarify why 'limit' is typed as a string rather than an integer.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description identifies the resource (a quality ranking derived from recorded probe history), which adds a little beyond the bare name 'ranking'. But it is a noun phrase with no verb, so it is ambiguous whether the tool retrieves, computes, or refreshes the ranking, and it draws no distinction from siblings like 'recommend' or 'report' that could plausibly surface comparable ranked output.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no exclusions, and no reference to any sibling tool. An agent has to infer from the name alone that this is the place to get a ranking rather than calling 'recommend' or 'report'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

recommendC

Best model for a task profile.

NameRequiredDescriptionDefault
profileNofast | cheap | capable

TDQS

C2.3/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full behavioral burden, and it discloses nothing: not whether the call is read-only, not what happens if profile is omitted (it is optional per the schema), not what an unrecognized profile yields, and not what the response contains. For a tool with zero annotation coverage, a single fragment is effectively no behavioral disclosure at all.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

There is zero wasted wording, but the content is a sentence fragment rather than a front-loaded statement of purpose, so the brevity comes at the cost of clarity. Conciseness here reads as under-specification rather than discipline.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no annotations, the description is the only place an agent could learn what a "recommendation" returns, and it says nothing about the return value, the effect of omitting the optional profile, or how results relate to the ranking sibling. A single sentence is not sufficient for a tool whose entire contract is otherwise undocumented.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the schema itself enumerates "fast | cheap | capable" in the profile description, so the description adds no meaning beyond the structured field. With the schema doing all the work, the baseline of 3 is appropriate; notably the description also fails to explain what the optional/empty case means, which the schema leaves ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Best model for a task profile" conveys the general idea of returning a model recommendation for a given profile, but it is a noun phrase with no verb and no indication of whether it returns one model, a ranked set, or a rationale. It also does nothing to distinguish itself from siblings like list_models or ranking, leaving the agent to guess which one produces a recommendation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to call this instead of list_models, ranking, probe, or report. The agent gets no signal about prerequisites, whether the call is deterministic, or when an alternative sibling is the better choice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

reportC

The whole picture as markdown.

NameRequiredDescriptionDefault

No parameters

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden, and it discloses only that output is markdown. It says nothing about whether the call is read-only, how expensive or slow a 'whole picture' aggregation is, or what sections the report contains. The single format hint is the only behavioral content.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is one short sentence with no wasted words, but the brevity comes from under-specification rather than economy. There is no structure, no front-loaded verb, and no indication of what the report covers.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no annotations, the description is the only source of information about what a caller receives, and 'the whole picture' is undefined. An agent cannot tell what data the markdown will contain, how large it will be, or when this is preferable to the more specific sibling tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is nothing for the description to disambiguate; the baseline of 4 applies. No parameter-related gaps exist because the schema is empty.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose2/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description "The whole picture as markdown" essentially restates the tool name 'report' plus an output format. It never states a verb, what subject is being reported on, or which data source is aggregated. Nothing distinguishes it from siblings like ranking or recommend, which could all plausibly produce 'the whole picture'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no preconditions, and no mention of alternatives among list_models, probe, ranking, or recommend. The agent must guess whether this replaces or supplements those tools. No exclusion or routing information is present at all.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 5 tool updatesv0.1.0
    • First observedlist_models
    • First observedprobe
    • First observedranking
    • First observedrecommend
    • First observedreport
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

B3/5.0

Scored across 5 tools

Disambiguation4/5

Each tool maps to a distinct pipeline stage (list → probe → rank → recommend → report), so purposes are mostly clear. The only soft overlap is between 'ranking' (quality ordering from history) and 'recommend' (best model for a task profile), which could be confused by an agent wanting a single 'which is best' answer.

Naming Consistency3/5

Only list_models follows a verb_noun pattern; the rest mix bare verbs (probe, recommend) with bare nouns (ranking, report). The names are readable and short, but the convention is not predictable.

Tool Count5/5

Five tools is well-scoped for a model evaluation/benchmarking server, and each maps to a necessary stage of the workflow. No tool feels redundant or padded.

Completeness4/5

The surface covers listing, probing, ranking, recommending, and reporting, which is a full analysis lifecycle. However, endpoints are referenced as 'configured' yet there is no tool to add, edit, or remove endpoints, and no way to clear or inspect probe history beyond the derived ranking.

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Related MCP Connectors

Related MCP Servers

  • ppb-mcp

    paulplee
    A
    license
    Not graded
    quality
    D
    maintenance
    Exposes queryable GPU inference benchmark data (quantization, throughput, VRAM, concurrent users) as tools for LLM clients.
    MIT

  • token-scout

    jackccrawford
    A
    license
    Not graded
    quality
    D
    maintenance
    Discovers LLM models in real time from cloud providers and local Ollama instances, returning compatibility profiles and live pricing so AI agents can route tasks to the cheapest viable model without breaking tool calls or context clipping.
    10
    MIT

  • llm-latency-tracker

    mazamaka
    A
    license
    A
    quality
    B
    maintenance
    Independent latency and uptime data for 46 hosted AI inference APIs, probed every 5 minutes from 4 regions with no gateway, plus a verified model deprecation calendar and Claude Code / Codex speed and accuracy drift. Four read-only tools, no API key; data CC BY 4.0.
    2
    4
    12
    MIT