DIO Predictive Inference Orchestrator
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| DIO_SLO_MS | No | Admission budget in milliseconds. | 5000 |
| DIO_ADMISSION_MODE | No | Admission mode: empirical, rank_only, or absolute (diagnostic). | empirical |
| DIO_CACHE_BONUS_MS | No | Session/prefix affinity bonus in milliseconds. | 200 |
| DIO_ENGINE_METRICS | No | Whether to scrape vLLM /metrics. | true |
| DIO_KV_CACHE_COST_MS | No | KV cache cost coefficient (c_kv) multiplied by KV utilization. | 800 |
| DIO_ENGINE_QUEUE_COST_MS | No | Engine queue cost coefficient (c_q) multiplied by waiting requests. | 50 |
| DIO_ENGINE_PREFIX_HIT_BONUS_MS | No | Prefix hit bonus coefficient (c_p) multiplied by prefix-hit rate. | 150 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| dio_get_modelsB | Query DIO gateway for available LLM models, active backends, and cluster health. |
| dio_predict_latencyA | Get latency and cost predictions before sending requests. Predicts queue delay, execution latency, and optimal backend using DIO's dual-timescale NLMS filter. |
| dio_route_promptC | Route a prompt through DIO's intelligent NLMS scheduler to the optimal backend and return the completion. |
| dio_cluster_statusA | Get live telemetry from the DIO cluster, including learned worker slopes, KV-cache pressure, and admission goodput statistics. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
The tools have mostly distinct purposes: listing models, predicting latency, routing prompts, and checking cluster telemetry. Minor overlap exists between dio_get_models and dio_cluster_status since both mention health, but their descriptions clarify that one is model/backend inventory and the other is deep operational telemetry.
Three tools follow a clear verb_noun pattern (get_models, predict_latency, route_prompt), while dio_cluster_status deviates by omitting the verb. The shared 'dio_' prefix and overall readability keep the set mostly consistent.
With 4 tools, the server is well-scoped for an inference orchestrator. Each tool covers a distinct, necessary capability (discover, predict, route, monitor) without unnecessary bloat.
The core workflow of checking available models, predicting latency, routing a prompt, and inspecting cluster health is fully covered. Minor gaps exist such as detailed per-model information or explicit policy management, but these are not critical for the server's stated purpose.