This server lets you interact with DIO, a smart LLM-serving gateway that routes prompts to optimal backends, predicts latency, and reports cluster health and telemetry.
List models and cluster health – dio_get_models shows available LLM models, active backends, and overall gateway health.
Predict latency before sending – dio_predict_latency estimates queue delay, execution latency, and the optimal backend for a prompt using DIO's NLMS-based prediction.
Route prompts intelligently – dio_route_prompt sends a prompt through DIO's scheduler, which picks the best backend and returns the completion.
Monitor live telemetry – dio_cluster_status provides learned worker slopes, KV-cache pressure, and admission/goodput statistics.
Leverage DIO's routing features – the gateway also supports session/prefix affinity, SLO-based admission control, and OpenAI/Ollama-compatible APIs.
Provides a gateway to route and load-balance requests across multiple Ollama instances, with session affinity and latency-based routing.
Scrapes backend engine Prometheus /metrics to inform routing decisions, such as KV-cache utilization and queue depth.
dio-serve is a thin OpenAI-compatible gateway that load-balances across several
stock vLLM replicas. No engine patches, no forked vLLM, no custom kernels — it
speaks the HTTP API and reads the Prometheus /metrics your engines already expose.
It exists because the usual answer, N vLLM processes behind Nginx or Envoy with Round-Robin, treats every replica as interchangeable while queue depth, KV-cache pressure, and transient per-replica slowdowns diverge in practice.
Paper: DIO: Hybrid Cost Routing, Session Affinity, and Calibration-Robust Admission for Multi-Instance LLM Serving over Stock vLLM — revised for a practice-oriented journal submission. Artifact DOI: https://doi.org/10.5281/zenodo.22085398
Because your replicas are not interchangeable. Round-robin assumes every backend has the same queue depth and the same free KV cache; on a real fleet that stops being true the moment one GPU picks up a co-tenant, hits a thermal limit, or gets handed a long context. DIO makes a different bet:
Nginx / Envoy round-robin | DIO | |
Backend choice | next in the ring | minimum predicted joint cost |
Predicts | nothing | per-engine |
Session affinity | none, or | content-prefix affinity, so the KV/prefix cache stays warm |
Overload | unbounded queue → latency cliff | SLO admission: |
Engine telemetry | ignored | optional read-only vLLM |
Drift | waits for a human | re-learns in O(1) per request, no training step |
It is not an edge proxy: DIO speaks the OpenAI and Ollama HTTP APIs and should sit behind an authenticating reverse proxy in production. It replaces the upstream-selection layer, not Nginx itself.
Related MCP server: OpenClaude MCP Server
pip install -e dio-serve # Python 3.9+
dio demo # mock fleet incl. a GPU that gets throttled mid-runOr containerised, in front of a real engine:
cd dio-serve && docker compose up # Ollama + DIO on :8085Then point any OpenAI / Ollama client at it:
curl http://127.0.0.1:8085/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"default","messages":[{"role":"user","content":"hello"}]}'
# -> X-DIO-Backend: gpu-a (which engine served it)/health reports degraded when a backend is out of rotation; /debug/metrics
exposes the learned slopes, affinity hit rate and admission counters.
dio demo runs eight agent sessions over three mock engines, throttles one of them
mid-run, then replays the same workload through three routers:
router | mean ms after the throttle | engine switches / session |
round-robin (Nginx default) | 1994 | 10.9 |
sticky round-robin ( | 1453 | 0.0 |
DIO | 1328 | 1.0 |
DIO moves traffic off the engine that degraded while keeping prefix affinity: it
takes most of the stickiness win without pinning half the fleet to a sick
backend. Mock engines, so this is a behaviour demo rather than a hardware
benchmark - full output in
docs/launch/evidence/swarm-demo-postfix.txt.
git clone https://github.com/nisaral/DIO.git
cd DIO/dio-serve
pip install -e .
# 1. Auto-discover local engines (Ollama, vLLM, SGLang) & create dio.yaml:
dio init
# 2. Start the gateway:
dio serve
# Or run zero-GPU instant demo:
dio demoPoint OpenAI SDK / LangChain at http://localhost:8085/v1
Point Ollama CLI / OpenWebUI at http://localhost:8085/api (or OLLAMA_HOST=http://localhost:8085)
Connect Cursor / Claude Desktop / VS Code via MCP: dio mcp
Full docs → dio-serve/README.md · Config-as-Code → dio.example.yaml · Architecture → docs/ARCHITECTURE.md · API → docs/API.md
Clients (OpenAI SDK / curl / LangChain)
│
▼
┌────────────────┐
│ DIO Gateway │ dual-timescale NLMS
│ :8085 /v1/* │ joint cost · affinity · admission
└───────┬────────┘
HTTP OpenAI API + /metrics scrape (engines unmodified)
┌───────┴────────┬────────────┐
▼ ▼ ▼
vLLM GPU0 vLLM GPU1 SGLang / TGI / OllamaDIO does not own kernels, KV caches, or continuous batching. Those stay in vLLM. DIO owns placement, learning, and admission — add GPUs by adding URLs.
Hybrid cost routing. Each backend is scored
S_w = wait_w + ŷ_w + tierCost + vramCost + c_kv·KV_w + c_q·W_w − cacheBonus_w − c_p·Hit_w.
The latency term ŷ_w = s_w·N + b_w comes from a dual-timescale NLMS filter
(s_eff = α·s_fast + (1−α)·s_slow) that tracks fast shifts without letting noise
destabilize the slow estimate. O(1) per update, no training step.
Engine-metric fusion. Scrapes each vLLM's /metrics for KV-cache utilization,
waiting-request count, and prefix-hit rate. Read-only.
Session and prefix affinity. Multi-turn conversations stay on the replica already holding their shared prefix, so the KV cache is reused rather than rebuilt. Hit rate and stickiness are exported, not assumed.
Admission decoupled from absolute prediction. Three modes — rank_only,
empirical (default), absolute (diagnostic only; see below).
A tempting design is to learn one latency model and use it for two jobs: ranking replicas, and gating admission against an SLO. The second job does not hold.
At MAPE ≈ 90–130%, the textbook policy "reject if min ŷ > SLO" rejects ~40% of a load that rank-relative and empirical gating complete in full. A model accurate enough to order replicas can be far too coarse to threshold on.
That is why absolute mode ships as a diagnostic and the default is empirical
(rolling observed percentile). Keeping ranking and admission as separate policies is a
correctness requirement for any predictive gateway built on black-box telemetry, not
an implementation detail.
Dual Tesla T4, Qwen2.5-3B-Instruct, two stock vLLM replicas, n=10 seeds.
Scenario | Result |
Matched backends | Near Round-Robin parity — no manufactured win |
Controlled ×2 service-time asymmetry | p99 −48.3% ± 0.7% vs RR; beats d=2 RLS |
Real multi-turn | p99 improves on 8/10 seeds (median 51.8%, Wilcoxon p ≈ 0.01) |
Session stickiness | 1.00 vs RR 0.50; 0.998 vs 0.75 under concurrent load |
Admission, tight SLO |
|
A caveat we state up front. In the multi-turn suite the two NLMS arms differ in
two knobs at once (engine-metric fusion and the affinity cache bonus), so that
margin is a joint effect. A decomposition over 100 live snapshots puts the
affinity bonus 194× above the scraped gauge terms on this hardware, so the gain
belongs to affinity, not the gauges — the reverse of how earlier drafts of our own
work read the same data. Reproduce:
scripts/audit_g1_confound.py.
Scope. Two T4s and a 3B model at low concurrency, not an A100/H100 fleet. The scraped gauge terms contribute little here precisely because queues barely diverge at that operating point; expect them to matter with deeper concurrency, more replicas, or heterogeneous peers.
The router was exercised the way it will actually be used: several agents holding
their own multi-turn sessions against a live gateway
(examples/swarm_live_stack.py), over OpenAI JSON, OpenAI SSE and Ollama ndjson at
the same time -- ~700 requests, including 80-way and 300-way bursts.
examples/agent_swarm_demo.py replays that workload offline, three ways, while the
fastest engine is throttled 8x part-way through (--agents 8):
router | engine switches / session | mean latency (after the throttle) | session affinity |
round-robin (per request, Nginx default) | 10.8 | 1990 ms | -- |
sticky round-robin ( | 0.0 | 1438 ms | -- |
DIO | 1.2 | 1298 ms | 86% |
Raw per-run metrics, the verification tables and the repro commands live in
docs/launch/evidence/.
Read that honestly. The whole-run p95/p99 is dominated by whatever was in flight when the engine degraded, so the mean and the traffic share are the columns that carry information; DIO keeps sessions pinned (1.2 switches vs 10.8) and still holds the best post-throttle mean. It also keeps sending some work to a degraded engine when the affinity bonus outweighs the latency gap -- a deliberate trade, because a warm prefix cache is worth real milliseconds, not a bug. The engines are local behaviour mocks, so none of this is a hardware performance claim.
Using it for real is also how most of the bugs in
CHANGELOG.md were found, e.g.:
streaming requests were sent to token-gated engines without the engine's API key, so every stream 401'd while the JSON path worked;
failed requests were counted as successes, so goodput_fraction reported 1.00
on a run where half the requests failed;
POST /debug/backends accepted any base_url, which made the admin plane an
SSRF primitive (http://169.254.169.254/... now returns 400);
max_tokens: 10**9 was forwarded unbounded and held an engine until the client
gave up, while negative max_tokens and empty messages were accepted.
A second round -- four agents on one live gateway, ~1000 requests over OpenAI JSON, OpenAI SSE, Ollama ndjson, plus 80/200/500-way storms -- found the rest:
Ollama streaming reported estimated token counts while the JSON path reported the
engine's, so the same prompt produced two different answers depending on stream;
the gateway now requests stream_options.include_usage (and retries once without it
for engines that reject the field);
one absurd max_tokens made the learned prediction ~10^9 ms and left a permanent
crater in mae/mape; the routing feature is now clamped (token_feature_cap),
never the forwarded request;
streamed tool calls were dropped and done_reason was hardcoded to stop;
an unknown model returned 200 and was served anyway -- now 404 model_not_found;
unparseable JSON returned a framework 422 -- now OpenAI's 400 envelope;
in a 500-wide storm at p95 ~29 s against a 5 s budget, nothing ever told the
client it was over budget (no 503, no header). Every response now carries
X-DIO-Budget-Ms / X-DIO-Over-Budget (and X-DIO-Predicted-Ms on streams).
All in dio-serve/scripts/:
Script | Produces |
| G1–G3 dual-T4 suites (hybrid, affinity, admission) |
| The 194× decomposition behind the caveat above |
| NLMS vs d=2 recursive least squares |
| Controlled service-time asymmetry (Regime C) |
| 2×2 factorial on mock engines, no GPU required |
| Coefficient and ablation checks |
Runbooks: scripts/GPU_ABC_RUNBOOK.md and
scripts/GPU_CLUSTER_RUNBOOK.md.
Raw multi-seed outputs are committed under dio-serve/results_*/.
Every field is settable by flag or DIO_-prefixed env var
(src/dio/config.py); paper defaults shown.
Setting | Default | Meaning |
|
|
|
|
| Session/prefix affinity bonus |
|
| Admission budget |
|
| Scrape vLLM |
|
|
|
|
|
|
|
|
|
Live introspection while running: /debug/predictions, /debug/affinity,
/debug/admission, /debug/engine, /debug/workers.
This repo holds two separate systems. The paper is about dio-serve/ only.
Path | What it is |
The paper's system. Python/FastAPI gateway over stock vLLM. Start here. | |
Earlier, unrelated prototype: Go control plane, BoltDB, gRPC to a Python data plane. Not used for any result in the paper. | |
Manuscript sources (Springer submission). | |
Paper figures. |
Cite the paper, not the software, once the preprint is announced:
@misc{dio2026,
title = {DIO: Hybrid Cost Routing, Session Affinity, and Calibration-Robust Admission
for Multi-Instance LLM Serving over Stock vLLM},
author = {Nisar, Keyush and Parikh, Krishil and Maisheri, Krisha and
Gawade, Aruna and Rathod, Nilesh T. and Florence A, Angelin},
year = {2026},
doi = {10.5281/zenodo.22085398},
url = {https://github.com/nisaral/DIO},
note = {Software and experimental artifact release}
}⭐ Star this repo if a predictive, non-invasive gateway is useful to you — it is how the next person running a vLLM/Ollama fleet finds it.
🐛 Bug reports with a reproduction go straight into the regression suite: Issues.
🔧 Contributing: start with CONTRIBUTING.md.
💬 Design questions: Discussions.
Apache-2.0 — see dio-serve/LICENSE.
Get live telemetry from the DIO cluster, including learned worker slopes, KV-cache pressure, and admission goodput statistics.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. 'Get live telemetry' reasonably implies a side-effect-free read and hints at the return content, but it does not disclose access requirements, rate limits, or data freshness. It discloses the essential read-only nature without contradiction or deeper detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler; the action and scope appear first, followed by three distinct telemetry categories that each earn their place. This is appropriately sized for a 0-parameter tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 0-parameter read tool this is nearly complete: it states the action, the scope, and the three telemetry categories, which is most of what an agent needs to decide to call it. It omits any hint of the response shape or access prerequisites, but with no output schema and no annotations, the description covers the essentials well.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With zero parameters there is nothing for the description to explain, earning it the 0-params baseline of 4. The metric list adds useful context about the response, though no parameter semantics are relevant.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get') with a clear resource ('live telemetry from the DIO cluster') and enumerates three concrete metric types (learned worker slopes, KV-cache pressure, admission goodput). This content clearly separates it from the sibling tools, which by name cover model listing, latency prediction, and prompt routing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is given about when to use this tool versus dio_get_models, dio_predict_latency, or dio_route_prompt. The description states only what the tool does, not when an agent should prefer it over alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query DIO gateway for available LLM models, active backends, and cluster health.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of behavioral disclosure. The verb 'Query' implies a read-only operation, and the listed result categories give some sense of what the caller receives. However, it does not disclose any potential side effects, permission requirements, rate limits, or error behavior, though the operation appears safely read-oriented.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that names the gateway and the three key result categories without wasted words. It is concise and readable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the zero-parameter call, the description provides enough to invoke the tool, but the overlap with sibling dio_cluster_status on 'cluster health' creates ambiguity. Without an output schema, it also leaves the exact return shape unspecified, though the listed result categories mitigate this.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is no parameter semantic burden on the description. The baseline of 4 applies because no parameter documentation is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action ('Query DIO gateway') and the resources/result categories: available LLM models, active backends, and cluster health. It is specific enough to understand what the tool does, but it does not distinguish itself from sibling dio_cluster_status, which likely covers the health portion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus its siblings, particularly dio_cluster_status for cluster health or dio_route_prompt/dio_predict_latency for other concerns. The description implies a general querying purpose but does not state exclusions or alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get latency and cost predictions before sending requests. Predicts queue delay, execution latency, and optimal backend using DIO's dual-timescale NLMS filter.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | Target model name (optional) | |
| prompt | Yes | The prompt text or query to estimate | |
| tokens | No | Estimated token count (optional) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It frames the tool as a non-executing predictor and identifies its method and outputs, implying no request is actually sent. It stops short of stating auth requirements or response format, but nothing contradicts the non-destructive nature of the operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler: the first states the core value proposition, the second enumerates outputs and method. Every clause earns its place and the description avoids repeating the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple predictor with one required string and two optional parameters, the description provides what an agent needs: what is predicted, when to call it, and the kind of result to expect. It could name the routing sibling or specify units, but those are minor gaps given the tool's low complexity and no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so all parameters are already documented; the description adds no extra meaning about how model or tokens influence the prediction. The baseline of 3 applies because the schema carries the semantic weight.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description anchors on a concrete action ('Get') and object ('latency and cost predictions'), then lists specific outputs: queue delay, execution latency, and optimal backend. These outputs clearly distinguish it from sibling tools that list models, report cluster status, or route prompts.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly states when to use it: 'before sending requests,' establishing this as a pre-flight prediction step. It doesn't explicitly name dio_route_prompt as the alternative for actually dispatching, but the predictive-vs-executive distinction is strongly implied; no exclusions are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Route a prompt through DIO's intelligent NLMS scheduler to the optimal backend and return the completion.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | Target model name (optional) | |
| prompt | Yes | Prompt text to send | |
| max_tokens | No | Maximum tokens to generate (default: 256) | |
| temperature | No | Sampling temperature (default: 0.7) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It only states the routing action and return of a completion. It does not reveal potential side effects, error conditions, or what 'optimal backend' entails. This is insufficient for a tool with no safety profile.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with the primary action. It is concise with no redundant content. However, it uses the jargon 'NLMS' without explanation, which slightly reduces clarity but does not harm structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description should clarify the return format. It only says 'returns the completion' without specifying structure, pagination, or error behavior. For a tool with four parameters and no annotations, this is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of parameters with descriptions, so the schema already documents prompt, model, max_tokens, and temperature. The description adds no additional semantics beyond the schema; it does not explain parameter interactions or defaults beyond what the schema states. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action: 'Route a prompt through DIO's intelligent NLMS scheduler to the optimal backend and return the completion.' It clearly identifies the verb, resource, and outcome. It differentiates from siblings implicitly—dio_get_models, dio_predict_latency, dio_cluster_status—by focusing on routing and completion, though it does not explicitly name them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus its siblings. It does not mention conditions, alternatives, or exclusions. An agent cannot infer when to choose this over dio_get_models or dio_predict_latency from the description alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0dio_cluster_statusdio_get_modelsdio_predict_latencydio_route_promptScored across 4 tools
The tools have mostly distinct purposes: listing models, predicting latency, routing prompts, and checking cluster telemetry. Minor overlap exists between dio_get_models and dio_cluster_status since both mention health, but their descriptions clarify that one is model/backend inventory and the other is deep operational telemetry.
Three tools follow a clear verb_noun pattern (get_models, predict_latency, route_prompt), while dio_cluster_status deviates by omitting the verb. The shared 'dio_' prefix and overall readability keep the set mostly consistent.
With 4 tools, the server is well-scoped for an inference orchestrator. Each tool covers a distinct, necessary capability (discover, predict, route, monitor) without unnecessary bloat.
The core workflow of checking available models, predicting latency, routing a prompt, and inspecting cluster health is fully covered. Minor gaps exist such as detailed per-model information or explicit policy management, but these are not critical for the server's stated purpose.
AI routing, memory, guardrails, and governance. Routes across Claude, GPT, Gemini.
LLM Observability & Orchestration Agent
Enterprise AI Control Plane: governance, guardrails, spend tracking, compliance & smart routing.
AI model routing on your own vendor keys: pick the best model per prompt, or route and run it.
