Which integrations are available for this server?

Provides cloud fallback to Google Gemini for large-context ingestion or complex reasoning tasks when local execution is not sufficient. Routes local inference through Ollama, automatically selecting and sizing models based on detected VRAM to handle routine coding and QA tasks at zero token cost. Provides cloud fallback to OpenAI frontier models and exposes an OpenAI-compatible API for integrating with existing OpenAI-based clients.

How do I use CascadeGateway?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@CascadeGateway Refactor this Python function locally and draft unit tests for it" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

CascadeGateway

by jadberro

Overview Schema Source Related Servers Discussions Admin

Python

Local

CascadeGateway ⚡

Hardware-Adaptive Local LLM & Cloud Cascading Architecture

License: MIT Python: 3.10+ Ollama Compatible OpenAI Compatible Protocol

CascadeGateway is an intelligent, hardware-adaptive cascading proxy that routes AI queries between local GPUs (via Ollama at $0 token cost) and frontier cloud models (Google Gemini, OpenAI, etc.).

It automatically detects your GPU hardware and physical VRAM on startup—whether running a flagship RTX 5090 (32GB), RTX 4090 (24GB), mainstream RTX 3080 (10GB), or Apple Silicon—and dynamically selects and sizes the optimal models without requiring manual reconfiguration.

flowchart TD
    Client["AI Client: Cursor / Continue / Antigravity"] --> Gateway["CascadeGateway :8000/v1"]

    Gateway --> Phase1{"Phase 1: Structural Guard"}
    Phase1 -->|"Context > 32k or Tools"| CloudFallback["Cloud Fallback: Gemini"]
    Phase1 -->|"Passed"| Phase2{"Phase 2: Lexical Automaton &lt;1ms"}

    Phase2 -->|"Architecture / Reasoning"| LocalArchitect["Local Architect: DeepSeek-R1 / Gemma 4 (RTX 5090)"]
    Phase2 -->|"Coding / Tests / Routine"| LocalBuilder["Local Builder: Qwen 2.5 Coder 32B (RTX 5090)"]

    LocalArchitect --> Phase3{"Phase 3: 3-Token Lookahead"}
    LocalBuilder --> Phase3
    Phase3 -->|"Stalled / Timeout"| CloudFallback
    Phase3 -->|"Healthy Stream"| LoopBreaker["Sliding-Window Loop Breaker"]
    LoopBreaker --> ClientStream["SSE Stream to Client"]

    CloudFallback --> ClientStream

    VerifyRoute["Asymmetric Verification: /verify"] --> LocalDraft["Local Generator: RTX 5090"]
    LocalDraft --> CloudCritic["Cloud Critic: Gemini Flash"]
    CloudCritic --> ClientStream

    Gateway -. Telemetry .-> HUD["🖥️ Always-On-Top Desktop HUD"]

🚀 Key Features

🖥️ Always-On-Top Desktop HUD: Sleek, frameless, draggable obsidian overlay (#0b0f19) that sits above your IDEs next to the clock or anywhere on screen. Displays live routing decisions (🟢 Local Builder, 🟣 Local Architect, 🔵 Asymmetric Verify, 🟠 Gemini Cloud), real-time VRAM gauge (GB, %, °C, W), decision latency (sub-5ms), prompt snippet, dollar savings counter, pin toggle, compact mini-mode, and 1-click mode switcher buttons.
🧠 Local GPU Architectural Reasoning: True multi-model orchestration on local silicon. System design, race condition, thread safety, and algorithmic queries are dispatched directly to the local reasoning model (deepseek-r1:14b or gemma4:26b) on the RTX 5090 rather than escaping to cloud tokens. Routine code generation is routed to qwen2.5-coder:32b.
Hardware-Adaptive VRAM Sizing: Automatically profiles your GPU (nvidia-smi / Metal / CPU) and assigns the optimal model tier.
Autonomous $0 Offload: Resolves 85–95% of routine coding, unit tests, refactoring, and QA directly on local VRAM.
Dynamic Token Biasing Engine:
- Adaptive Mode: Automatically increases local GPU bias as cloud token budgets get consumed.
- Manual Slider: Fine-tune local preference from 0% (Quality First) to 100% (Maximum Local Offload).
- Strict Local: 100% execution on local VRAM with automatic cloud overflow only if context exceeds physical limit.
🧠 Multi-Task Workflow Modes & Interactive Review Gate:
- 🧠 Architect & Builder: Reasoning model drafts architecture, data contracts, and edge cases → Pauses at a Human Review Gate → You approve or refine → Builder synthesizes production code.
- 🚀 Solo Sprint: Instant local coding via qwen2.5-coder:32b for quick functions, tests, and scripts.
- 🌐 Deep Context: Gemini 2.5 Flash ingests massive repository files (1M context) → modular execution on local 5090.
- 🔬 Math & Algo Proof: Deep Chain-of-Thought formal verification for cryptography and concurrency algorithms.
- 🛡️ Asymmetric Verification (/verify): High-speed local draft synthesis on RTX 5090 ($0) combined with an anonymous strict cloud auditor (Gemini) providing production-grade critique and code enhancement without leaking full history.
🛡️ Sub-5ms Intelligent Routing & 3-Token Lookahead Failover: Three-phase classification pipeline: Phase 1 Structural & VRAM Budget Validation (<1ms, prevents swapping to system RAM), Phase 2 Single-Pass Lexical Scan (<1ms), and Phase 3 Resilient Streaming with an in-memory 3-token lookahead buffer that transparently replays stalled local requests to Gemini Cloud without dropping client connections or throwing IDE error popups.
🔁 Anti-Hallucination Sliding-Window Loop Breaker: In-flight ring buffer tracking token emission sequences (lengths 2, 3, 4 repeated ≥ 4×), terminating runaway generative loops immediately.
⚡ 1-Click IDE Auto-Configuration: Automated zero-friction setup endpoint (/api/ide/auto-config) and UI button for VSCode Continue (~/.continue/config.json) and Cursor.
💰 Zero-Surprise Dollar Savings Telemetry: Hardware-native token accounting (eval_count) calculating real-time dollar savings based on commercial frontier rates ($3.00/1M tokens).
⚡ Persistent GPU VRAM Residency & Zero-Cold-Start: Models remain permanently resident in GPU VRAM (keep_alive: -1) across idle periods and tasks, ensuring instant zero-delay responses. Includes manual 1-Click "Pause GPU" (instant VRAM purge for AAA gaming) and "Warm GPU" (preloads weights into VRAM).
Native Model Context Protocol (MCP): Exposes a high-performance Streamable HTTP and Stdio MCP endpoint for Google Antigravity and Claude Desktop.
🎛️ Dynamic Model Selection: Select any installed local model for both the Architect role (deepseek-r1:14b, gemma4:26b, etc.) and the Builder role (qwen2.5-coder:32b, etc.) directly from the Web UI toolbar or Windows Tray submenus. Automatically detects newly pulled models from Ollama without restarting the gateway.
Zero-Window Background Tray App: Runs silently in the system tray, boots with Windows/Linux, and includes single-instance mutex protection.
🎮 1-Click Pause & Free GPU (Instant VRAM Purge): Evicts loaded models from VRAM in <1s via a dedicated button on the Web UI, Windows Tray, or POST /api/models/unload. Frees 20–30+ GB of VRAM immediately for AAA gaming, Blender, or video editing without terminating the server. Models reload automatically on demand when coding.
Live Hardware Telemetry: In-browser control center showing real-time VRAM allocation, GPU power draw (W), temperature (°C), lifetime token savings, and an interactive prompt runner.
Standard OpenAI-Compatible API: Seamless drop-in replacement (/v1/chat/completions) for Cursor, VSCode (Continue.dev), Aider, Claude Dev, and custom scripts.

Related MCP server: local-mmcp

🧠 The Interactive Review Gate

sequenceDiagram
    autonumber
    actor Dev as You (Developer)
    participant Arch as 🧠 Architect - DeepSeek-R1 / Gemma 4
    participant Gate as 🛑 Review Gate - You in the Loop
    participant Build as ⚡ Builder - Qwen 2.5 Coder 32B

    Dev->>Arch: "Add rate-limiting and burst protection to API routes"
    Note over Arch: Deep CoT: Identifies race conditions,<br/>evaluates Redis vs in-memory,<br/>drafts interface contracts.
    Arch->>Gate: Presents Architectural Blueprint + Edge Cases
    Note over Gate: PAUSE: No code written yet.<br/>You review the proposed interfaces & strategy.

    alt If you want adjustments
        Dev->>Gate: "Use Redis, and add IPv6 CIDR subnet matching"
        Gate->>Arch: Quick amendment - 100 tokens
        Arch->>Gate: Updated spec
    end

    Dev->>Gate: "Approve & Build"
    Gate->>Build: Sends final structured specification
    Note over Build: Zero ambiguity.<br/>High-speed code synthesis - 70 t/s.
    Build-->>Dev: Delivers complete implementation + unit tests - $0 Cost

📊 VRAM Hardware Sizing Matrix

Hardware Tier	Supported GPUs	VRAM	Recommended Coding Model	General / Reasoning	Expected Speed
Tier 1 (Flagship)	RTX 5090, 4090, 3090, A6000	24 GB – 32 GB	`qwen2.5-coder:32b`	`gemma4:26b` / `deepseek-r1:32b`	60 – 75+ tokens/sec
Tier 2 (Enthusiast)	RTX 4080, 4070 Ti, 4070, 3080 12GB	12 GB – 16 GB	`qwen2.5-coder:14b`	`mistral-small:22b-q4`	45 – 55+ tokens/sec
Tier 3 (Mainstream)	RTX 3080 10GB, 3070, 4060 Ti, 2080 Ti	8 GB – 10 GB	`qwen2.5-coder:7b`	`llama3.1:8b`	85 – 100+ tokens/sec
Tier 4 (Entry / CPU)	RTX 3050, 4060 laptop, Apple Silicon, CPU	< 8 GB	`qwen2.5-coder:1.5b` / `3b`	`llama3.2:3b`	40 – 70 tokens/sec

⚡ Quick Start

Prerequisites

Ollama installed and running on your machine.
Python 3.10+ installed.
(Optional) A free API key from Google AI Studio for cloud fallback.

Windows (One-Click)

git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
scripts\setup.bat

Creates .venv, installs dependencies, auto-generates your desktop shortcut, and places an auto-start shortcut in your Windows Startup menu.

Linux / macOS

git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
chmod +x scripts/setup.sh
./scripts/setup.sh

🛠️ Connecting Your IDEs & Tools

Once running, the control center is available at http://127.0.0.1:8000/.

1. Google Antigravity

Add to ~/.gemini/config/mcp_config.json:

{
  "mcpServers": {
    "rtx5090-cascade": {
      "serverUrl": "http://127.0.0.1:8000/mcp"
    }
  }
}

Antigravity automatically detects the toolset (query_local_5090, cascade_llm, get_cascade_metrics) and dynamically dispatches code generation directly to your GPU.

2. Cursor

Navigate to Settings > Models > OpenAI API Key.
Set Base URL: http://127.0.0.1:8000/v1
Set API Key: cascading-local (any string)
Model: cascade-auto

3. Continue.dev (`config.json`)

{
  "models": [
    {
      "title": "Cascade Gateway (Local + Cloud)",
      "provider": "openai",
      "model": "cascade-auto",
      "apiBase": "http://127.0.0.1:8000/v1",
      "apiKey": "cascading-local"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Local Autocomplete",
    "provider": "openai",
    "model": "cascade-auto",
    "apiBase": "http://127.0.0.1:8000/v1",
    "apiKey": "cascading-local"
  }
}

4. Claude Desktop (`claude_desktop_config.json`)

{
  "mcpServers": {
    "local-cascade": {
      "serverUrl": "http://127.0.0.1:8000/mcp"
    }
  }
}

5. Python OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="cascading-local"
)

response = client.chat.completions.create(
    model="cascade-auto",
    messages=[
        {"role": "user", "content": "Write an optimized LRU cache in Python."}
    ]
)
print(response.choices[0].message.content)

📁 Repository Architecture

CascadeGateway is structured as a modern, modular Python package:

CascadeGateway/
├── src/
│   └── cascadegateway/
│       ├── core/                  # Hardware profiler, model sizing & routing engine
│       │   ├── config.py          # YAML config & environment loader
│       │   ├── hardware.py        # GPU VRAM auto-profiling (5090/4090/etc.)
│       │   ├── classifier.py      # Phase 1 structural validation & Phase 2 regex automaton (<1ms)
│       │   ├── streaming.py       # Phase 3 lookahead failover, loop breaker & asymmetric audit
│       │   └── router.py          # Biasing engine, metrics & model resolution
│       ├── api/                   # Modular FastAPI endpoints
│       │   ├── server.py          # App initialization, SSE streaming & /v1/chat/completions
│       │   ├── pipeline.py        # Architect, Review Gate & Builder endpoints
│       │   └── models.py          # Pause & Free GPU (VRAM purge) & 1-click IDE auto-config
│       ├── web/                   # Clean decoupled web assets & responsive UI
│       │   ├── templates/
│       │   │   └── dashboard.html # Responsive HTML5 dashboard & real-time telemetry
│       │   └── static/
│       │       ├── css/dashboard.css
│       │       └── js/dashboard.js
│       ├── hud/                   # Always-On-Top Desktop HUD & Telemetry Overlay
│       │   ├── __init__.py
│       │   └── overlay.py         # Tkinter floating dark obsidian window
│       ├── tray/                  # Windows system tray controller
│       │   └── app.py             # Pystray background service & menus
│       ├── mcp/                   # Model Context Protocol endpoints
│       │   ├── __init__.py
│       │   └── server.py          # Antigravity & Claude Desktop integration
├── scripts/                       # Platform launchers & utilities
│   ├── start_cascade.bat          # Windows batch launcher
│   ├── start_tray.vbs             # Silent windowless tray runner
│   ├── start_hud.bat              # Desktop HUD launcher
│   ├── start_hud.vbs              # Silent windowless HUD launcher
│   ├── setup.bat / setup.sh       # One-click installers
│   └── create_shortcuts.ps1       # Desktop & startup shortcut generator
├── tests/                         # Comprehensive automated test suites
│   ├── test_gateway.py            # End-to-end API integration tests
│   ├── test_resilience.py         # Sub-5ms SLA, structural validation & lookahead tests
│   └── test_new_features.py       # Loop breaker, IDE auto-config, dollar savings & /verify tests
├── assets/                        # Icons & diagrams
├── config.yaml                    # Gateway configuration
├── pyproject.toml                 # Modern pip/uv packaging metadata
└── requirements.txt

📡 API Reference

Endpoint	Method	Description
`/v1/chat/completions`	`POST`	OpenAI-compatible chat completion endpoint supporting resilient SSE streaming, lookahead failover, loop breaking, and diagnostic headers (`X-Cascade-*`).
`/v1/models`	`GET`	Returns available virtual model aliases (`cascade-auto`, `local-5090`) and physical Ollama models.
`/api/hud/state`	`GET`	Desktop HUD Telemetry: Real-time status, active route, model, VRAM used/total, temp (°C), power (W), latency, and dollar savings.
`/api/hardware`	`GET`	Real-time GPU telemetry: VRAM allocation, temperature, power draw (W), and detected hardware tier.
`/api/models/unload`	`POST`	Instant VRAM Purge ("Pause GPU"): Evicts loaded models to 0 MB VRAM in <1s for AAA gaming or rendering.
`/api/models/preload`	`POST`	Warm GPU: Preloads and pins model into VRAM with indefinite residency (`keep_alive: -1`).
`/api/ide/auto-config`	`POST`	1-Click IDE Setup: Injects CascadeGateway configuration into `~/.continue/config.json` and detects Cursor environments.
`/api/workflow/mode`	`POST`	Sets active workflow mode (`architect`, `builder`, `solo`, `verify`, `deep_context`, `math`).
`/metrics`	`GET`	Telemetry: exact hardware token accounting (`eval_count`), local offload percentage, and real-time dollar savings.
`/mcp`	`POST`	Model Context Protocol streamable HTTP endpoint for Google Antigravity and Claude Desktop.
`/`	`GET`	Interactive browser control center, real-time hardware gauges, and prompt workbench.

⚙️ Configuration (`config.yaml`)

server:
  host: "127.0.0.1"
  port: 8000

local:
  provider: "ollama"
  base_url: "http://127.0.0.1:11434"
  # Set empty to let the hardware profiler auto-select the best model for your GPU
  primary_model: ""
  max_context_tokens: 32768
  timeout_seconds: 120

cloud:
  provider: "gemini"
  default_model: "gemini-2.5-flash"
  pro_model: "gemini-2.5-pro"
  max_context_tokens: 1048576

biasing:
  mode: "adaptive"        # "adaptive", "manual", or "local_only"
  bias_factor: 0.35       # Baseline local bias (0.0 to 1.0)
  cloud_token_budget: 500000
  allow_context_overflow_to_cloud: true

📜 License

This project is licensed under the MIT License.

Available Tools

3 tools

cascade_llmB

Run a query through the intelligent cascading gateway (RTX 5090 -> Gemini) with dynamic biasing.

ParametersJSON Schema

Name	Required	Description	Default
`prompt`	Yes	The prompt to process.
`system_prompt`	No	Optional system prompt.	You are an expert coding assistant.

TDQS

B3.1/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It does disclose a genuine behavioral trait — that requests route local-first (RTX 5090) and then fall through to Gemini — but leaves 'intelligent'/'dynamic biasing' undefined and says nothing about fallback triggers, latency/cost tradeoffs, or data leaving the local machine.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no wasted framing, though 'intelligent' and 'dynamic biasing' are unearned filler that consumes space without informing the agent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations and no output schema, the description should explain more about the gateway's behavior and what a caller gets back (e.g., which model answered, fallback signals). As written, an agent cannot tell what distinguishes a cascaded response from a local one.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% with only two self-explanatory parameters, so the schema does the heavy lifting; baseline 3 applies. The description adds no format or content guidance for 'prompt' or 'system_prompt' beyond what the schema already states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a concrete verb and resource ('Run a query through the intelligent cascading gateway') and the parenthetical (RTX 5090 -> Gemini) clarifies that this is the multi-tier path, implicitly distinguishing it from query_local_5090. However, the phrase 'dynamic biasing' is undefined jargon, and the sibling is never named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use guidance, no conditions under which the cascade should be preferred over query_local_5090, and no mention of when to reach for get_cascade_metrics instead. The cascade chain is implied usage at best.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_cascade_metricsA

Get live metrics on tokens saved locally on the RTX 5090, local offload percentage, and biasing state.

ParametersJSON Schema

Name	Required	Description	Default
No parameters

TDQS

A3.5/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. 'Get live metrics' implies a read-only, non-destructive operation and it discloses the data categories returned, but it omits any mention of permissions, rate limits, caching behavior, or potential side effects. It provides minimal but useful behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that front-loads the core action ('Get live metrics') and then succinctly lists the specific metrics. There is no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simplicity (no parameters, no output schema) and the description's enumeration of returned metrics, it is mostly complete. It does not specify the exact format or units of the metrics, but for a straightforward read-only metrics endpoint this is sufficient for an agent to understand what it provides.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline of 4 applies. There are no parameter semantics to document, and the description does not need to compensate for schema gaps.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Get') and resource ('live metrics') and enumerates the exact quantities returned (tokens saved locally on the RTX 5090, offload percentage, biasing state). It clearly distinguishes the tool's purpose, though it does not explicitly differentiate from sibling query_local_5090.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is given on when to use this tool versus alternatives like query_local_5090 or cascade_llm. The usage is implied (retrieve metrics), but there are no conditions, prerequisites, or exclusions stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

query_local_5090A

Run a query or code task directly on the local NVIDIA RTX 5090 (qwen2.5-coder:32b) at $0 token cost.

ParametersJSON Schema

Name	Required	Description	Default
`prompt`	Yes	The prompt or coding task to execute on the RTX 5090.
`system_prompt`	No	Optional system prompt.	You are an expert coding assistant.

TDQS

A3.5/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden and does supply meaningful context: local execution, a specific model, and a zero-cost cost model. However it omits operational traits such as whether the machine must be online, latency/timeout expectations, and output format, leaving notable gaps for an unannotated tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence that conveys action, target, model, and cost with no filler. Every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter tool with no output schema and no annotations, the definition covers the essentials but not return shape or runtime expectations. It is adequate to invoke but not fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and both parameters (prompt, system_prompt) are fully documented in the schema. The description only restates the concept of a query/code task and adds no syntax, format, or constraint details beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Run) and resource (query or code task) and uniquely identifies the backend (local RTX 5090, qwen2.5-coder:32b). It implicitly contrasts with the cascade sibling via 'directly on the local', but never names cascade_llm, so sibling differentiation is only suggested rather than explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The '$0 token cost' framing hints at the condition that favors this tool over a paid/cascade path, but no explicit when-to-use, when-not-to-use, or named alternative (cascade_llm) is given. Usage is inferable rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

3 tool updatesv2.0.0Sep 12, 2026
- First observedcascade_llm
- First observedget_cascade_metrics
- First observedquery_local_5090

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.5/5.0

Scored 2026-09-12 05:20 across 3 tools

Disambiguation4/5

query_local_5090 and cascade_llm both execute queries, but the descriptions clearly distinguish local-only vs cascading backends. get_cascade_metrics is clearly a metrics tool. Minor overlap remains in the query execution purpose, but an agent can tell them apart.

Naming Consistency3/5

All names are snake_case, but the patterns are mixed: query_local_5090 starts with a verb, cascade_llm is a noun phrase, and get_cascade_metrics is verb_noun. Readable but not a predictable convention.

Tool Count5/5

Three tools are well-scoped for a cascading LLM gateway: local query, cascaded query, and metrics. Each tool has a clear role and the count avoids redundancy.

Completeness4/5

The core operations—run a local query, run a cascaded query, and monitor metrics—are covered. However, there is no tool to configure or set dynamic biasing, which is a minor gap for controlling gateway behavior.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

TokenOracle
com.guffeyholdings
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
Finance Monitoring
2
QuallaaOAuth
com.quallaa
Talk to your public-facing AI from any MCP client — Claude, ChatGPT, Cursor, Cline, Windsurf.
AI & Machine Learning Communication
2
Hive Compute A
io.github.srotzin
HiveCompute MCP Server — decentralized inference router for AI agents
Agent Orchestration AI & Machine Learning
2
Bay Run A
io.github.barneywohl
Free OpenAI-compatible inference with signed provenance receipts and 3 focused MCP tools.
AI & Machine Learning RAG Systems
31

Related MCP Servers

MCP LLM Integration Server
Developer Tools Autonomous Agents Code Execution
raptor7197
F
license
B
quality
D
maintenance
Enables integration of local LLM capabilities with MCP-compatible clients like Claude Desktop, Continue.dev, and Cline. Provides tools for processing text prompts through local language models using a customizable inference function.
Updated a year ago (2025-08-22 07:49 UTC)
2
1
-
local-mmcp
Image & Video Processing Audio Processing App Automation
rorojiao
F
license
A
quality
D
maintenance
Local multimodal MCP server for vision, document, audio, video, and GUI automation, all processed locally on Apple Silicon for privacy and security.
Updated 3 months ago (2026-06-10 12:34 UTC)
9
-
Local AI MCPofficial
AI & Machine Learning Developer Tools
TMHSDigital
A
license
A
quality
A
maintenance
Unified MCP server for managing local model runtimes (Ollama, LM Studio, etc.), enabling provider-agnostic discovery, lifecycle management, hardware-fit checks, and delegated inference.
Updated 13 days ago (2026-08-31 06:22 UTC)
16
18 npm
Creative Commons Attribution Non Commercial No Derivatives 4.0 International
llm-localfirst
AI & Machine Learning Agent Orchestration
shaxzodbek-uzb
A
license
Not graded
quality
B
maintenance
A local-first LLM routing MCP server that keeps sensitive data on your own models, with fail-closed privacy and manager-worker delegation, exposing route and complete tools to any MCP client.
Updated a month ago (2026-08-16 07:09 UTC)
MIT

CascadeGateway ⚡

Hardware-Adaptive Local LLM & Cloud Cascading Architecture

🚀 Key Features

🧠 The Interactive Review Gate

📊 VRAM Hardware Sizing Matrix

⚡ Quick Start

Prerequisites

Windows (One-Click)

Linux / macOS

🛠️ Connecting Your IDEs & Tools

1. Google Antigravity

2. Cursor

3. Continue.dev (config.json)

4. Claude Desktop (claude_desktop_config.json)

5. Python OpenAI SDK

📁 Repository Architecture

📡 API Reference

⚙️ Configuration (config.yaml)

📜 License

Available Tools

TDQS

TDQS

TDQS

Tool Schema Changelog

TDQS

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

MCP LLM Integration Server

local-mmcp

Local AI MCPofficial

llm-localfirst

3. Continue.dev (`config.json`)

4. Claude Desktop (`claude_desktop_config.json`)

⚙️ Configuration (`config.yaml`)