CascadeGateway
This MCP server exposes three tools for running LLM coding queries locally on an RTX 5090 or through a cascading local-to-cloud gateway, plus live telemetry.
query_local_5090 — Run a prompt or coding task entirely on the local NVIDIA RTX 5090 (
qwen2.5-coder:32b) at $0 token cost; accepts a requiredpromptand an optionalsystem_prompt(defaults to "You are an expert coding assistant.").cascade_llm — Route a prompt through the intelligent cascading gateway (RTX 5090 → Gemini) with dynamic biasing; accepts a required
promptand an optionalsystem_prompt.get_cascade_metrics — Retrieve live metrics on tokens saved locally on the RTX 5090, local offload percentage, and current biasing state; takes no input arguments.
Provides cloud fallback to Google Gemini for large-context ingestion or complex reasoning tasks when local execution is not sufficient.
Routes local inference through Ollama, automatically selecting and sizing models based on detected VRAM to handle routine coding and QA tasks at zero token cost.
Provides cloud fallback to OpenAI frontier models and exposes an OpenAI-compatible API for integrating with existing OpenAI-based clients.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@CascadeGateway Refactor this Python function locally and draft unit tests for it"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
CascadeGateway ⚡
Hardware-Adaptive Local LLM & Cloud Cascading Architecture
CascadeGateway is an intelligent, hardware-adaptive cascading proxy that routes AI queries between local GPUs (via Ollama at $0 token cost) and frontier cloud models (Google Gemini, OpenAI, etc.).
It automatically detects your GPU hardware and physical VRAM on startup—whether running a flagship RTX 5090 (32GB), RTX 4090 (24GB), mainstream RTX 3080 (10GB), or Apple Silicon—and dynamically selects and sizes the optimal models without requiring manual reconfiguration.
flowchart TD
Client["AI Client: Cursor / Continue / Antigravity"] --> Gateway["CascadeGateway :8000/v1"]
Gateway --> Phase1{"Phase 1: Structural Guard"}
Phase1 -->|"Context > 32k or Tools"| CloudFallback["Cloud Fallback: Gemini"]
Phase1 -->|"Passed"| Phase2{"Phase 2: Lexical Automaton <1ms"}
Phase2 -->|"Architecture / Reasoning"| LocalArchitect["Local Architect: DeepSeek-R1 / Gemma 4 (RTX 5090)"]
Phase2 -->|"Coding / Tests / Routine"| LocalBuilder["Local Builder: Qwen 2.5 Coder 32B (RTX 5090)"]
LocalArchitect --> Phase3{"Phase 3: 3-Token Lookahead"}
LocalBuilder --> Phase3
Phase3 -->|"Stalled / Timeout"| CloudFallback
Phase3 -->|"Healthy Stream"| LoopBreaker["Sliding-Window Loop Breaker"]
LoopBreaker --> ClientStream["SSE Stream to Client"]
CloudFallback --> ClientStream
VerifyRoute["Asymmetric Verification: /verify"] --> LocalDraft["Local Generator: RTX 5090"]
LocalDraft --> CloudCritic["Cloud Critic: Gemini Flash"]
CloudCritic --> ClientStream
Gateway -. Telemetry .-> HUD["🖥️ Always-On-Top Desktop HUD"]🚀 Key Features
🖥️ Always-On-Top Desktop HUD: Sleek, frameless, draggable obsidian overlay (
#0b0f19) that sits above your IDEs next to the clock or anywhere on screen. Displays live routing decisions (🟢 Local Builder, 🟣 Local Architect, 🔵 Asymmetric Verify, 🟠 Gemini Cloud), real-time VRAM gauge (GB, %, °C, W), decision latency (sub-5ms), prompt snippet, dollar savings counter, pin toggle, compact mini-mode, and 1-click mode switcher buttons.🧠 Local GPU Architectural Reasoning: True multi-model orchestration on local silicon. System design, race condition, thread safety, and algorithmic queries are dispatched directly to the local reasoning model (
deepseek-r1:14borgemma4:26b) on the RTX 5090 rather than escaping to cloud tokens. Routine code generation is routed toqwen2.5-coder:32b.Hardware-Adaptive VRAM Sizing: Automatically profiles your GPU (
nvidia-smi/ Metal / CPU) and assigns the optimal model tier.Autonomous $0 Offload: Resolves 85–95% of routine coding, unit tests, refactoring, and QA directly on local VRAM.
Dynamic Token Biasing Engine:
Adaptive Mode: Automatically increases local GPU bias as cloud token budgets get consumed.
Manual Slider: Fine-tune local preference from 0% (Quality First) to 100% (Maximum Local Offload).
Strict Local: 100% execution on local VRAM with automatic cloud overflow only if context exceeds physical limit.
🧠 Multi-Task Workflow Modes & Interactive Review Gate:
🧠 Architect & Builder: Reasoning model drafts architecture, data contracts, and edge cases → Pauses at a Human Review Gate → You approve or refine → Builder synthesizes production code.
🚀 Solo Sprint: Instant local coding via
qwen2.5-coder:32bfor quick functions, tests, and scripts.🌐 Deep Context: Gemini 2.5 Flash ingests massive repository files (1M context) → modular execution on local 5090.
🔬 Math & Algo Proof: Deep Chain-of-Thought formal verification for cryptography and concurrency algorithms.
🛡️ Asymmetric Verification (
/verify): High-speed local draft synthesis on RTX 5090 ($0) combined with an anonymous strict cloud auditor (Gemini) providing production-grade critique and code enhancement without leaking full history.
🛡️ Sub-5ms Intelligent Routing & 3-Token Lookahead Failover: Three-phase classification pipeline: Phase 1 Structural & VRAM Budget Validation (<1ms, prevents swapping to system RAM), Phase 2 Single-Pass Lexical Scan (<1ms), and Phase 3 Resilient Streaming with an in-memory 3-token lookahead buffer that transparently replays stalled local requests to Gemini Cloud without dropping client connections or throwing IDE error popups.
🔁 Anti-Hallucination Sliding-Window Loop Breaker: In-flight ring buffer tracking token emission sequences (lengths 2, 3, 4 repeated ≥ 4×), terminating runaway generative loops immediately.
⚡ 1-Click IDE Auto-Configuration: Automated zero-friction setup endpoint (
/api/ide/auto-config) and UI button for VSCode Continue (~/.continue/config.json) and Cursor.💰 Zero-Surprise Dollar Savings Telemetry: Hardware-native token accounting (
eval_count) calculating real-time dollar savings based on commercial frontier rates ($3.00/1M tokens).⚡ Persistent GPU VRAM Residency & Zero-Cold-Start: Models remain permanently resident in GPU VRAM (
keep_alive: -1) across idle periods and tasks, ensuring instant zero-delay responses. Includes manual 1-Click "Pause GPU" (instant VRAM purge for AAA gaming) and "Warm GPU" (preloads weights into VRAM).Native Model Context Protocol (MCP): Exposes a high-performance Streamable HTTP and Stdio MCP endpoint for Google Antigravity and Claude Desktop.
🎛️ Dynamic Model Selection: Select any installed local model for both the Architect role (
deepseek-r1:14b,gemma4:26b, etc.) and the Builder role (qwen2.5-coder:32b, etc.) directly from the Web UI toolbar or Windows Tray submenus. Automatically detects newly pulled models from Ollama without restarting the gateway.Zero-Window Background Tray App: Runs silently in the system tray, boots with Windows/Linux, and includes single-instance mutex protection.
🎮 1-Click Pause & Free GPU (Instant VRAM Purge): Evicts loaded models from VRAM in <1s via a dedicated button on the Web UI, Windows Tray, or
POST /api/models/unload. Frees 20–30+ GB of VRAM immediately for AAA gaming, Blender, or video editing without terminating the server. Models reload automatically on demand when coding.Live Hardware Telemetry: In-browser control center showing real-time VRAM allocation, GPU power draw (W), temperature (°C), lifetime token savings, and an interactive prompt runner.
Standard OpenAI-Compatible API: Seamless drop-in replacement (
/v1/chat/completions) for Cursor, VSCode (Continue.dev), Aider, Claude Dev, and custom scripts.
Related MCP server: Local AI MCP
🧠 The Interactive Review Gate
sequenceDiagram
autonumber
actor Dev as You (Developer)
participant Arch as 🧠 Architect - DeepSeek-R1 / Gemma 4
participant Gate as 🛑 Review Gate - You in the Loop
participant Build as ⚡ Builder - Qwen 2.5 Coder 32B
Dev->>Arch: "Add rate-limiting and burst protection to API routes"
Note over Arch: Deep CoT: Identifies race conditions,<br/>evaluates Redis vs in-memory,<br/>drafts interface contracts.
Arch->>Gate: Presents Architectural Blueprint + Edge Cases
Note over Gate: PAUSE: No code written yet.<br/>You review the proposed interfaces & strategy.
alt If you want adjustments
Dev->>Gate: "Use Redis, and add IPv6 CIDR subnet matching"
Gate->>Arch: Quick amendment - 100 tokens
Arch->>Gate: Updated spec
end
Dev->>Gate: "Approve & Build"
Gate->>Build: Sends final structured specification
Note over Build: Zero ambiguity.<br/>High-speed code synthesis - 70 t/s.
Build-->>Dev: Delivers complete implementation + unit tests - $0 Cost🏛️ Hybrid Architect-Builder Execution (Google Antigravity & CascadeGateway)
When running agentic coding assistants like Google Antigravity on a Pro plan, repeatedly shipping 20k to 50k tokens of repo file dumps, compiler logs, and multi-turn message history quickly exhausts 5-hour quota refill blocks.
CascadeGateway implements the native Hybrid Architect-Builder Pattern:
flowchart TD
subgraph Cloud ["☁️ Cloud Tier (Pro Plan Quota)"]
Architect["Cloud Architect: Gemini Pro / Claude<br/>• High-level system decomposition<br/>• Milestone roadmap planning<br/>• Concise diff reviews"]
end
subgraph Gateway ["⚡ CascadeGateway :8000/v1"]
Router["Context Pruning & Intent Router"]
Clamp["Terminal Clamping (>60 lines → 15 head + 30 tail)"]
Dedup["Repeat File Buffer Deduplication"]
end
subgraph Local ["🖥️ Local Workstation Tier (NVIDIA RTX 5090 - 32GB VRAM)"]
Builder["Local Builder (vLLM / Ollama): Qwen 2.5 Coder 32B<br/>• Code generation, unit tests, refactoring<br/>• Iterative bug fixes & lint diagnostics<br/>• $0 Cloud Token Cost"]
LocalMCP["Local MCP Tools<br/>• AST Symbol Lookup (@llmtooling/code-search-mcp)<br/>• Targeted File Slicing"]
end
Architect -->|"Milestone Specifications"| Router
Router --> Clamp
Clamp --> Dedup
Dedup -->|"Iterative Coding Tasks"| Builder
Builder <-->|"Precise Symbol Queries"| LocalMCP
Dedup -. Escalation on repeated test failures .-> Architect📉 Token Efficiency & Telemetry Audit Benchmark
Benchmarked using python -m cascadegateway.audit --compare baseline.jsonl optimized.jsonl:
Metric | Baseline (Cloud Pass-Through) | Optimized (Architect-Builder on 5090) | Impact |
Total Task Turns | 15 turns | 16 turns | Consistent scope |
Average Prompt Tokens / Turn | 38,200 tokens | 3,897 tokens | -89.8% context bloat |
Tokens Billed to Cloud Quota | 579,300 tokens | 3,550 tokens | -99.4% Cloud Token Reduction |
Tokens Absorbed by Local GPU | 0 tokens | 68,500 tokens | 95.1% Local RTX 5090 Execution Ratio |
Prefix Cache Hit Rate | 0.0% | 76.3% | Instantaneous prefill |
Average Latency | 1,450 ms | 800 ms | Sub-second response |
🚀 Cross-Platform Local Engine Runners
Windows Native (Zero-Window Flash):
powershell -File scripts\start_local_engine.ps1Auto-detects Ollama or vLLM, warms up
qwen2.5-coder:32bin 32GB VRAM withkeep_alive: -1.Linux / WSL (vLLM with Automatic Prefix Caching):
./scripts/start_local_vllm.shLaunches
vllmwith--enable-prefix-caching,--enable-chunked-prefill, and--tool-call-parser hermes.Antigravity SDK & Standalone Orchestration:
python scripts/run_architect_builder.py --task "Draft and build target module"
📊 VRAM Hardware Sizing Matrix
Hardware Tier | Supported GPUs | VRAM | Recommended Coding Model | General / Reasoning | Expected Speed |
Tier 1 (Flagship) | RTX 5090, 4090, 3090, A6000 | 24 GB – 32 GB |
|
| 60 – 75+ tokens/sec |
Tier 2 (Enthusiast) | RTX 4080, 4070 Ti, 4070, 3080 12GB | 12 GB – 16 GB |
|
| 45 – 55+ tokens/sec |
Tier 3 (Mainstream) | RTX 3080 10GB, 3070, 4060 Ti, 2080 Ti | 8 GB – 10 GB |
|
| 85 – 100+ tokens/sec |
Tier 4 (Entry / CPU) | RTX 3050, 4060 laptop, Apple Silicon, CPU | < 8 GB |
|
| 40 – 70 tokens/sec |
⚡ Quick Start
Prerequisites
Ollama installed and running on your machine.
Python 3.10+ installed.
(Optional) A free API key from Google AI Studio for cloud fallback.
Windows (One-Click)
git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
scripts\setup.batCreates .venv, installs dependencies, auto-generates your desktop shortcut, and places an auto-start shortcut in your Windows Startup menu.
Linux / macOS
git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
chmod +x scripts/setup.sh
./scripts/setup.sh🛠️ Connecting Your IDEs & Tools
Once running, the control center is available at http://127.0.0.1:8000/.
1. Google Antigravity
Add to ~/.gemini/config/mcp_config.json:
{
"mcpServers": {
"rtx5090-cascade": {
"serverUrl": "http://127.0.0.1:8000/mcp"
}
}
}Antigravity automatically detects the toolset (query_local_5090, cascade_llm, get_cascade_metrics) and dynamically dispatches code generation directly to your GPU.
2. Cursor
Navigate to Settings > Models > OpenAI API Key.
Set Base URL:
http://127.0.0.1:8000/v1Set API Key:
cascading-local(any string)Model:
cascade-auto
3. Continue.dev (config.json)
{
"models": [
{
"title": "Cascade Gateway (Local + Cloud)",
"provider": "openai",
"model": "cascade-auto",
"apiBase": "http://127.0.0.1:8000/v1",
"apiKey": "cascading-local"
}
],
"tabAutocompleteModel": {
"title": "Local Autocomplete",
"provider": "openai",
"model": "cascade-auto",
"apiBase": "http://127.0.0.1:8000/v1",
"apiKey": "cascading-local"
}
}4. Claude Desktop (claude_desktop_config.json)
{
"mcpServers": {
"local-cascade": {
"serverUrl": "http://127.0.0.1:8000/mcp"
}
}
}5. Python OpenAI SDK
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="cascading-local"
)
response = client.chat.completions.create(
model="cascade-auto",
messages=[
{"role": "user", "content": "Write an optimized LRU cache in Python."}
]
)
print(response.choices[0].message.content)📁 Repository Architecture
CascadeGateway is structured as a modern, modular Python package:
CascadeGateway/
├── src/
│ └── cascadegateway/
│ ├── core/ # Hardware profiler, model sizing & routing engine
│ │ ├── config.py # YAML config & environment loader
│ │ ├── hardware.py # GPU VRAM auto-profiling (5090/4090/etc.)
│ │ ├── classifier.py # Phase 1 structural validation & Phase 2 regex automaton (<1ms)
│ │ ├── streaming.py # Phase 3 lookahead failover, loop breaker & asymmetric audit
│ │ └── router.py # Biasing engine, metrics & model resolution
│ ├── api/ # Modular FastAPI endpoints
│ │ ├── server.py # App initialization, SSE streaming & /v1/chat/completions
│ │ ├── pipeline.py # Architect, Review Gate & Builder endpoints
│ │ └── models.py # Pause & Free GPU (VRAM purge) & 1-click IDE auto-config
│ ├── web/ # Clean decoupled web assets & responsive UI
│ │ ├── templates/
│ │ │ └── dashboard.html # Responsive HTML5 dashboard & real-time telemetry
│ │ └── static/
│ │ ├── css/dashboard.css
│ │ └── js/dashboard.js
│ ├── hud/ # Always-On-Top Desktop HUD & Telemetry Overlay
│ │ ├── __init__.py
│ │ └── overlay.py # Tkinter floating dark obsidian window
│ ├── tray/ # Windows system tray controller
│ │ └── app.py # Pystray background service & menus
│ ├── mcp/ # Model Context Protocol endpoints
│ │ ├── __init__.py
│ │ └── server.py # Antigravity & Claude Desktop integration
├── scripts/ # Platform launchers & utilities
│ ├── start_cascade.bat # Windows batch launcher
│ ├── start_tray.vbs # Silent windowless tray runner
│ ├── start_hud.bat # Desktop HUD launcher
│ ├── start_hud.vbs # Silent windowless HUD launcher
│ ├── setup.bat / setup.sh # One-click installers
│ └── create_shortcuts.ps1 # Desktop & startup shortcut generator
├── tests/ # Comprehensive automated test suites
│ ├── test_gateway.py # End-to-end API integration tests
│ ├── test_resilience.py # Sub-5ms SLA, structural validation & lookahead tests
│ └── test_new_features.py # Loop breaker, IDE auto-config, dollar savings & /verify tests
├── assets/ # Icons & diagrams
├── config.yaml # Gateway configuration
├── pyproject.toml # Modern pip/uv packaging metadata
└── requirements.txt📡 API Reference
Endpoint | Method | Description |
|
| OpenAI-compatible chat completion endpoint supporting resilient SSE streaming, lookahead failover, loop breaking, and diagnostic headers ( |
|
| Returns available virtual model aliases ( |
|
| Desktop HUD Telemetry: Real-time status, active route, model, VRAM used/total, temp (°C), power (W), latency, and dollar savings. |
|
| Real-time GPU telemetry: VRAM allocation, temperature, power draw (W), and detected hardware tier. |
|
| Instant VRAM Purge ("Pause GPU"): Evicts loaded models to 0 MB VRAM in <1s for AAA gaming or rendering. |
|
| Warm GPU: Preloads and pins model into VRAM with indefinite residency ( |
|
| 1-Click IDE Setup: Injects CascadeGateway configuration into |
|
| Sets active workflow mode ( |
|
| Telemetry: exact hardware token accounting ( |
|
| Model Context Protocol streamable HTTP endpoint for Google Antigravity and Claude Desktop. |
|
| Interactive browser control center, real-time hardware gauges, and prompt workbench. |
⚙️ Configuration (config.yaml)
server:
host: "127.0.0.1"
port: 8000
local:
provider: "ollama"
base_url: "http://127.0.0.1:11434"
# Set empty to let the hardware profiler auto-select the best model for your GPU
primary_model: ""
max_context_tokens: 32768
timeout_seconds: 120
cloud:
provider: "gemini"
default_model: "gemini-2.5-flash"
pro_model: "gemini-2.5-pro"
max_context_tokens: 1048576
biasing:
mode: "adaptive" # "adaptive", "manual", or "local_only"
bias_factor: 0.35 # Baseline local bias (0.0 to 1.0)
cloud_token_budget: 500000
allow_context_overflow_to_cloud: true📜 License
This project is licensed under the MIT License.
Available Tools
3 toolscascade_llmB
Run a query through the intelligent cascading gateway (RTX 5090 -> Gemini) with dynamic biasing.
Inputs: prompt (required), system_prompt.
get_cascade_metricsA
Get live metrics on tokens saved locally on the RTX 5090, local offload percentage, and biasing state.
Inputs: no named parameters.
query_local_5090A
Run a query or code task directly on the local NVIDIA RTX 5090 (qwen2.5-coder:32b) at $0 token cost.
Inputs: prompt (required), system_prompt.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 3 tools
query_local_5090 and cascade_llm both execute queries, but the descriptions clearly distinguish local-only vs cascading backends. get_cascade_metrics is clearly a metrics tool. Minor overlap remains in the query execution purpose, but an agent can tell them apart.
All names are snake_case, but the patterns are mixed: query_local_5090 starts with a verb, cascade_llm is a noun phrase, and get_cascade_metrics is verb_noun. Readable but not a predictable convention.
Three tools are well-scoped for a cascading LLM gateway: local query, cascaded query, and metrics. Each tool has a clear role and the count avoids redundancy.
The core operations—run a local query, run a cascaded query, and monitor metrics—are covered. However, there is no tool to configure or set dynamic biasing, which is a minor gap for controlling gateway behavior.
Maintenance
Related MCP Connectors
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
Chat, image, video, and audio models from many vendors behind one remote MCP server and one key.
Talk to your public-facing AI from any MCP client — Claude, ChatGPT, Cursor, Cline, Windsurf.
HiveCompute MCP Server — decentralized inference router for AI agents
Related MCP Servers
- FlicenseAqualityDmaintenanceLocal multimodal MCP server for vision, document, audio, video, and GUI automation, all processed locally on Apple Silicon for privacy and security.9-

Local AI MCPofficial
AlicenseAqualityBmaintenanceUnified MCP server for managing local model runtimes (Ollama, LM Studio, etc.), enabling provider-agnostic discovery, lifecycle management, hardware-fit checks, and delegated inference.22104 npmCreative Commons Attribution Non Commercial No Derivatives 4.0 International- FlicenseAqualityDmaintenanceUnified MCP server for querying multiple LLM providers (Gemini, OpenAI, Anthropic, etc.) with advanced routing, cost optimization, and fallback resilience.2-
- AlicenseNot gradedqualityBmaintenanceMCP server that delegates mechanical tasks like summarization, classification, extraction, and drafting to a local Llama.cpp LLM, serving as a cost-optimization layer while Claude handles reasoning and quality control.MIT