Skip to main content
Glama

CascadeGateway ⚡

Hardware-Adaptive Local LLM & Cloud Cascading Architecture

License: MIT Python: 3.10+ Ollama Compatible OpenAI Compatible Protocol

CascadeGateway is an intelligent, hardware-adaptive cascading proxy that routes AI queries between local GPUs (via Ollama at $0 token cost) and frontier cloud models (Google Gemini, OpenAI, etc.).

It automatically detects your GPU hardware and physical VRAM on startup—whether running a flagship RTX 5090 (32GB), RTX 4090 (24GB), mainstream RTX 3080 (10GB), or Apple Silicon—and dynamically selects and sizes the optimal models without requiring manual reconfiguration.

flowchart TD
    Client["AI Client: Cursor / Continue / Antigravity"] --> Gateway["CascadeGateway :8000/v1"]

    Gateway --> Phase1{"Phase 1: Structural Guard"}
    Phase1 -->|"Context > 32k or Tools"| CloudFallback["Cloud Fallback: Gemini"]
    Phase1 -->|"Passed"| Phase2{"Phase 2: Lexical Automaton <1ms"}

    Phase2 -->|"Architecture / Reasoning"| LocalArchitect["Local Architect: DeepSeek-R1 / Gemma 4 (RTX 5090)"]
    Phase2 -->|"Coding / Tests / Routine"| LocalBuilder["Local Builder: Qwen 2.5 Coder 32B (RTX 5090)"]

    LocalArchitect --> Phase3{"Phase 3: 3-Token Lookahead"}
    LocalBuilder --> Phase3
    Phase3 -->|"Stalled / Timeout"| CloudFallback
    Phase3 -->|"Healthy Stream"| LoopBreaker["Sliding-Window Loop Breaker"]
    LoopBreaker --> ClientStream["SSE Stream to Client"]

    CloudFallback --> ClientStream

    VerifyRoute["Asymmetric Verification: /verify"] --> LocalDraft["Local Generator: RTX 5090"]
    LocalDraft --> CloudCritic["Cloud Critic: Gemini Flash"]
    CloudCritic --> ClientStream

    Gateway -. Telemetry .-> HUD["🖥️ Always-On-Top Desktop HUD"]

🚀 Key Features

  • 🖥️ Always-On-Top Desktop HUD: Sleek, frameless, draggable obsidian overlay (#0b0f19) that sits above your IDEs next to the clock or anywhere on screen. Displays live routing decisions (🟢 Local Builder, 🟣 Local Architect, 🔵 Asymmetric Verify, 🟠 Gemini Cloud), real-time VRAM gauge (GB, %, °C, W), decision latency (sub-5ms), prompt snippet, dollar savings counter, pin toggle, compact mini-mode, and 1-click mode switcher buttons.

  • 🧠 Local GPU Architectural Reasoning: True multi-model orchestration on local silicon. System design, race condition, thread safety, and algorithmic queries are dispatched directly to the local reasoning model (deepseek-r1:14b or gemma4:26b) on the RTX 5090 rather than escaping to cloud tokens. Routine code generation is routed to qwen2.5-coder:32b.

  • Hardware-Adaptive VRAM Sizing: Automatically profiles your GPU (nvidia-smi / Metal / CPU) and assigns the optimal model tier.

  • Autonomous $0 Offload: Resolves 85–95% of routine coding, unit tests, refactoring, and QA directly on local VRAM.

  • Dynamic Token Biasing Engine:

    • Adaptive Mode: Automatically increases local GPU bias as cloud token budgets get consumed.

    • Manual Slider: Fine-tune local preference from 0% (Quality First) to 100% (Maximum Local Offload).

    • Strict Local: 100% execution on local VRAM with automatic cloud overflow only if context exceeds physical limit.

  • 🧠 Multi-Task Workflow Modes & Interactive Review Gate:

    • 🧠 Architect & Builder: Reasoning model drafts architecture, data contracts, and edge cases → Pauses at a Human Review Gate → You approve or refine → Builder synthesizes production code.

    • 🚀 Solo Sprint: Instant local coding via qwen2.5-coder:32b for quick functions, tests, and scripts.

    • 🌐 Deep Context: Gemini 2.5 Flash ingests massive repository files (1M context) → modular execution on local 5090.

    • 🔬 Math & Algo Proof: Deep Chain-of-Thought formal verification for cryptography and concurrency algorithms.

    • 🛡️ Asymmetric Verification (/verify): High-speed local draft synthesis on RTX 5090 ($0) combined with an anonymous strict cloud auditor (Gemini) providing production-grade critique and code enhancement without leaking full history.

  • 🛡️ Sub-5ms Intelligent Routing & 3-Token Lookahead Failover: Three-phase classification pipeline: Phase 1 Structural & VRAM Budget Validation (<1ms, prevents swapping to system RAM), Phase 2 Single-Pass Lexical Scan (<1ms), and Phase 3 Resilient Streaming with an in-memory 3-token lookahead buffer that transparently replays stalled local requests to Gemini Cloud without dropping client connections or throwing IDE error popups.

  • 🔁 Anti-Hallucination Sliding-Window Loop Breaker: In-flight ring buffer tracking token emission sequences (lengths 2, 3, 4 repeated ≥ 4×), terminating runaway generative loops immediately.

  • ⚡ 1-Click IDE Auto-Configuration: Automated zero-friction setup endpoint (/api/ide/auto-config) and UI button for VSCode Continue (~/.continue/config.json) and Cursor.

  • 💰 Zero-Surprise Dollar Savings Telemetry: Hardware-native token accounting (eval_count) calculating real-time dollar savings based on commercial frontier rates ($3.00/1M tokens).

  • ⚡ Persistent GPU VRAM Residency & Zero-Cold-Start: Models remain permanently resident in GPU VRAM (keep_alive: -1) across idle periods and tasks, ensuring instant zero-delay responses. Includes manual 1-Click "Pause GPU" (instant VRAM purge for AAA gaming) and "Warm GPU" (preloads weights into VRAM).

  • Native Model Context Protocol (MCP): Exposes a high-performance Streamable HTTP and Stdio MCP endpoint for Google Antigravity and Claude Desktop.

  • 🎛️ Dynamic Model Selection: Select any installed local model for both the Architect role (deepseek-r1:14b, gemma4:26b, etc.) and the Builder role (qwen2.5-coder:32b, etc.) directly from the Web UI toolbar or Windows Tray submenus. Automatically detects newly pulled models from Ollama without restarting the gateway.

  • Zero-Window Background Tray App: Runs silently in the system tray, boots with Windows/Linux, and includes single-instance mutex protection.

  • 🎮 1-Click Pause & Free GPU (Instant VRAM Purge): Evicts loaded models from VRAM in <1s via a dedicated button on the Web UI, Windows Tray, or POST /api/models/unload. Frees 20–30+ GB of VRAM immediately for AAA gaming, Blender, or video editing without terminating the server. Models reload automatically on demand when coding.

  • Live Hardware Telemetry: In-browser control center showing real-time VRAM allocation, GPU power draw (W), temperature (°C), lifetime token savings, and an interactive prompt runner.

  • Standard OpenAI-Compatible API: Seamless drop-in replacement (/v1/chat/completions) for Cursor, VSCode (Continue.dev), Aider, Claude Dev, and custom scripts.


Related MCP server: Local AI MCP

🧠 The Interactive Review Gate

sequenceDiagram
    autonumber
    actor Dev as You (Developer)
    participant Arch as 🧠 Architect - DeepSeek-R1 / Gemma 4
    participant Gate as 🛑 Review Gate - You in the Loop
    participant Build as ⚡ Builder - Qwen 2.5 Coder 32B

    Dev->>Arch: "Add rate-limiting and burst protection to API routes"
    Note over Arch: Deep CoT: Identifies race conditions,<br/>evaluates Redis vs in-memory,<br/>drafts interface contracts.
    Arch->>Gate: Presents Architectural Blueprint + Edge Cases
    Note over Gate: PAUSE: No code written yet.<br/>You review the proposed interfaces & strategy.

    alt If you want adjustments
        Dev->>Gate: "Use Redis, and add IPv6 CIDR subnet matching"
        Gate->>Arch: Quick amendment - 100 tokens
        Arch->>Gate: Updated spec
    end

    Dev->>Gate: "Approve & Build"
    Gate->>Build: Sends final structured specification
    Note over Build: Zero ambiguity.<br/>High-speed code synthesis - 70 t/s.
    Build-->>Dev: Delivers complete implementation + unit tests - $0 Cost

🏛️ Hybrid Architect-Builder Execution (Google Antigravity & CascadeGateway)

When running agentic coding assistants like Google Antigravity on a Pro plan, repeatedly shipping 20k to 50k tokens of repo file dumps, compiler logs, and multi-turn message history quickly exhausts 5-hour quota refill blocks.

CascadeGateway implements the native Hybrid Architect-Builder Pattern:

flowchart TD
    subgraph Cloud ["☁️ Cloud Tier (Pro Plan Quota)"]
        Architect["Cloud Architect: Gemini Pro / Claude<br/>• High-level system decomposition<br/>• Milestone roadmap planning<br/>• Concise diff reviews"]
    end

    subgraph Gateway ["⚡ CascadeGateway :8000/v1"]
        Router["Context Pruning & Intent Router"]
        Clamp["Terminal Clamping (&gt;60 lines &rarr; 15 head + 30 tail)"]
        Dedup["Repeat File Buffer Deduplication"]
    end

    subgraph Local ["🖥️ Local Workstation Tier (NVIDIA RTX 5090 - 32GB VRAM)"]
        Builder["Local Builder (vLLM / Ollama): Qwen 2.5 Coder 32B<br/>• Code generation, unit tests, refactoring<br/>• Iterative bug fixes & lint diagnostics<br/>• $0 Cloud Token Cost"]
        LocalMCP["Local MCP Tools<br/>• AST Symbol Lookup (@llmtooling/code-search-mcp)<br/>• Targeted File Slicing"]
    end

    Architect -->|"Milestone Specifications"| Router
    Router --> Clamp
    Clamp --> Dedup
    Dedup -->|"Iterative Coding Tasks"| Builder
    Builder <-->|"Precise Symbol Queries"| LocalMCP
    Dedup -. Escalation on repeated test failures .-> Architect

📉 Token Efficiency & Telemetry Audit Benchmark

Benchmarked using python -m cascadegateway.audit --compare baseline.jsonl optimized.jsonl:

Metric

Baseline (Cloud Pass-Through)

Optimized (Architect-Builder on 5090)

Impact

Total Task Turns

15 turns

16 turns

Consistent scope

Average Prompt Tokens / Turn

38,200 tokens

3,897 tokens

-89.8% context bloat

Tokens Billed to Cloud Quota

579,300 tokens

3,550 tokens

-99.4% Cloud Token Reduction

Tokens Absorbed by Local GPU

0 tokens

68,500 tokens

95.1% Local RTX 5090 Execution Ratio

Prefix Cache Hit Rate

0.0%

76.3%

Instantaneous prefill

Average Latency

1,450 ms

800 ms

Sub-second response

🚀 Cross-Platform Local Engine Runners

  • Windows Native (Zero-Window Flash):

    powershell -File scripts\start_local_engine.ps1

    Auto-detects Ollama or vLLM, warms up qwen2.5-coder:32b in 32GB VRAM with keep_alive: -1.

  • Linux / WSL (vLLM with Automatic Prefix Caching):

    ./scripts/start_local_vllm.sh

    Launches vllm with --enable-prefix-caching, --enable-chunked-prefill, and --tool-call-parser hermes.

  • Antigravity SDK & Standalone Orchestration:

    python scripts/run_architect_builder.py --task "Draft and build target module"

📊 VRAM Hardware Sizing Matrix

Hardware Tier

Supported GPUs

VRAM

Recommended Coding Model

General / Reasoning

Expected Speed

Tier 1 (Flagship)

RTX 5090, 4090, 3090, A6000

24 GB – 32 GB

qwen2.5-coder:32b

gemma4:26b / deepseek-r1:32b

60 – 75+ tokens/sec

Tier 2 (Enthusiast)

RTX 4080, 4070 Ti, 4070, 3080 12GB

12 GB – 16 GB

qwen2.5-coder:14b

mistral-small:22b-q4

45 – 55+ tokens/sec

Tier 3 (Mainstream)

RTX 3080 10GB, 3070, 4060 Ti, 2080 Ti

8 GB – 10 GB

qwen2.5-coder:7b

llama3.1:8b

85 – 100+ tokens/sec

Tier 4 (Entry / CPU)

RTX 3050, 4060 laptop, Apple Silicon, CPU

&lt; 8 GB

qwen2.5-coder:1.5b / 3b

llama3.2:3b

40 – 70 tokens/sec


⚡ Quick Start

Prerequisites

  1. Ollama installed and running on your machine.

  2. Python 3.10+ installed.

  3. (Optional) A free API key from Google AI Studio for cloud fallback.

Windows (One-Click)

git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
scripts\setup.bat

Creates .venv, installs dependencies, auto-generates your desktop shortcut, and places an auto-start shortcut in your Windows Startup menu.

Linux / macOS

git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
chmod +x scripts/setup.sh
./scripts/setup.sh

🛠️ Connecting Your IDEs & Tools

Once running, the control center is available at http://127.0.0.1:8000/.

1. Google Antigravity

Add to ~/.gemini/config/mcp_config.json:

{
  "mcpServers": {
    "rtx5090-cascade": {
      "serverUrl": "http://127.0.0.1:8000/mcp"
    }
  }
}

Antigravity automatically detects the toolset (query_local_5090, cascade_llm, get_cascade_metrics) and dynamically dispatches code generation directly to your GPU.

2. Cursor

  • Navigate to Settings > Models > OpenAI API Key.

  • Set Base URL: http://127.0.0.1:8000/v1

  • Set API Key: cascading-local (any string)

  • Model: cascade-auto

3. Continue.dev (config.json)

{
  "models": [
    {
      "title": "Cascade Gateway (Local + Cloud)",
      "provider": "openai",
      "model": "cascade-auto",
      "apiBase": "http://127.0.0.1:8000/v1",
      "apiKey": "cascading-local"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Local Autocomplete",
    "provider": "openai",
    "model": "cascade-auto",
    "apiBase": "http://127.0.0.1:8000/v1",
    "apiKey": "cascading-local"
  }
}

4. Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "local-cascade": {
      "serverUrl": "http://127.0.0.1:8000/mcp"
    }
  }
}

5. Python OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="cascading-local"
)

response = client.chat.completions.create(
    model="cascade-auto",
    messages=[
        {"role": "user", "content": "Write an optimized LRU cache in Python."}
    ]
)
print(response.choices[0].message.content)

📁 Repository Architecture

CascadeGateway is structured as a modern, modular Python package:

CascadeGateway/
├── src/
│   └── cascadegateway/
│       ├── core/                  # Hardware profiler, model sizing & routing engine
│       │   ├── config.py          # YAML config & environment loader
│       │   ├── hardware.py        # GPU VRAM auto-profiling (5090/4090/etc.)
│       │   ├── classifier.py      # Phase 1 structural validation & Phase 2 regex automaton (<1ms)
│       │   ├── streaming.py       # Phase 3 lookahead failover, loop breaker & asymmetric audit
│       │   └── router.py          # Biasing engine, metrics & model resolution
│       ├── api/                   # Modular FastAPI endpoints
│       │   ├── server.py          # App initialization, SSE streaming & /v1/chat/completions
│       │   ├── pipeline.py        # Architect, Review Gate & Builder endpoints
│       │   └── models.py          # Pause & Free GPU (VRAM purge) & 1-click IDE auto-config
│       ├── web/                   # Clean decoupled web assets & responsive UI
│       │   ├── templates/
│       │   │   └── dashboard.html # Responsive HTML5 dashboard & real-time telemetry
│       │   └── static/
│       │       ├── css/dashboard.css
│       │       └── js/dashboard.js
│       ├── hud/                   # Always-On-Top Desktop HUD & Telemetry Overlay
│       │   ├── __init__.py
│       │   └── overlay.py         # Tkinter floating dark obsidian window
│       ├── tray/                  # Windows system tray controller
│       │   └── app.py             # Pystray background service & menus
│       ├── mcp/                   # Model Context Protocol endpoints
│       │   ├── __init__.py
│       │   └── server.py          # Antigravity & Claude Desktop integration
├── scripts/                       # Platform launchers & utilities
│   ├── start_cascade.bat          # Windows batch launcher
│   ├── start_tray.vbs             # Silent windowless tray runner
│   ├── start_hud.bat              # Desktop HUD launcher
│   ├── start_hud.vbs              # Silent windowless HUD launcher
│   ├── setup.bat / setup.sh       # One-click installers
│   └── create_shortcuts.ps1       # Desktop & startup shortcut generator
├── tests/                         # Comprehensive automated test suites
│   ├── test_gateway.py            # End-to-end API integration tests
│   ├── test_resilience.py         # Sub-5ms SLA, structural validation & lookahead tests
│   └── test_new_features.py       # Loop breaker, IDE auto-config, dollar savings & /verify tests
├── assets/                        # Icons & diagrams
├── config.yaml                    # Gateway configuration
├── pyproject.toml                 # Modern pip/uv packaging metadata
└── requirements.txt

📡 API Reference

Endpoint

Method

Description

/v1/chat/completions

POST

OpenAI-compatible chat completion endpoint supporting resilient SSE streaming, lookahead failover, loop breaking, and diagnostic headers (X-Cascade-*).

/v1/models

GET

Returns available virtual model aliases (cascade-auto, local-5090) and physical Ollama models.

/api/hud/state

GET

Desktop HUD Telemetry: Real-time status, active route, model, VRAM used/total, temp (°C), power (W), latency, and dollar savings.

/api/hardware

GET

Real-time GPU telemetry: VRAM allocation, temperature, power draw (W), and detected hardware tier.

/api/models/unload

POST

Instant VRAM Purge ("Pause GPU"): Evicts loaded models to 0 MB VRAM in &lt;1s for AAA gaming or rendering.

/api/models/preload

POST

Warm GPU: Preloads and pins model into VRAM with indefinite residency (keep_alive: -1).

/api/ide/auto-config

POST

1-Click IDE Setup: Injects CascadeGateway configuration into ~/.continue/config.json and detects Cursor environments.

/api/workflow/mode

POST

Sets active workflow mode (architect, builder, solo, verify, deep_context, math).

/metrics

GET

Telemetry: exact hardware token accounting (eval_count), local offload percentage, and real-time dollar savings.

/mcp

POST

Model Context Protocol streamable HTTP endpoint for Google Antigravity and Claude Desktop.

/

GET

Interactive browser control center, real-time hardware gauges, and prompt workbench.


⚙️ Configuration (config.yaml)

server:
  host: "127.0.0.1"
  port: 8000

local:
  provider: "ollama"
  base_url: "http://127.0.0.1:11434"
  # Set empty to let the hardware profiler auto-select the best model for your GPU
  primary_model: ""
  max_context_tokens: 32768
  timeout_seconds: 120

cloud:
  provider: "gemini"
  default_model: "gemini-2.5-flash"
  pro_model: "gemini-2.5-pro"
  max_context_tokens: 1048576

biasing:
  mode: "adaptive"        # "adaptive", "manual", or "local_only"
  bias_factor: 0.35       # Baseline local bias (0.0 to 1.0)
  cloud_token_budget: 500000
  allow_context_overflow_to_cloud: true

📜 License

This project is licensed under the MIT License.

Available Tools

3 tools
cascade_llmB

Run a query through the intelligent cascading gateway (RTX 5090 -> Gemini) with dynamic biasing.

Inputs: prompt (required), system_prompt.

get_cascade_metricsA

Get live metrics on tokens saved locally on the RTX 5090, local offload percentage, and biasing state.

Inputs: no named parameters.

query_local_5090A

Run a query or code task directly on the local NVIDIA RTX 5090 (qwen2.5-coder:32b) at $0 token cost.

Inputs: prompt (required), system_prompt.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 3 tools

    Disambiguation4/5

    query_local_5090 and cascade_llm both execute queries, but the descriptions clearly distinguish local-only vs cascading backends. get_cascade_metrics is clearly a metrics tool. Minor overlap remains in the query execution purpose, but an agent can tell them apart.

    Naming Consistency3/5

    All names are snake_case, but the patterns are mixed: query_local_5090 starts with a verb, cascade_llm is a noun phrase, and get_cascade_metrics is verb_noun. Readable but not a predictable convention.

    Tool Count5/5

    Three tools are well-scoped for a cascading LLM gateway: local query, cascaded query, and metrics. Each tool has a clear role and the count avoids redundancy.

    Completeness4/5

    The core operations—run a local query, run a cascaded query, and monitor metrics—are covered. However, there is no tool to configure or set dynamic biasing, which is a minor gap for controlling gateway behavior.

    Maintenance

    ActivitySlowing
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      Unified MCP server for managing local model runtimes (Ollama, LM Studio, etc.), enabling provider-agnostic discovery, lifecycle management, hardware-fit checks, and delegated inference.
      22
      104 npm
      Creative Commons Attribution Non Commercial No Derivatives 4.0 International
    • F
      license
      A
      quality
      D
      maintenance
      Unified MCP server for querying multiple LLM providers (Gemini, OpenAI, Anthropic, etc.) with advanced routing, cost optimization, and fallback resilience.
      2
      -
    • A
      license
      Not graded
      quality
      B
      maintenance
      MCP server that delegates mechanical tasks like summarization, classification, extraction, and drafting to a local Llama.cpp LLM, serving as a cost-optimization layer while Claude handles reasoning and quality control.
      MIT