MCP-AI-Gateway

Unified local MCP AI Gateway that routes across Groq, OpenRouter, Mistral, and local Ollama providers, with OpenAI-compatible APIs, MCP tools, fallback/racing router, monitoring, and web dashboard.

🚀 Features

• Async Python gateway with provider abstraction layer

• Built-in providers:

  • Groq (openai/gpt-oss-120b default, plus moonshotai/kimi-k2-instruct-0905)

  • OpenRouter (openai/gpt-4o-mini default)

  • Mistral (codestral-latest default, plus devstral-medium-latest)

  • Ollama

• Intelligent router:

  • priority routing

  • fallback routing

  • parallel racing mode

  • rate-limit aware provider scoring

• MCP server capabilities:

  • tool registration/discovery

  • prompt template discovery/rendering

  • schema validation

  • SSE and HTTP transport

• OpenAI-compatible API:

  • POST /v1/chat/completions

  • POST /v1/completions

  • GET /v1/models

• Web UI (React + Vite + Tailwind):

  • Dashboard

  • Providers

  • Models

  • Chat

  • Logs

  • Stats

  • Runtime gateway API key generation

• Observability:

  • Prometheus metrics (/metrics)

  • structured logs

  • SQLite request statistics

• Extensible plugin system for additional providers

• CLI: mcp-ai start|stop|status|providers|models

• Docker + docker-compose support

• GitHub Actions CI for backend lint/tests and frontend build

Related MCP server: Multi-LLM Gateway MCP

🗂️ Repository Structure

mcp-ai-gateway/
  api/
  providers/
  router/
  mcp_server/
  plugins/
  database/
  web_ui/
  config/
  cli/
  tests/
  Dockerfile
  docker-compose.yml
  README.md

⚡ Quick Start (Local)

1) 📦 Prerequisites

• Python 3.11+

• Node.js 20+

• Optional: local Ollama daemon at http://localhost:11434

2) 🐍 Install backend

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev]

3) 🔐 Configure environment

cp .env.example .env
# edit .env values

4) ⚙️ Edit config

config/config.yaml controls providers, routing, security, and default models.

5) ▶️ Start backend

uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

6) 🖥️ Start web UI

cd web_ui
npm install
VITE_API_BASE=http://localhost:8000 VITE_API_KEY=change-me npm run dev

Open http://localhost:3000

🐳 Docker Deployment

docker compose up --build

Services:

• Gateway API: http://localhost:8000

• Web UI: http://localhost:3000

• Redis: localhost:6379 (optional cache/rate-limit backend)

🧩 API Overview

OpenAI-Compatible 🤝

• POST /v1/chat/completions

• POST /v1/completions

• GET /v1/models

Use header X-API-Key: <key> (or Authorization: Bearer <key>).

MCP Endpoints 🔌

• JSON-RPC MCP transport: POST /mcp

• SSE streaming output: POST /mcp/stream

Supported MCP methods:

• initialize

• notifications/initialized

• ping

• tools/list

• tools/call

• prompts/list

• prompts/get

Dashboard APIs 📊

• GET /api/dashboard

• GET /api/providers

• POST /api/providers/{name}/toggle

• POST /api/providers/{name}/priority

• POST /api/providers/{name}/config

• GET /api/models

• GET /api/settings

• POST /api/settings/default-model

• POST /api/settings/api-key/generate

• GET /api/logs

• GET /api/stats

Monitoring 📈

• GET /health

• GET /metrics

🛠️ CLI

mcp-ai start
mcp-ai status
mcp-ai providers
mcp-ai models
mcp-ai stop

🧭 Warp MCP Integration

Add this to Warp MCP config:

{
  "mcpServers": {
    "mcp-ai-gateway": {
      "url": "http://localhost:8000/mcp",
      "headers": {
        "Authorization": "Bearer REPLACE_WITH_MCP_GATEWAY_API_KEY"
      }
    }
  }
}

🧱 Configuration Example

providers:
  groq:
    enabled: true
    api_key: ${GROQ_API_KEY}
    priority: 1
    default_model: openai/gpt-oss-120b
  mistral:
    enabled: false
    api_key: ${MISTRAL_API_KEY}
    priority: 12
    default_model: codestral-latest
  openrouter:
    enabled: true
    priority: 2
    default_model: openai/gpt-4o-mini
  ollama:
    enabled: true
    priority: 3
routing:
  fallback_enabled: true
  racing_mode: false

🧪 Plugin Providers

New providers can be added under plugins/ by implementing ProviderPlugin and enabling provider config in config/config.yaml.

Included plugin examples:

• openai_provider.py

• together_provider.py

• openrouter_provider.py

✅ Testing

pytest

CI runs on pushes and pull requests to main via .github/workflows/ci.yml:

• Backend job: ruff check . and pytest -q

• Frontend job: npm install and npm run build in web_ui/

🔄 Release Workflow

• Open PR from dev/* branch to main

• Ensure CI is green

• Merge using GitHub CLI or GitHub UI

📚 Developer Guide

Detailed guide: docs/developer-guide.md