MCP LLM Bridge

Encrypted LLM gateway and MCP server for routing API keys, CLI subscriptions, and model selection through one OpenAI-compatible endpoint.

Read this in: English · Español

Demo Links

• Live gateway: https://gateway.javierzader.com

• GHAGGA integration target: https://github.com/JNZader/ghagga

• OpenCode: https://github.com/anomalyco/opencode

Visuals coming soon.

Related MCP server: Proxima

Quick Portfolio Snapshot

• One service for LLM routing, encrypted credential storage, MCP tooling, and OpenAI-compatible HTTP access.

• 11 provider adapters today: 5 direct API providers plus 6 CLI-backed providers.

• Supports API keys and auth-file workflows, including auth.json and .credentials.json.

• Includes task-aware bridge routing, model routing, project-scoped credentials with global fallback, semantic code search, context compression, and CRDT shared state.

• Ships as a local dev tool, self-hosted HTTP gateway, MCP stdio server, and Docker deployment.

Why It Matters

• Centralizes secrets instead of scattering provider tokens across every project and tool.

• Lets you reuse CLI subscriptions such as OpenCode, Claude, Gemini, Codex, Qwen, and Copilot behind one interface.

• Gives OpenAI-compatible tools a single stable endpoint while preserving provider/model resolution metadata.

• Supports multi-project setups where project-specific credentials override _global defaults cleanly.

• Exposes MCP tools beyond plain generation: vault operations, code search, shared state, usage inspection, and provider-group management.

Quick Start

pnpm install
pnpm run serve

Open http://localhost:3456.

Store a credential and generate text:

curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-..."}'

curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

If you set LLM_GATEWAY_AUTH_TOKEN, add Authorization: Bearer <token> to every protected route.

Jump to Technical Docs

• Full API reference: Technical README

• Auth and credential model: Authentication, Credential Management

• MCP integration: MCP Server

• Docker and self-hosting: Docker Deployment

Technical README

Table of Contents

1. Quick Start

2. Dashboard

3. API Reference

4. Providers

5. Authentication

6. Credential Management

7. Cross-Model Bridge

8. Context Compression

9. Semantic Code Search

10. CRDT Multi-Agent State

11. Integrations

12. Docker Deployment

13. MCP Server

14. Configuration

15. Security Profiles

16. Approval Flows

17. Three-Part Prompt

18. RTK Output Compression

19. Local LLM Offloading

20. Model Routing

21. HF Auto-Discovery

22. Architecture

23. Security

24. Development

25. License

Quick Start

# Install dependencies
pnpm install

# Start the HTTP server + dashboard
pnpm run serve

# MCP stdio mode only
pnpm run start

Basic HTTP flow:

# Store a global Anthropic key
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-..."}'

# Generate text with automatic provider selection
curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

If auth is enabled:

curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

Dashboard

The repo currently has two dashboard surfaces:

• Local inline shell at http://localhost:3456/ — legacy local ops surface served directly by the bridge. This remains the source of truth for local credential/auth-file management and quick test generation.

• React admin app under dashboard/ (built into docs/) — admin/observability surface for overview, providers, usage, groups, circuit breakers, settings, and related views.

They intentionally coexist for now and do not have full feature parity.

• Hosted demo: https://gateway.javierzader.com

• Local inline shell: http://localhost:3456

First-Time Setup

1. Start the gateway with pnpm run serve.

2. Open the dashboard.

3. Enter the base URL for your gateway.

4. Enter the bearer token if LLM_GATEWAY_AUTH_TOKEN is configured.

5. Test the connection and save.

Local Inline Shell Capabilities

• Add, list, filter, and delete encrypted API keys.

• Upload auth files for CLI-backed providers.

• Inspect provider availability and available models.

• Send test prompts and inspect returned provider/model metadata.

• Work with project-scoped credentials without exposing raw secrets.

React Admin App Capabilities

• Overview / provider status / usage / groups / circuit breakers / settings

• Admin-facing operational visibility over bridge subsystems

• Hosted separately from the inline shell via the dashboard/ app

Recommended auth-file mappings in the UI and API:

• opencode -> auth.json

• claude -> .credentials.json

• codex -> auth.json

• gemini -> settings.json and oauth_creds.json

• qwen -> settings.json and oauth_creds.json

• copilot -> use token credentials instead of auth files

API Reference

All protected endpoints require:

Authorization: Bearer <your-token>

When LLM_GATEWAY_AUTH_TOKEN is not set, auth is disabled for local development. GET /health always stays public.

Core HTTP Endpoints

POST /v1/generate

Native generation endpoint with provider/model selection and project-scoped credential resolution.

# Auto-select provider
curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"prompt":"Explain quicksort in one paragraph"}'

# Explicit provider + model + project
curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'X-Project: my-app' \
  -d '{
    "prompt":"Write a haiku about Rust",
    "provider":"groq",
    "model":"llama-3.3-70b-versatile",
    "maxTokens":256,
    "system":"You are a poet.",
    "project":"my-app"
  }'

Request body:

Response:

{
  "text": "Quicksort is a divide-and-conquer...",
  "provider": "anthropic",
  "model": "claude-sonnet-4-20250514",
  "tokensUsed": 150,
  "requestedProvider": null,
  "requestedModel": null,
  "resolvedProvider": "anthropic",
  "resolvedModel": "claude-sonnet-4-20250514",
  "fallbackUsed": false
}

POST /v1/chat/completions

OpenAI-compatible chat endpoint. This is the drop-in path for tools that already speak OpenAI format.

• Non-streaming and streaming requests are supported.

• System messages are collapsed into the system prompt.

• Conversation context is reconstructed from earlier messages.

• Response stays OpenAI-compatible and adds x_gateway metadata.

curl -X POST http://localhost:3456/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "model":"claude-sonnet-4-20250514",
    "messages":[
      {"role":"system","content":"You are a helpful assistant."},
      {"role":"user","content":"What is the capital of France?"}
    ],
    "max_tokens":1024
  }'

Response:

{
  "id": "chatcmpl-<uuid>",
  "object": "chat.completion",
  "created": 1710000000,
  "model": "claude-sonnet-4-20250514",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "The capital of France is Paris." },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 150 },
  "x_gateway": {
    "requestedProvider": null,
    "requestedModel": "claude-sonnet-4-20250514",
    "resolvedProvider": "anthropic",
    "resolvedModel": "claude-sonnet-4-20250514",
    "fallbackUsed": false,
    "tokensUsed": 150
  }
}

GET /v1/models

Lists available models in OpenAI-compatible format.

curl http://localhost:3456/v1/models \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "object": "list",
  "data": [
    {
      "id": "claude-sonnet-4-20250514",
      "object": "model",
      "created": 0,
      "owned_by": "llm-gateway",
      "name": "Claude Sonnet 4",
      "provider": "anthropic",
      "max_tokens": 8192
    }
  ]
}

GET /v1/providers

Lists registered providers and their availability.

curl http://localhost:3456/v1/providers \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "providers": [
    { "id": "anthropic", "name": "Anthropic", "type": "api", "available": true },
    { "id": "openai", "name": "OpenAI", "type": "api", "available": false },
    { "id": "opencode-cli", "name": "OpenCode CLI", "type": "cli", "available": true }
  ]
}

Credentials API

Store API keys encrypted at rest. Upsert key is (provider, keyName, project).

# Global credential
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"anthropic",
    "keyName":"default",
    "apiKey":"sk-ant-api03-..."
  }'

# Project-scoped credential
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"openai",
    "keyName":"default",
    "apiKey":"sk-proj-...",
    "project":"my-app"
  }'

{ "id": 1, "provider": "anthropic", "keyName": "default", "project": "_global" }

List credentials:

curl http://localhost:3456/v1/credentials \
  -H 'Authorization: Bearer YOUR_TOKEN'

curl 'http://localhost:3456/v1/credentials?project=my-app' \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "credentials": [
    {
      "id": 1,
      "provider": "anthropic",
      "keyName": "default",
      "project": "_global",
      "maskedValue": "sk-ant-...***",
      "createdAt": "2025-01-15 10:30:00",
      "updatedAt": "2025-01-15 10:30:00"
    }
  ]
}

Delete a credential:

curl -X DELETE http://localhost:3456/v1/credentials/1 \
  -H 'Authorization: Bearer YOUR_TOKEN'

Auth Files API

Store auth files for CLI-backed providers encrypted at rest. Upsert key is (provider, fileName, project).

This is the path that preserves the older auth.json and .credentials.json workflows.

# OpenCode auth.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"opencode",
    "fileName":"auth.json",
    "content":"{\"token\":\"oc-...\"}",
    "project":"_global"
  }'

# Claude CLI .credentials.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"claude",
    "fileName":".credentials.json",
    "content":"{\"claudeAiOauth\":{...}}",
    "project":"my-app"
  }'

{ "id": 1, "provider": "opencode", "fileName": "auth.json", "project": "_global" }

List auth files:

curl http://localhost:3456/v1/files \
  -H 'Authorization: Bearer YOUR_TOKEN'

curl 'http://localhost:3456/v1/files?project=my-app' \
  -H 'Authorization: Bearer YOUR_TOKEN'

{
  "files": [
    {
      "id": 1,
      "provider": "opencode",
      "fileName": "auth.json",
      "project": "_global",
      "createdAt": "2025-01-15"
    }
  ]
}

Delete an auth file:

curl -X DELETE http://localhost:3456/v1/files/1 \
  -H 'Authorization: Bearer YOUR_TOKEN'

Usage, Cost, Metrics, and Health

Usage records:

curl 'http://localhost:3456/v1/usage?project=my-app&limit=50' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Usage summary:

curl 'http://localhost:3456/v1/usage/summary?groupBy=provider&project=my-app' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Cost estimate:

curl 'http://localhost:3456/v1/cost/estimate?model=claude-sonnet-4-20250514&inputTokens=1000&outputTokens=500' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Prometheus metrics:

curl http://localhost:3456/metrics \
  -H 'Authorization: Bearer YOUR_TOKEN'

Health check:

curl http://localhost:3456/health

GET /health returns the runtime VERSION constant (src/core/constants.ts) plus uptime, auth mode, and provider counts:

{
  "status": "ok",
  "version": "0.3.1",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "uptime": 3600,
  "auth": { "enabled": true, "mode": "bearer" },
  "providers": { "total": 11, "available": 3 }
}

Note: the VERSION constant and the version field in package.json are not kept in lockstep — /health reports the former.

Provider Groups

Provider groups let you define logical pools for balancing and failover.

curl -X POST http://localhost:3456/v1/groups \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "name":"fast-models",
    "modelPattern":"gpt-*,claude-*",
    "members":[
      {"provider":"groq","weight":2,"priority":1},
      {"provider":"anthropic","weight":1,"priority":2}
    ],
    "strategy":"weighted",
    "stickyTTL":300
  }'

Providers

API Providers

CLI Providers

OpenCode Model Coverage

OpenCode is the biggest catalog here and is one reason this bridge is useful.

• Free tier models under opencode/*

• OpenCode Go subscription models under opencode-go/*

• Anthropic models under anthropic/*

• GitHub Copilot-routed models under github-copilot/*

• OpenAI-routed models under openai/*

Representative examples from the current adapter list:

• opencode/gpt-5-nano

• anthropic/claude-sonnet-4.5

• github-copilot/gpt-5.4

• openai/gpt-5.4

Provider Priority and Fallback

Default behavior without an explicit provider/model:

1. API providers are tried first.

2. CLI providers follow as fallback.

3. If a model is explicitly requested, the owning provider is preferred.

4. If bridge routing is enabled, the bridge can override the initial provider choice and then walk the configured fallback chain.

Authentication

Bearer Token

Set LLM_GATEWAY_AUTH_TOKEN to protect HTTP routes.

# Generate a secure token
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

export LLM_GATEWAY_AUTH_TOKEN="your-64-char-hex-token"

The token must be at least 32 characters.

Auth Rules

Important behavior:

• The bearer-auth middleware skips the entire /v1/admin/* prefix, not just /v1/admin/auth-config. Admin routes gate themselves with their own dashboard/GitHub-OAuth JWT checks (verifyDashboardJwt) rather than the static bearer token. Keep this in mind when exposing the gateway publicly.

• The dashboard (non-admin routes) is protected when bearer auth is enabled.

• MCP stdio does not use HTTP bearer auth because it runs as a local process.

• Token comparison is constant-time via timingSafeEqual.

Project Scoping

Project scope can be supplied in either place:

1. JSON body field: "project": "my-app"

2. Header: X-Project: my-app

Body field wins over header.

Credential Management

Global vs Project Credentials

Credential resolution follows the same pattern for API keys and auth files:

1. Try the project-specific entry.

2. Fall back to _global.

That lets you keep a shared default while still isolating overrides per app or customer.

API Keys

API keys are encrypted with AES-256-GCM and stored in SQLite.

# Global key
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-..."}'

# Project key
curl -X POST http://localhost:3456/v1/credentials \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{"provider":"anthropic","apiKey":"sk-ant-project-...","project":"my-app"}'

Auth Files

CLI adapters use file-based auth where necessary. These files are also encrypted and stored in the vault.

# OpenCode auth.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"opencode",
    "fileName":"auth.json",
    "content":"{\"token\":\"oc-...\"}"
  }'

# Claude CLI .credentials.json
curl -X POST http://localhost:3456/v1/files \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "provider":"claude",
    "fileName":".credentials.json",
    "content":"{\"claudeAiOauth\":{...}}"
  }'

Claude and OpenCode Credential Sync Pattern

The vault layer also contains a Claude OAuth integration that:

1. Reads ~/.claude/.credentials.json

2. Refreshes the token when needed

3. Syncs the token into OpenCode-style auth.json

That matters because this bridge can unify Claude CLI and OpenCode auth flows instead of treating them as separate credential silos.

Cross-Model Bridge

The bridge is an optional routing layer driven by ~/.llm-gateway/bridge.yaml.

Flow:

1. Classify the prompt into a task type.

2. Resolve a preferred provider from routes.

3. Try that provider first.

4. Walk fallback_order sequentially if it fails.

Supported Task Types

Example bridge.yaml

routes:
  large-context: gemini-cli
  code-review: claude-cli
  fast-completion: groq

default: claude-cli

fallback_order:
  - claude-cli
  - gemini-cli
  - opencode-cli
  - anthropic
  - groq

If the file is missing, the bridge is disabled and the normal router behavior is used.

Bridge Response Metadata

Context Compression

The CompressorService adds background context compression with caching.

Strategies

Usage

import { CompressorService } from './context-compression/index.js';

const compressor = new CompressorService({
  maxCacheSize: 200,
  workerIntervalMs: 5000,
  defaultStrategy: 'extractive',
  defaultRatio: 0.5,
});

compressor.submit(longContext);
const compressed = compressor.getCompressed(longContext);
const immediate = compressor.compressNow(longContext, 'structural');
compressor.destroy();

Operational Characteristics

• LRU cache for repeated content

• Background worker for non-blocking pre-computation

• Synchronous compression when you need the result immediately

• Useful for prompt pipelines where raw context would otherwise blow up token budgets

Semantic Code Search

The code-search subsystem exposes three search modes through MCP:

• keyword (default): exact/prefix/fuzzy matching with inverted index

• vector: semantic similarity via dense embeddings

• hybrid: RRF fusion of keyword + BM25 + vector for best results

It combines:

• regex-based chunking

• trigram fuzzy search

• BM25 keyword scoring (via MiniSearch)

• dense vector similarity (via transformer embeddings)

• Reciprocal Rank Fusion (RRF) for hybrid ranking

• optional multi-hop import following

Supported Languages

DEFAULT_EXTENSIONS (indexed by default) covers:

.ts, .tsx, .js, .jsx, .mjs, .cjs, .py, .go, .rs, .java, .rb, .lua

Dedicated chunk patterns exist for TypeScript/JavaScript, Python, Go, and Rust. Other indexed extensions (.java, .rb, .lua) fall back to the TypeScript/C-family chunk patterns.

MCP Search Tools

index_codebase:

{
  "rootDir": "/path/to/project",
  "extensions": [".ts", ".js"],
  "ignorePatterns": ["node_modules", "dist"]
}

code_search:

{
  "query": "authentication middleware",
  "scope": "/path/to/project",
  "limit": 10,
  "followImports": true,
  "mode": "hybrid"
}

Returned results include file path, symbol name, kind, content, line numbers, score, and related chunks when import following is enabled.

Search Modes

Keyword mode is the default and requires no setup. It scores exact name matches highest, then prefix matches, then keyword-in-content, then trigram fuzzy similarity.

Vector mode uses a local embedding model (Xenova/all-MiniLM-L6-v2, a small 384-dimensional model). On first run the model downloads automatically from HuggingFace and caches locally. Vector search finds semantically related code even when keywords don't overlap.

Hybrid mode runs all three strategies in parallel and fuses the rankings with Reciprocal Rank Fusion (RRF). Results include rrfScore (the fused score) and methodCount (how many strategies found the result). Items found by multiple methods rank higher, giving the best overall coverage.

Embedding Model

• Model: Xenova/all-MiniLM-L6-v2 (small, 384-dim)

• Backend: @xenova/transformers (ONNX runtime, runs locally)

• First run: model auto-downloads and caches to ~/.cache/huggingface/

• Fallback: if the local model fails to load, the embedder can fall back to OpenAI API (text-embedding-3-small) when OPENAI_API_KEY is set

Environment Variables

CRDT Multi-Agent State

The shared_state MCP tool gives agents a conflict-free shared state layer.

Supported CRDTs

Example Operations

{ "op": "write", "key": "tokens", "type": "g-counter", "nodeId": "agent-1", "amount": 150 }
{ "op": "write", "key": "status", "type": "lww-register", "nodeId": "agent-1", "value": "analyzing" }
{ "op": "write", "key": "findings", "type": "or-set", "nodeId": "agent-1", "action": "add", "element": "Issue in auth.ts:42" }
{ "op": "read", "key": "findings" }
{ "op": "snapshot" }
{ "op": "merge", "snapshot": { "entries": {} } }

This is useful when multiple coding or review agents need to coordinate without central locking.

Integrations

OpenCode

Configure OpenCode to treat the gateway as an OpenAI-compatible provider.

{
  "provider": {
    "llm-gateway": {
      "name": "LLM Gateway",
      "api": "openai",
      "apiKey": "env:LLM_GATEWAY_TOKEN",
      "baseURL": "https://llm-gateway.yourdomain.com/v1",
      "models": {
        "gateway-anthropic": {
          "name": "Anthropic via Gateway",
          "id": "claude-sonnet-4-20250514",
          "contextWindow": 200000,
          "maxOutput": 8192
        },
        "gateway-groq": {
          "name": "Groq via Gateway",
          "id": "llama-3.3-70b-versatile",
          "contextWindow": 128000,
          "maxOutput": 4096
        }
      }
    }
  }
}

export LLM_GATEWAY_TOKEN="your-gateway-auth-token"
opencode

GHAGGA

GHAGGA can use the bridge as a provider.

1. Select LLM Gateway in the GHAGGA dashboard.

2. Enter the gateway base URL.

3. Enter the gateway bearer token.

4. Pick a model.

Typical review modes routed through the gateway:

• simple

• workflow

• consensus

Any OpenAI-Compatible Tool

General settings:

Works with LangChain, LlamaIndex, Cursor, Continue, and any HTTP client that can call /v1/chat/completions.

LangChain Python example:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://llm-gateway.yourdomain.com/v1",
    api_key="your-gateway-token",
    model="claude-sonnet-4-20250514",
)

response = llm.invoke("Explain quicksort")
print(response.content)

LangChain TypeScript example:

import { ChatOpenAI } from '@langchain/openai';

const llm = new ChatOpenAI({
  configuration: {
    baseURL: 'https://llm-gateway.yourdomain.com/v1',
  },
  apiKey: 'your-gateway-token',
  model: 'claude-sonnet-4-20250514',
});

const response = await llm.invoke('Explain quicksort');

Docker Deployment

Docker Compose

services:
  llm-gateway:
    build: .
    ports:
      - "3456:3456"
    volumes:
      - llm-data:/root/.llm-gateway
    environment:
      - LLM_GATEWAY_PORT=3456
      - LLM_GATEWAY_AUTH_TOKEN=your-secure-token-here
      - LLM_GATEWAY_MASTER_KEY=your-64-char-hex-key
volumes:
  llm-data:

docker compose up -d

Docker Build and Run

docker build -t llm-gateway .

docker run -d \
  -p 3456:3456 \
  -v llm-data:/root/.llm-gateway \
  -e LLM_GATEWAY_AUTH_TOKEN="your-token" \
  -e LLM_GATEWAY_MASTER_KEY="your-64-char-hex-key" \
  llm-gateway

What the Image Includes

The Dockerfile currently installs:

• pnpm 9

• OpenCode CLI

• Claude Code CLI

• Gemini CLI

• Codex CLI

• Qwen CLI

• GitHub Copilot CLI

Coolify

1. Create a new service pointing at this repository.

2. Use the Dockerfile build pack.

3. Set environment variables such as LLM_GATEWAY_PORT, LLM_GATEWAY_AUTH_TOKEN, and optionally LLM_GATEWAY_MASTER_KEY.

4. Mount a persistent volume at /root/.llm-gateway.

5. Use /health for health checks.

MCP Server

The project runs as an MCP stdio server by default.

Primary MCP Tools

PageIndex Conversation Tools

Seven additional static MCP tools (defined in src/pageindex/tools.ts) handle long-conversation pagination and reasoning-based navigation over stored conversation history:

These are categorized as read tools, so they are available under both local-dev and restricted security profiles.

Claude Code Config

Add to ~/.config/claude/mcp.json:

{
  "mcpServers": {
    "llm-bridge": {
      "command": "mcp-llm-bridge"
    }
  }
}

For a local source checkout:

{
  "mcpServers": {
    "llm-bridge": {
      "command": "npx",
      "args": ["tsx", "/path/to/mcp-llm-bridge/src/index.ts"]
    }
  }
}

Claude Desktop Config

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "llm-bridge": {
      "command": "mcp-llm-bridge"
    }
  }
}

MCP stdio runs locally and does not use the HTTP bearer-token middleware.

Dynamic MCP Servers

The bridge supports loading external .mcp-server.js plugin files at runtime. This lets you extend the toolset without modifying the core codebase.

What It Is

Any .mcp-server.js file placed in the plugin directory is loaded at startup and its tools are registered alongside the static tools (30 at time of writing: 23 core tools plus 7 PageIndex conversation tools). Plugins export a McpServerDefinition object (or use the builder) with tools, resources, and prompts.

Enable

Set MCP_DYNAMIC_SERVERS=true:

export MCP_DYNAMIC_SERVERS=true
export MCP_SERVERS_DIR=./mcp-servers

Create a Plugin

Create a .mcp-server.js file in the plugin directory:

import { McpServerBuilder } from 'mcp-llm-bridge/mcp-builder';

export default new McpServerBuilder()
  .tool('greet', 'Say hello to someone', { name: { type: 'string' } }, async ({ name }) => {
    return { content: [{ type: 'text', text: `Hello, ${name}!` }] };
  })
  .build();

The builder validates naming conventions, schema completeness, and description quality. Tools are registered on the MCP server and appear in ListTools.

Directory

The default plugin directory is ./mcp-servers. Override with:

export MCP_SERVERS_DIR=./my-custom-plugins

Security

Dynamic tools are registered with the read category by default. This means they are:

• Allowed under local-dev and restricted profiles

• Blocked under the open profile (which only allows generate tools)

The enforcer applies the same category-based filtering to dynamic tools as it does to static tools.

Coexistence with Static Tools

Static tools (vault, search, generate, etc.) and dynamic tools appear together in the ListTools response. There is no namespacing — tool names must be unique across both sets. The approval flow and rate limiting apply uniformly to all tools.

Configuration

Core Environment Variables

Optional Runtime Features

Master Key Priority

1. LLM_GATEWAY_MASTER_KEY

2. existing ~/.llm-gateway/master.key

3. auto-generated new key written with mode 0600

If you lose the master key, stored credentials are unrecoverable. Back it up in production.

Bridge Config Path

~/.llm-gateway/bridge.yaml

If that file does not exist, bridge routing is disabled.

Security Profiles

Security profiles enforce trust-level-based access control on both MCP tools and HTTP endpoints. Three profiles are built-in:

Configuration

Set via environment variable:

LLM_GATEWAY_SECURITY_PROFILE=restricted

Default is local-dev (backward compatible — no restrictions).

Each profile also carries a sandbox flag (default false). Today this flag is best understood as prepared infrastructure, not a guarantee of sandboxed runtime execution: it is exposed on the profile schema and through the admin API, and the repo includes a Docker/process sandbox runner under src/sandbox/, but the active runtime does not yet expose sandboxed execution tools or route normal tool execution through that runner. Also note that the helper falls back to plain process execution with a timeout when Docker is unavailable, so this should not be treated as complete containment.

HTTP Enforcement

Under restricted or open, the gateway blocks destructive HTTP endpoints (e.g., POST /v1/credentials) and returns:

{ "error": "Access denied: endpoint blocked by security profile", "code": "SECURITY_PROFILE_DENIED" }

Read endpoints (GET /v1/providers, GET /v1/models) remain open under restricted.

MCP Enforcement

Under non-local-dev profiles, ListTools returns only tools in the allowed categories. CallTool is authorized before execution. Rate limiting is applied per profile.

Approval Flows

Destructive MCP tools can be paused for explicit human approval when the security profile is not local-dev.

How It Works

1. Client calls a destructive tool (e.g., vault_store).

2. If approval is required, the gateway returns an approvalRequired payload with a requestId.

3. Admin reviews pending requests via GET /v1/approvals or approval_list MCP tool.

4. Admin approves or denies via POST /v1/approvals/:id/approve or approval_approve MCP tool.

5. Original tool executes only after approval.

Auto-Approve List

Read-only tools (file_read, search, list, vault_list) bypass approval automatically.

HTTP Endpoints

MCP Tools

Three-Part Prompt

The three-part prompt pattern separates prompts into system (role/constraints), context (background data), and instruction (the actual task). Research shows measurable quality improvement, especially with smaller models.

HTTP API

Both /v1/generate and /v1/chat/completions accept the three fields:

curl -X POST http://localhost:3456/v1/generate \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -d '{
    "system": "You are a code reviewer.",
    "context": "We use Zod 4 and Hono.",
    "instruction": "Review this schema for edge cases."
  }'

Legacy flat prompt is still accepted and auto-detected when system/context/instruction are absent.

MCP Schema

The llm_generate tool exposes system, context, and instruction as optional fields alongside the legacy prompt:

{
  "system": "You are a helpful assistant.",
  "context": "The project uses TypeScript.",
  "instruction": "Explain strict mode benefits."
}

Enable/Disable

OPTIMIZE_MESSAGES_ENABLED=true   # default: true

RTK Output Compression

RTK-style compression strips redundant content from tool call results before passing them to LLMs. This saves token budget on large structured outputs.

Strategies

1. Filter — remove noise fields (created_at, id, etag, etc.)

2. Group — merge repeated similar entries into count + sample

3. Truncate — enforce max length on string values

4. Deduplicate — remove exact-duplicate array entries

Configuration

ENABLE_OUTPUT_COMPRESSION=true   # default: true

Analytics Endpoint

curl http://localhost:3456/v1/compression/stats \
  -H 'Authorization: Bearer YOUR_TOKEN'

Response:

{ "totalCalls": 42, "compressedCalls": 42, "avgRatio": 0.65, "totalSavingsChars": 15200 }

Local LLM Offloading

Offloadable tasks (summarization, formatting, classification) can be routed to local runtimes (Ollama, LM Studio) instead of cloud providers, cutting API token cost on those deterministic tasks. (The src/local-llm/ module documents an 86–95% design target for token savings on boilerplate tasks; this is a design goal, not a measured benchmark.)

Environment Variables

Detection

At startup, the gateway probes both backends. Models are listed at:

curl http://localhost:3456/v1/local/models \
  -H 'Authorization: Bearer YOUR_TOKEN'

Fallback

If the local LLM fails or the task is not offloadable, the gateway falls back to cloud providers automatically and emits a metric.

MCP Tool

Model Routing

Model routing adds task-aware provider selection that classifies each prompt and routes it based on configured rules, preferred endpoint order, cost tiers, and observed quality feedback.

What It Does

• Classifies incoming prompts into runtime task types such as code-review, large-context, fast-completion, summarization, and translation

• Matches the task against routing rules defined in model-routing.json

• Tries preferred endpoints in rule order while enforcing the configured cost cap

• Falls back to more expensive endpoints if quality drops below threshold

• Learns from feedback — records success/failure per endpoint+task for adaptive routing

Enable

MODEL_ROUTING_ENABLED=true

When enabled, the precedence stack becomes:

1. Session stickiness

2. Group-based routing

3. ModelRouter (task-aware selection)

4. Local-LLM offloading (only if ModelRouter is disabled or returns no match)

5. Standard resolution (model match → provider preference → API before CLI)

6. Latency reordering

Configuration

Create model-routing.json in the project root. The gateway loads it at startup.

{
  "enabled": true,
  "endpoints": [...],
  "rules": [...],
  "defaultEndpoint": "opencode-cli-default",
  "qualityThreshold": 0.7,
  "qualityWindowSize": 50
}

Endpoint fields:

Rule fields:

Example Task-to-Endpoint Mappings

Coexistence with Local-LLM Offloading

Local-LLM offloading and model routing work together with clear precedence:

• ModelRouter runs first. If it selects an endpoint, that provider is promoted to the top of the candidate list.

• Local-LLM offloading runs only when ModelRouter is disabled or returns no match. This prevents conflicts: explicit routing rules always beat heuristic offloading.

If you want local models in your routing mix, register them as endpoints with "costTier": "free" and include them in rule preferredEndpoints.

Example File

See model-routing.example.json in the repository root for a full template with multiple endpoints and routing rules.

HF Auto-Discovery

At startup (when enabled), the gateway scans local backends and enriches detected models with HuggingFace metadata (tags, pipeline type, recommended tasks).

Configuration

AUTO_DISCOVER_MODELS=true   # default: false
HF_TOKEN=hf_xxxxxxxxxx       # optional, for private repos

Admin Endpoint

Trigger discovery on demand:

curl -X POST http://localhost:3456/v1/admin/discover \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{ "hfToken": "optional-override" }'

Response:

{
  "ok": true,
  "backendsScanned": ["ollama", "lm-studio"],
  "models": [...],
  "enrichedCount": 3,
  "unenrichedCount": 1
}

Cache

Enriched metadata is persisted to SQLite (hf_model_cache table) so subsequent startups are fast even without HF API access.

Architecture

Clients (GHAGGA, OpenCode, curl, LangChain, any OpenAI-compatible tool)
    |
    |  POST /v1/chat/completions  |  POST /v1/generate  |  MCP stdio
    v
+-------------------------------------------------------------------+
|                    MCP LLM Bridge (Hono + MCP)                    |
|                                                                   |
|  HTTP Server                       MCP Server                     |
|  - /v1/chat/completions            - llm_generate                 |
|  - /v1/generate                    - vault_*                      |
|  - /v1/models                      - code_search                  |
|  - /v1/providers                   - index_codebase               |
|  - /v1/credentials CRUD            - shared_state                 |
|  - /v1/files CRUD                  - usage_*                      |
|  - /v1/groups CRUD                 - circuit_breaker_*            |
|  - /metrics /health                - group tools                  |
|  - /v1/compression/stats           - approval_*                   |
|  - /v1/local/models                - local_llm_generate           |
|  - /v1/admin/discover              - discover_models              |
+-------------------------------------------------------------------+
|  Bridge routing         | Context compression | Code search        |
|  Provider groups        | Cost tracking       | CRDT state         |
|  Security profiles      | Approval flows      | Local LLM          |
|  HF discovery           | Three-part prompt   | Output compression |
+-------------------------+---------------------+--------------------+
| Router (model -> provider)       | Vault (AES-256-GCM + SQLite)   |
+-------------------------+---------------------+--------------------+
    |                                                  |
    v                                                  v
  API providers                                   CLI providers
  Anthropic, OpenAI, Google, Groq, OpenRouter     OpenCode, Claude,
                                                   Gemini, Codex, Qwen, Copilot

Design Notes

• Hono keeps the HTTP layer small and fast.

• better-sqlite3 keeps the vault single-file and operationally simple.

• SQLite WAL mode improves concurrent read behavior.

• API providers are preferred before CLI providers unless bridge logic says otherwise.

• Vault writes use upsert semantics for repeatable automation.

• CLI adapters materialize auth files into temp homes and clean them up in finally blocks.

• Bridge routing is intentionally optional and file-driven.

• Code search stays in-memory for speed and freshness.

• CRDTs reduce coordination pain for parallel agent workflows.

Experimental Modules

• src/acp/ — Agent Client Protocol implementation (server.ts, translator.ts, types.ts).
  Present in the repo but not wired into the active runtime. There is no import path from src/index.ts, no active HTTP/stdio ACP transport, and no live MCP tool-execution bridge yet. Treat it as a tested protocol prototype that still needs a dedicated ACP integration sprint.

• src/sandbox/ — Docker/process sandbox runner.
  The sandbox flag now exists in security profiles, but the runtime still does not expose sandboxed execution tools like execute_code or shell_command. In other words: the infrastructure is prepared, but the feature is not complete.

Session Systems

The gateway now uses SessionManager for both session-affinity concerns:

1. Router sticky sessions (SessionManager.pinRouterStickySession) — Pins a specific clientId + model to a provider/key with TTL-based expiry.

2. Group/API sessions (src/session/session-manager.ts) — Manages session affinity for multi-turn conversations and dashboard metrics.

They are separate by design inside the same manager instance: router stickiness handles provider selection, while group/API sessions handle conversation continuity.
Do not conflate the two entry kinds.

GET /v1/admin/sessions reports them separately for that reason:

• routerStickySessions comes from SessionManager router-sticky entries and reflects the pins the Router actually uses at request time.

• groupSessions comes from SessionManager and reflects group-level session affinity metrics.

• The endpoint includes a note explaining the split so the dashboard does not imply a single shared session pool.

Security

• AES-256-GCM encryption for stored keys and auth files

• constant-time bearer-token comparison

• master key file stored with mode 0600

• config directory created with mode 0700

• credentials are never returned raw from listing endpoints

• temp auth files are cleaned up after CLI invocations

• minimum 32-character auth token requirement

• public /health endpoint for safe monitoring

Development

pnpm run dev
pnpm run serve
pnpm run start
pnpm test
pnpm run typecheck
pnpm run build

Scripts

Requirements

• Node.js 22+

• pnpm 9+

License

MIT, as declared in package.json ("license": "MIT"). Note: a standalone LICENSE file is not yet committed to the repository.