Multi-Cloud LLM Platform

A provider-agnostic LLM routing layer spanning AWS Bedrock, OpenAI, and local Ollama models, instrumented with Prometheus metrics, wrapped in a LangGraph iterative research agent, and exposed to any MCP-compatible client (e.g. Claude Desktop) as callable tools over an MCP stdio server.

Overview

              ┌─────────────── src/providers.py ───────────────┐
              │  claude-sonnet-bedrock   (AWS Bedrock)          │
prompt ──────▶│  llama3-bedrock          (AWS Bedrock)          │──▶ LLMResponse
   or          │  gemini-flash-vertex     (low-cost tier)        │    (+ Prometheus metrics:
route(task) ─▶│  llama3-local            (Ollama, local/free)   │     requests, latency, cost, tokens)
              └──────────────────────────────────────────────────┘
                          ▲                        ▲
                          │                        │
              agents/research_agent.py      mcp_server/server.py
              (LangGraph loop: gather        (exposes generate/route/
               → evaluate → write report)     list_providers as MCP tools)

route(prompt, task_type) picks a provider based on task type (code/reason → Claude Sonnet on Bedrock, summarize → Llama 3 on Bedrock, low-cost → the low-cost tier, general → local Ollama), so callers don't need to know which backend is cheapest or best suited for a given job.

Related MCP server: MCP-AI-Gateway

Project structure

.
├── src/
│   ├── providers.py       # Provider catalog + generate()/route(), Prometheus instrumentation
│   └── metrics.py         # Prometheus Counter/Histogram definitions
├── agents/
│   └── research_agent.py  # LangGraph agent: iteratively researches a topic, then writes a report
├── mcp_server/
│   └── server.py          # MCP stdio server exposing generate/route/list_providers as tools
├── tests/
│   └── test_platform.py   # Provider registration, routing table, and research-agent tests
├── run_mcp.sh              # Convenience launcher for the MCP server
└── requirements.txt

Providers

Every call increments Prometheus counters/histograms for request count, latency, estimated cost, and token usage, labeled by provider.

Research agent

agents/research_agent.py builds a small LangGraph loop:

1. gather_information — asks the model (local Llama 3 by default) for 3–5 new facts on the topic not already covered.

2. evaluate_sufficiency — asks the model whether enough has been gathered for a 3-paragraph report.

3. Loops back to step 1 (up to 3 iterations) or proceeds to write_report, which synthesizes all notes into the final report.

MCP server

mcp_server/server.py runs an MCP stdio server (multi-cloud-llm-platform) that advertises three tools — generate, route, and list_providers — so any MCP client can call these LLM providers directly. Launch it with:

python -m mcp_server.server

Setup

python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

• AWS Bedrock (claude-sonnet-bedrock, llama3-bedrock): configure AWS credentials (~/.aws/credentials, env vars, or an IAM role) with Bedrock model access enabled in us-east-1 for the Claude 3.5 Sonnet and Llama 3 models.

• Low-cost tier (gemini-flash-vertex): requires export GOOGLE_API_KEY=<your-google-ai-studio-key>.

• Local (llama3-local): install Ollama and run ollama pull llama3.

Usage

# Run a single generation or routed call
python -c "from src.providers import route; print(route('Summarize this quarter', task_type='summarize').text)"

# Run the iterative research agent
python -m agents.research_agent

# Run the MCP server
./run_mcp.sh   # or: python -m mcp_server.server

# Run tests
pytest

Tech stack

LangChain, LangGraph, AWS Bedrock (boto3), Google Gemini, Ollama, MCP (mcp), Prometheus client, FastAPI/uvicorn, pytest.