ML Task Router MCP Server
Provides streaming chat capabilities using OpenAI's GPT models with automatic retry logic and rate limit handling for conversational AI tasks.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
π§ MCP Server (Model Compute Paradigm)
A modular, production-ready FastAPI server built to route and orchestrate multiple AI/LLM-powered models behind a unified, scalable interface. It supports streaming chat, LLM-based routing, and multi-model pipelines (like analyze β summarize β recommend) β all asynchronously and fully Dockerized.
π― Project Score (Production Readiness)
Capability | Status | Details |
π§ Multi-Model Orchestration | β Complete | Dynamic routing between |
π€ LLM-Based Task Router | β Complete | GPT-powered routing via |
π Async FastAPI + Concurrency | β Complete | Async/await + concurrent task execution with simulated/model API delays |
π GPT Streaming Support | β Complete |
|
π§ͺ Unit + Mocked API Tests | β Complete | Pytest-based test suite with mocked |
π³ Dockerized + Clean Layout | β Complete | Python 3.13 base image, no Conda dependency, production-ready Dockerfile |
π¦ Metadata-Driven Registry | β Complete | Model metadata loaded from external YAML config |
π Rate Limiting & Retry | β³ In Progress | Handles 429 retry loop; rate limiting controls WIP |
π§ͺ CI + Docs | β³ Next | GitHub Actions + Swagger/Redoc planned |
Related MCP server: CbetaMCP
π§© Why This Project? (Motivation)
Modern ML/LLM deployments often involve:
Multiple task types and model backends (OpenAI, HF, local, REST)
Routing decisions based on input intent
Combining outputs of multiple models (e.g.,
summarize+recommend)Handling 429 retries, async concurrency, streaming responses
π§ However, building such an LLM backend API server that is:
Async + concurrent
Streamable
Pluggable (via metadata)
Testable
Dockerized β¦ is non-trivial and not easily found in one single place.
π‘ What Weβve Built (Solution)
This repo is a production-ready PoC of an MCP (Model-Compute Paradigm) architecture:
β FastAPI-based microserver to handle multiple tasks via
/taskendpointβ Task router that can:
π Dispatch to specific model types (
chat,sentiment,summarize,recommend)π€ Use an LLM to infer which task to run (
auto)π§ Run multiple models in sequence (
analyze)
β GPT streaming via
text/event-streamβ Async/await enabled architecture for concurrency
β Clean modular code for easy extension
β Dockerized for deployment
β Tested using Pytest with mocking
π οΈ Use Cases
Use Case | MCP Server Support |
Build your own ChatGPT-style API | β
|
Build intelligent task router | β
|
Build AI pipelines (like RAG/RL) | β
|
Swap between OpenAI/HuggingFace APIs | β
Via |
Add custom models (e.g., OCR, vision) | β Just add a new module + registry entry |
π Features
β Async FastAPI server
π§ Task-based Model Routing (
chat,sentiment,recommender,summarize)π Model Registry from YAML/JSON
π Automatic Retry and Rate Limit Handling for APIs
π Streaming Responses for Chat
π§ͺ Unit Tests + Mocked API Calls
π³ Dockerized for production deployment
π¦ Modular structure, ready for CI/CD
π Architecture Overview
ββββββββββββββ
β Frontend β
βββββββ¬βββββββ
β
βΌ
ββββββββββββββ YAML/JSON
β FastAPI βββββββ Model Registry
β Server β β
βββββββ¬βββββββ βΌ
ββββββββββββββββΌβββββββββββββββ
β β β
βΌ βΌ βΌ
[chat] [sentiment] [recommender]
GPT-4 HF pipeline stub logic / API
---
π Setup
π¦ Install dependencies
git clone https://github.com/YOUR_USERNAME/mcp-server.git
cd mcp-server
---
# Optional: create virtualenv
python -m venv .venv
source .venv/bin/activate # or .venv\Scripts\activate on Windows
or
conda create -n <env_name>
conda activate <env_name>
pip install -r requirements.txt
βΆοΈ Run the server
uvicorn app:app --reload
Access the docs at: http://localhost:8000/docs
π§ͺ Running Tests
pytest tests/
Unit tests mock external API calls using unittest.mock.AsyncMock.
π³ Docker Support
π¨ Build image
docker build -t mcp-server .
π Run container
docker run -p 8000:8000 mcp-server
π§° Example API Request
curl -X POST http://localhost:8000/task \
-H "Content-Type: application/json" \
-d '{
"type": "chat",
"input": "What are the benefits of restorative yoga?"
}'
π Directory Structure
mcp/
βββ app.py # FastAPI entry
βββ models/ # ML models (chat, sentiment, etc.)
βββ agent/
β βββ task_router.py # Task router
β βββ model_registry.py # Registry loader
βββ registry/models.yaml # YAML registry of model metadata
βββ tests/ # Unit tests
βββ Dockerfile
βββ requirements.txt
βββ README.md
βββ .env / .gitignore
π€ Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what youβd like to change.
π License
MIT
β¨ Author
Built by Sriram Kumar Reddy ChallaTool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
This server cannot be deployed
Maintenance
Related MCP Connectors
OpenAI-compatible BYOK LLM gateway: per-task routing, shadow evaluation, promotion gates.
AI model routing on your own vendor keys: pick the best model per prompt, or route and run it.
AI routing, memory, guardrails, and governance. Routes across Claude, GPT, Gemini.
Chat, image, video, and audio models from many vendors behind one remote MCP server and one key.
Related MCP Servers
- AlicenseBqualityDmaintenanceThis server facilitates the invocation of AI models from providers like Anthropic, OpenAI, and Groq, enabling users to manage and configure large language model interactions seamlessly.213MIT
- AlicenseNot gradedqualityDmaintenanceA modular, extensible FastAPI-based platform that aggregates multiple AI tools and microservices into a unified interface with standardized I/O formats, perfect for frontend integration or LLM system orchestration.4MIT
- -licenseNot gradedqualityNot gradedmaintenanceIntelligent routing service that selects optimal AI models based on capability requirements and normalizes input/output formats across multiple providers like OpenAI, Anthropic, Google, and others.-
- FlicenseNot gradedqualityCmaintenanceEnables agents to discover, evaluate, and serve task-specialist models (embeddings, reranking, classification, extraction) with OpenAI-compatible endpoints and MCP tools for routing, embedding, reranking, and extraction.-