A
licenseNot graded
qualityC
maintenanceAn MCP server that allows agents to test and compare LLM prompts across OpenAI and Anthropic models, supporting single tests, side-by-side comparisons, and multi-turn conversations.
MIT
这是关于如何使用 MCP 以交互方式对 vLLM 进行基准测试的概念证明。
我们对基准测试并不陌生,请阅读我们的博客:
这只是对 MCP 可能性的探索。
克隆存储库
将其添加到您的 MCP 服务器:
{
"mcpServers": {
"mcp-vllm": {
"command": "uv",
"args": [
"run",
"/Path/TO/mcp-vllm-benchmarking-tool/server.py"
]
}
}
}然后你可以像这样提示:
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.Related MCP server: LLM API Benchmark MCP Server
由于 vllm 的一些随机输出,它可能显示发现了一些无效的 json。我还没有深入研究过这个问题。
This server cannot be deployed
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
A paid remote MCP for AI SDK benchmark dashboard, built to return verdicts, receipts, usage logs, an