Skip to main content
Glama
Eliovp-BV

MCP vLLM Benchmarking Tool

by Eliovp-BV

MCP vLLM 基准测试工具

这是关于如何使用 MCP 以交互方式对 vLLM 进行基准测试的概念证明。

我们对基准测试并不陌生,请阅读我们的博客:

对 vLLM 进行基准测试

这只是对 MCP 可能性的探索。

用法

  1. 克隆存储库

  2. 将其添加到您的 MCP 服务器:

{
    "mcpServers": {
        "mcp-vllm": {
            "command": "uv",
            "args": [
                "run",
                "/Path/TO/mcp-vllm-benchmarking-tool/server.py"
            ]
        }
    }
}

然后你可以像这样提示:

Do a vllm benchmark for this endpoint: http://10.0.101.39:8888 
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B 
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.

Related MCP server: LLM API Benchmark MCP Server

待办事项:

  • 由于 vllm 的一些随机输出,它可能显示发现了一些无效的 json。我还没有深入研究过这个问题。

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    An MCP server that allows agents to test and compare LLM prompts across OpenAI and Anthropic models, supporting single tests, side-by-side comparisons, and multi-turn conversations.
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.
    8 npm
    8
    Apache 2.0
  • A
    license
    B
    quality
    A
    maintenance
    Enables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.
    3
    MIT