Skip to main content
Glama
MCP vLLM Benchmarking Tool

MCP vLLM Benchmarking Tool

by Eliovp-BV

MCP vLLM Benchmarking Tool

This is proof of concept on how to use MCP to interactively benchmark vLLM.

We are not new to benchmarking, read our blog:

Benchmarking vLLM

This is just an exploration of possibilities with MCP.

Usage

  1. Clone the repository

  2. Add it to your MCP servers:

{
    "mcpServers": {
        "mcp-vllm": {
            "command": "uv",
            "args": [
                "run",
                "/Path/TO/mcp-vllm-benchmarking-tool/server.py"
            ]
        }
    }
}

Then you can prompt for example like this:

Do a vllm benchmark for this endpoint: http://10.0.101.39:8888 
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B 
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.

Related MCP server: metrillm-mcp

Todo:

  • Due to some random outputs by vllm it may show that it found some invalid json. I have not really looked into it yet.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.
      16 npm
      8
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Vendor-neutral local LLM inference benchmark and hardware-config advisor for mlx and llama.cpp. Exposes an MCP tool that measures real tokens/second on your own hardware.
      11 npm
      Apache 2.0