MCP vLLM Benchmarking Tool

This is proof of concept on how to use MCP to interactively benchmark vLLM.

We are not new to benchmarking, read our blog:

This is just an exploration of possibilities with MCP.

Usage

Clone the repository
Add it to your MCP servers:

{
    "mcpServers": {
        "mcp-vllm": {
            "command": "uv",
            "args": [
                "run",
                "/Path/TO/mcp-vllm-benchmarking-tool/server.py"
            ]
        }
    }
}

Then you can prompt for example like this:

Do a vllm benchmark for this endpoint: http://10.0.101.39:8888 
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B 
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.

Todo:

Due to some random outputs by vllm it may show that it found some invalid json. I have not really looked into it yet.

This server cannot be installed

security - not tested

license - not found

quality - not tested

How are these scores calculated?

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

An interactive tool that enables users to benchmark vLLM endpoints through MCP, allowing performance testing of LLM models with customizable parameters.

Related MCP Servers

MCP Toolbox
ai-zerolab
-
security
A
license
-
quality
A comprehensive toolkit that enhances LLM capabilities through the Model Context Protocol, allowing LLMs to interact with external services including command-line operations, file management, Figma integration, and audio processing.
Last updated -
17
Python
Apache 2.0
MCP Prompt Tester
rt96-hub
-
security
A
license
-
quality
An MCP server that allows agents to test and compare LLM prompts across OpenAI and Anthropic models, supporting single tests, side-by-side comparisons, and multi-turn conversations.
Last updated -
Python
MIT License
ChuckNorris MCP Server
pollinations
A
security
F
license
A
quality
An experimental MCP gateway that provides specialized LLM enhancement prompts based on the L1B3RT4S repository, primarily intended to enhance weaker models' capabilities.
Last updated -
1
2,012
7
JavaScript
cutterMCP
ap425q
-
security
A
license
-
quality
An MCP server that enables LLMs to autonomously reverse engineer applications through Cutter, allowing them to decompile binaries, analyze code, and rename methods programmatically.
Last updated -
6
Python
Apache 2.0

View all related MCP servers

Appeared in Searches

A service for checking the weather forecast

MCP vLLM Benchmarking Tool