Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@MCP vLLM Benchmarking Tool benchmark deepseek-ai/DeepSeek-R1-Distill-Llama-8B on http://10.0.101.39:8888 with 32 prompts, 3 runs, skip warmup"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
This is proof of concept on how to use MCP to interactively benchmark vLLM.
We are not new to benchmarking, read our blog:
This is just an exploration of possibilities with MCP.
Clone the repository
Add it to your MCP servers:
{
"mcpServers": {
"mcp-vllm": {
"command": "uv",
"args": [
"run",
"/Path/TO/mcp-vllm-benchmarking-tool/server.py"
]
}
}
}Then you can prompt for example like this:
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.Related MCP server: MCP Prompt Tester
Due to some random outputs by vllm it may show that it found some invalid json. I have not really looked into it yet.
This server cannot be installed
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
A paid remote MCP for AI SDK benchmark dashboard, built to return verdicts, receipts, usage logs, an
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/Eliovp-BV/mcp-vllm-benchmark'
If you have feedback or need assistance with the MCP directory API, please join our Discord server