MCP vLLM Benchmarking Tool
MCP vLLM ベンチマークツール
これは、MCP を使用して vLLM をインタラクティブにベンチマークする方法の概念実証です。
私たちはベンチマークに新しいわけではありません。私たちのブログをお読みください。
これは、MCP の可能性を探る試みにすぎません。
使用法
リポジトリをクローンする
MCP サーバーに追加します:
{
"mcpServers": {
"mcp-vllm": {
"command": "uv",
"args": [
"run",
"/Path/TO/mcp-vllm-benchmarking-tool/server.py"
]
}
}
}次に、たとえば次のようにプロンプトできます。
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.Related MCP server: metrillm-mcp
やること:
vllm のランダムな出力により、無効な JSON が検出された可能性があります。まだ詳しく調べていません。
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
Related MCP Servers
- AlicenseNot gradedqualityNot gradedmaintenanceEnables benchmarking of Large Language Model APIs by measuring performance metrics such as generation throughput, prompt throughput, and Time To First Token (TTFT) with configurable concurrency levels and parameters.1MIT
- AlicenseNot gradedqualityDmaintenanceEnables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.16 npm8Apache 2.0
- AlicenseAqualityBmaintenanceEnables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.8MIT
- AlicenseNot gradedqualityBmaintenanceVendor-neutral local LLM inference benchmark and hardware-config advisor for mlx and llama.cpp. Exposes an MCP tool that measures real tokens/second on your own hardware.4 npmApache 2.0