A
licenseNot graded
qualityC
maintenanceAn MCP server that allows agents to test and compare LLM prompts across OpenAI and Anthropic models, supporting single tests, side-by-side comparisons, and multi-turn conversations.
MIT
MCP vLLM Benchmarking Tool
by Eliovp-BV
MCP vLLM 벤치마킹 도구
이는 MCP를 사용하여 vLLM을 대화형으로 벤치마킹하는 방법에 대한 개념 증명입니다.
우리는 벤치마킹에 익숙하지 않습니다. 저희 블로그를 읽어보세요.
이는 단지 MCP의 가능성을 탐구하는 것입니다.
용법
저장소를 복제합니다
MCP 서버에 추가하세요:
지엑스피1
그러면 다음과 같이 프롬프트를 표시할 수 있습니다.
Do a vllm benchmark for this endpoint: http://10.0.101.39:8888
benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
run the benchmark 3 times with each 32 num prompts, then compare the results, but ignore the first iteration as that is just a warmup.Related MCP server: LLM API Benchmark MCP Server
할 일:
vllm의 무작위 출력으로 인해 잘못된 JSON을 발견했을 수 있습니다. 아직 자세히 살펴보지는 않았습니다.
This server cannot be deployed
Maintenance
ActivityInactive
ResponsivenessNo issues
Related MCP Connectors
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
A paid remote MCP for AI SDK benchmark dashboard, built to return verdicts, receipts, usage logs, an
Related MCP Servers
- AlicenseNot gradedqualityNot gradedmaintenanceEnables benchmarking of Large Language Model APIs by measuring performance metrics such as generation throughput, prompt throughput, and Time To First Token (TTFT) with configurable concurrency levels and parameters.1MIT
- AlicenseNot gradedqualityCmaintenanceEnables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.8 npm8Apache 2.0
- AlicenseBqualityAmaintenanceEnables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.3MIT