Enables benchmarking of Large Language Model APIs by measuring performance metrics such as generation throughput, prompt throughput, and Time To First Token (TTFT) with configurable concurrency levels and parameters.
Enables building and running custom LLM benchmarks with multi-judge evaluation, supporting GUI, MCP client, and CLI usage for ranked, auditable results.
Vendor-neutral local LLM inference benchmark and hardware-config advisor for mlx and llama.cpp. Exposes an MCP tool that measures real tokens/second on your own hardware.
Provides an MCP-compatible profiler for measuring edge and on-device LLM inference latency, including TTFT, TPOT, tokens-per-second throughput, and P50/P90/P99 jitter distributions. It enables agents and developers to run these telemetry benchmarks from MCP clients such as Claude Desktop, Cursor, and Windsurf.
Enables users to benchmark AI models across English, Urdu, and Roman Urdu by running question sets, grading answers, measuring confidently wrong and hedged responses, and producing reports, comparisons, and CSV exports through an MCP client or CLI.