⚡ ArmBench — Arm64 LLM Inference Benchmark Suite + MCP Server

KleidiAI-optimized LLM benchmarking and inference server for Arm64 cloud infrastructure. Built for the Arm AI Optimization Challenge 2026.

🎯 What is ArmBench?

ArmBench is a one-command benchmarking tool that:

1. Deploys LLMs (Llama 3.2) on Arm64 cloud instances using llama.cpp + KleidiAI

2. Measures real performance — tokens/sec, time-to-first-token, memory usage across quantization levels (Q4_K_M vs Q8_0)

3. Serves results via an MCP-compatible FastAPI server any agent framework can call

4. Visualizes everything in a clean real-time dashboard

Related MCP server: Hugging Face MCP Server

🏗️ Architecture

armbench/

├── benchmark/ # llama.cpp + KleidiAI inference engine + metrics

├── mcp_server/ # FastAPI MCP-compatible LLM endpoint

├── dashboard/ # Real-time results dashboard (HTML)

├── scripts/ # One-command setup + benchmark + server scripts

└── docker/ # Arm64-optimized Docker configuration

🚀 Quick Start (Arm64 Instance)

1. Clone and setup

git clone https://github.com/sirmos/armbench.git
cd armbench
bash scripts/setup.sh

2. Run benchmark

bash scripts/run_benchmark.sh

3. Start MCP server

bash scripts/start_mcp.sh

4. Open dashboard

Navigate to http://your-instance-ip:8000 in your browser.

☁️ Tested Arm64 Platforms

📊 What We Benchmark

Models

🔌 MCP Server API

Example: Generate

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is KleidiAI?", "model": "Llama-3.2-3B-Q4_K_M"}'

⚙️ Arm-Specific Optimizations

• KleidiAI: Arm's optimized kernel library for ML workloads

• llama.cpp Arm SVE: Scalable Vector Extension support enabled at build time

• Native CPU tuning: -DLLAMA_NATIVE=ON compiles for exact CPU microarchitecture

• Thread optimization: Automatically uses all available Arm cores

📄 License

MIT License — see LICENSE

Built for the Arm AI Optimization Challenge 2026