Skip to main content
Glama
0xConsole

ArmSight

by 0xConsole
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Hybrid

ArmSight

Autonomous AI agent for Arm64 ML model inference optimization. Analyzes ONNX models, applies REAL INT8 quantization, and generates Arm64-optimized deployment packages — all through an MCP-compatible tool interface that an AI agent can call autonomously.

Built for the Arm Create: AI Optimization Challenge (Cloud AI track).

🎯 The Problem

Deploying ML models on Arm64 (AWS Graviton, Cortex-A, Neoverse) requires platform-specific knowledge: which quantization scheme to use, how to tune thread parallelism for multi-core, when to leverage NEON SIMD, and how to package everything into an Arm64-optimized container. Most developers ship unoptimized FP32 models and leave significant performance on the table.

Related MCP server: ArmBench MCP Server

💡 The Solution

ArmSight is an autonomous AI agent that:

  1. Analyzes any ONNX model — operators, layers, precision, parameter count, input/output shapes

  2. Recommends Arm64-specific optimizations — INT8 quantization, NEON SIMD fusion, thread parallelism, memory layout, ACL provider

  3. Applies real INT8 dynamic quantization using onnxruntime.quantization — producing measurably smaller models (typically ~4x size reduction)

  4. Benchmarks before/after — real inference latency, throughput, and speedup measurements

  5. Generates a complete Arm64-optimized deployment package — Dockerfile (linux/arm64), FastAPI inference server, benchmark script

🏆 Unique Angle

Unlike generic model optimizers, ArmSight exposes its capabilities as MCP (Model Context Protocol) tools that an AI agent can call autonomously — analyze_model, optimize_model, benchmark_model, recommend_optimizations, generate_deployment, full_pipeline. This makes ArmSight not just a tool, but an agent-native optimization platform.

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Web UI (HTML/CSS/JS)                  │
│  Upload ONNX → Analyze → Recommend → Quantize → Benchmark    │
└──────────────────────────┬──────────────────────────────────┘
                           │ HTTP
┌──────────────────────────▼──────────────────────────────────┐
│                    FastAPI Backend (Python)                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────────┐  │
│  │ Analyzer │  │Quantizer │  │  Recomm. │  │  Deployment │  │
│  │ (onnx)   │  │(onnxrt)  │  │  Engine  │  │  Generator  │  │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └──────┬──────┘  │
│       └──────────┬───┴─────────────┴───────────────┘         │
│                  ▼                                           │
│         ┌──────────────────┐                                 │
│         │   MCP Server     │ ← AI agent calls these tools     │
│         │  (tool registry) │   autonomously via MCP protocol  │
│         └──────────────────┘                                 │
└──────────────────────────────────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────────┐
│                    Vercel (Serverless)                       │
│         FastAPI on Python runtime — free tier               │
└─────────────────────────────────────────────────────────────┘

⚡ Quick Start

Prerequisites

  • Python 3.9+

  • An ONNX model file (or use the built-in example model generator)

Setup (< 5 commands)

# 1. Clone
git clone https://github.com/0xConsole/arm-sight-agent.git
cd arm-sight-agent

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run locally
uvicorn app.main:app --reload --port 8000

# 4. Open the UI
open http://localhost:8000

Use via API / MCP

# List MCP tools (what an AI agent sees)
curl http://localhost:8000/mcp/tools | python -m json.tool

# Call the full pipeline autonomously (analyze → quantize → benchmark → deploy)
curl -X POST http://localhost:8000/mcp/call \
  -H "Content-Type: application/json" \
  -d '{"name": "analyze_model", "arguments": {"model_path": "examples/example_model.onnx"}}'

🛠️ Tech Stack

Component

Technology

Backend

Python + FastAPI

Model analysis

onnx + onnxruntime

Quantization

onnxruntime.quantization.quantize_dynamic (REAL INT8)

Agent interface

MCP (Model Context Protocol) tool pattern

Frontend

Vanilla HTML/CSS/JS (no framework)

Deployment

Vercel serverless (Python runtime)

Target platform

linux/arm64 (AWS Graviton, Cortex-A, Neoverse)

✅ What's Real vs. Mocked

Feature

Status

Notes

ONNX model analysis

✅ REAL

Uses onnx + onnxruntime to parse graph, count operators/parameters

INT8 quantization

✅ REAL

onnxruntime.quantization.quantize_dynamic — produces genuinely smaller ONNX files

Size measurement

✅ REAL

Byte-level before/after file size comparison

Inference benchmarking

✅ REAL

Actual session.run() timing on CPU (mean/p50/p95 latency, throughput)

Arm64 recommendations

✅ REAL

Based on actual model architecture (operators, precision, param count)

Deployment package

✅ REAL

Generates working Dockerfile targeting linux/arm64 + FastAPI server + benchmark script

MCP tool interface

✅ REAL

Tools are callable via POST /mcp/call — any MCP client can invoke them

Nothing is mocked. Every measurement comes from real ONNX runtime operations.

📊 Measurable Improvements (Example)

For a typical FP32 ONNX model:

Metric

Before (FP32)

After (INT8)

Improvement

Model size

~4.2 MB

~1.1 MB

4.0x reduction

Inference latency

~2.5 ms

~1.8 ms

~28% faster

Throughput

~400 ops/s

~550 ops/s

~37% higher

Actual numbers vary by model. The quantization and benchmarking are real — run it on your model to see your results.

🐳 Generated Deployment Package

The generate_deployment tool produces:

deploy_package/
├── Dockerfile          # linux/arm64 target, ONNX Runtime with NEON
├── server.py           # FastAPI inference server (optimized session options)
├── model.onnx          # Your (optionally quantized) model
├── benchmark.py        # Latency/throughput benchmark script
├── docker-compose.yml  # One-command deployment
└── README.md           # Usage instructions
# Build and run on Arm64
docker buildx build --platform linux/arm64 -t armsight-inference .
docker run --rm -p 8000:8000 armsight-inference
python benchmark.py http://localhost:8000

🔌 MCP Tool Reference

ArmSight exposes 6 tools via the MCP interface:

Tool

Description

analyze_model

Analyze ONNX architecture: operators, precision, params

optimize_model

Apply INT8 dynamic quantization (real size reduction)

benchmark_model

Measure inference latency and throughput

recommend_optimizations

Generate Arm64-specific recommendations

generate_deployment

Create Arm64 Docker + FastAPI deployment package

full_pipeline

Run all of the above autonomously

📁 Project Structure

arm-sight-agent/
├── api/
│   └── index.py          # Vercel serverless entry point
├── app/
│   ├── main.py           # FastAPI app + routes
│   ├── analyzer.py       # ONNX model analysis
│   ├── quantizer.py      # INT8 quantization (REAL)
│   ├── recommendations.py # Arm64 optimization recommendations
│   ├── deployment.py     # Deployment package generator
│   └── mcp_server.py     # MCP tool registry + dispatch
├── static/
│   └── index.html        # Web UI
├── requirements.txt
├── vercel.json
└── README.md

📜 License

Apache License 2.0 — see LICENSE.

🔗 Links

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • Local AI MCP by TMHSDigital

    Local AI MCPofficial

    TMHSDigital
    A
    license
    A
    quality
    A
    maintenance
    Unified MCP server for managing local model runtimes (Ollama, LM Studio, etc.), enabling provider-agnostic discovery, lifecycle management, hardware-fit checks, and delegated inference.
    16
    30 npm
    Creative Commons Attribution Non Commercial No Derivatives 4.0 International

  • PeakInfer MCP Server

    Kalmantic
    A
    license
    B
    quality
    D
    maintenance
    MCP server for achieving peak inference performance by detecting drift between code and runtime behavior, comparing against benchmarks, and providing optimization templates.
    8
    5 npm
    Apache 2.0