Patronus MCP Serverofficial
A
licenseNot graded
qualityD
maintenanceEnables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.
16
Apache 2.0
NM Hard Tools
Officialby neuralmagic
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@NM Hard Toolsrun lm-eval for llama-3-8b on the mmlu benchmark"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
NM Hard Tools
Deployable, stateless MCP services for rigorous inference evaluation and benchmarking. This repository owns the service implementations, container images, and Helm charts.
Included services:
a single-tool llm-manifesto model deployment service;
a profiled
lm-evaluation-harnesscontroller and worker;a bounded MCP adapter for the vLLM forward-pass benchmark service;
deployment charts for lm-eval, forward-pass, Nyann, and AgentX.
Development
uv run --extra test pytest
uv run --extra test ruff format --check src tests
uv run --extra test ruff check src testsRepository governance is stored in gov/ and validated with
govctl:
cargo install govctl --version 0.19.1 --locked
govctl status
govctl checkThe lm-eval image must be built with an explicit Python base image:
podman build -f Dockerfile.lm-eval-service \
--build-arg BASE_IMAGE=python:3.12-slim -t inference-hard-lm-eval:dev .The model deployment service is described in
docs/model-deployment-service.md.
Related MCP server: ArmBench MCP Server
License
Licensed under the Apache License, Version 2.0. See LICENSE.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues
Related MCP Connectors
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
Benchmark-first release surface with a read-only MCP endpoint and operator CLI.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables benchmarking and inference of LLMs on Arm64 cloud instances with KleidiAI optimizations, providing an MCP-compatible API for serving results.1MIT
Local AI MCPofficial
AlicenseAqualityAmaintenanceUnified MCP server for managing local model runtimes (Ollama, LM Studio, etc.), enabling provider-agnostic discovery, lifecycle management, hardware-fit checks, and delegated inference.1618 npmCreative Commons Attribution Non Commercial No Derivatives 4.0 International- AlicenseNot gradedqualityCmaintenanceEnables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.8 npm8Apache 2.0