phoenix-mcp-eval
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@phoenix-mcp-eval list evaluation scores for my chatbot project"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
phoenix-mcp-eval
MCP server for Arize Phoenix — LLM tracing, evaluation, and dataset management via AI agents.
What is this?
phoenix-mcp-eval is an MCP (Model Context Protocol) server that exposes Arize Phoenix's LLM observability capabilities to AI agents. It enables AI-driven analysis of traces, evaluation of LLM outputs, and management of evaluation datasets — directly from an MCP-compatible agent.
Built for platform engineers and ML teams running LLM pipelines on AI Foundry, LangChain, or LlamaIndex who need automated quality assurance and tracing.
Related MCP server: mcp-llm-eval
Available Tools
Tool | Description |
| List all Phoenix tracing projects |
| Retrieve LLM traces for a project with filters |
| Get individual spans with input/output/latency data |
| List evaluation datasets in Phoenix |
| Fetch dataset examples for review or comparison |
| List evaluation runs and their scores |
| Get aggregated evaluation metrics (precision, recall, etc.) |
| Run structured queries over trace data |
Quick Start
Prerequisites
Python 3.11+
Arize Phoenix instance (self-hosted or cloud)
Phoenix API key or local server URL
Installation
git clone [https://github.com/akkireddy-challa/phoenix-mcp-eval.git](https://github.com/akkireddy-challa/phoenix-mcp-eval.git)
cd phoenix-mcp-eval
pip install -r requirements.txtConfiguration
export PHOENIX_HOST=http://localhost:6006
export PHOENIX_API_KEY=<your-api-key> # if using cloudRun
python server.pyMCP Client Config (Claude Desktop)
{
"mcpServers": {
"phoenix": {
"command": "python",
"args": ["/path/to/phoenix-mcp-eval/server.py"],
"env": {
"PHOENIX_HOST": "http://localhost:6006"
}
}
}
}Security Model
Connects to Phoenix via API key or local network only
All operations are read-only by default (trace/eval retrieval)
No model weights, prompts, or PII are transmitted outside Phoenix
API key stored in environment variables, never in code
Designed for internal network use within a Kubernetes cluster
Use Cases at Telia
This pattern is used to allow AI agents to:
Automatically review LLM trace quality after AI Foundry deployments
Surface failing evaluation metrics to on-call engineers without manual Phoenix access
Compare evaluation datasets across model versions
Trigger re-evaluation jobs based on trace anomaly detection
Roadmap
run_evaluation— trigger evaluation jobs programmaticallycreate_dataset— export traces to evaluation datasetsget_prompt_templates— retrieve versioned prompts from PhoenixIntegration with Azure AI Foundry deployment events
GitHub Actions workflow for CI validation
Related Projects
Repo | Purpose |
Kubernetes cluster diagnostics via MCP | |
Azure resource management via MCP | |
Grafana dashboards and alerts via MCP |
License
MIT License. See LICENSE for details.
Built by Akkireddy Challa — Platform Engineer at Telia, Stockholm.
Available Tools
8 toolsget_datasetA
Fetch dataset examples for review or eval comparison.
Inputs: dataset_id (required), limit.
get_evaluation_summaryB
Get aggregated evaluation metrics (mean score, pass rate, etc.).
Inputs: eval_name (required), project_name (required).
get_spansA
Get individual spans with input, output, and latency data.
Inputs: trace_id (required), project_name (required).
get_tracesB
Retrieve LLM traces for a project with optional filters.
Inputs: project_name (required), limit, status.
list_datasetsB
List evaluation datasets in Phoenix.
Inputs: no named parameters.
list_evaluationsA
List evaluation runs and their scores for a project.
Inputs: project_name (required).
list_projectsA
List all Phoenix tracing projects.
Inputs: no named parameters.
query_tracesC
Run a structured query over trace data by time range or metadata filter.
Inputs: project_name (required), status, end_time, start_time.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 8 tools
Most tools have clearly distinct purposes: list vs get vs query operations are separated, and each targets a different entity (datasets, evaluations, traces, spans, projects). The main potential confusion is between get_traces and query_traces, though descriptions clarify query_traces as a structured filtering alternative.
All tool names follow the verb_noun pattern consistently: get_ for single entities, list_ for collections, and query_ for the structured trace query. No style mixing or vague verbs; the naming is predictable and uniform.
Eight tools provide a focused and well-scoped surface for a Phoenix evaluation/tracing server. Each tool addresses a distinct need without redundancy or bloat, fitting squarely in the ideal range.
The server covers the core read-only interactions with datasets, evaluations, traces, spans, and projects, including both list and detail views. Notable gaps include no tool to fetch a specific evaluation run's detailed results or a specific project's configuration, but these are minor for a query-focused eval server.
Maintenance
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
MCP observability. Query live traffic, errors, duration, and alerts from your AI agent.
Related MCP Servers
AlicenseBqualityBmaintenancePhoenix MCP Server is an implementation of the Model Context Protocol for the Arize Phoenix platform. It provides a unified interface to Phoenix's capabilites. You can use Phoenix MCP Server for: Prompts Management: Create, list, update, and iterate on prompts Datasets: Explore datasets, and synte271,272 npm12,325Apache 2.0- AlicenseAqualityBmaintenanceA local MCP server that packages LLM evaluation gates as reusable CI/CD primitives, enabling AI agents to run datasets against models, score responses, and enforce quality thresholds.1079 PyPIMIT
- AlicenseAqualityDmaintenanceMCP server that gives AI agents access to your application's OpenTelemetry traces for querying, analysis, and debugging.511 npm2MIT
- AlicenseNot gradedqualityCmaintenanceMCP server that enables AI agents to run a deterministic orchestration loop with decomposition, subagent execution, and review feedback across multiple LLM backends.61MIT