Skip to main content
Glama

phoenix-mcp-eval

MCP server for Arize Phoenix — LLM tracing, evaluation, and dataset management via AI agents.

Glama Quality Score CI License: MIT Python Arize Phoenix MCP


What is this?

phoenix-mcp-eval is an MCP (Model Context Protocol) server that exposes Arize Phoenix's LLM observability capabilities to AI agents. It enables AI-driven analysis of traces, evaluation of LLM outputs, and management of evaluation datasets — directly from an MCP-compatible agent.

Built for platform engineers and ML teams running LLM pipelines on AI Foundry, LangChain, or LlamaIndex who need automated quality assurance and tracing.


Related MCP server: mcp-llm-eval

Available Tools

Tool

Description

list_projects

List all Phoenix tracing projects

get_traces

Retrieve LLM traces for a project with filters

get_spans

Get individual spans with input/output/latency data

list_datasets

List evaluation datasets in Phoenix

get_dataset

Fetch dataset examples for review or comparison

list_evaluations

List evaluation runs and their scores

get_evaluation_summary

Get aggregated evaluation metrics (precision, recall, etc.)

query_traces

Run structured queries over trace data


Quick Start

Prerequisites

  • Python 3.11+

  • Arize Phoenix instance (self-hosted or cloud)

  • Phoenix API key or local server URL

Installation

git clone [https://github.com/akkireddy-challa/phoenix-mcp-eval.git](https://github.com/akkireddy-challa/phoenix-mcp-eval.git)
cd phoenix-mcp-eval
pip install -r requirements.txt

Configuration

export PHOENIX_HOST=http://localhost:6006
export PHOENIX_API_KEY=<your-api-key>  # if using cloud

Run

python server.py

MCP Client Config (Claude Desktop)

{
  "mcpServers": {
    "phoenix": {
      "command": "python",
      "args": ["/path/to/phoenix-mcp-eval/server.py"],
      "env": {
        "PHOENIX_HOST": "http://localhost:6006"
      }
    }
  }
}

Security Model

  • Connects to Phoenix via API key or local network only

  • All operations are read-only by default (trace/eval retrieval)

  • No model weights, prompts, or PII are transmitted outside Phoenix

  • API key stored in environment variables, never in code

  • Designed for internal network use within a Kubernetes cluster


Use Cases at Telia

This pattern is used to allow AI agents to:

  • Automatically review LLM trace quality after AI Foundry deployments

  • Surface failing evaluation metrics to on-call engineers without manual Phoenix access

  • Compare evaluation datasets across model versions

  • Trigger re-evaluation jobs based on trace anomaly detection


Roadmap

  • run_evaluation — trigger evaluation jobs programmatically

  • create_dataset — export traces to evaluation datasets

  • get_prompt_templates — retrieve versioned prompts from Phoenix

  • Integration with Azure AI Foundry deployment events

  • GitHub Actions workflow for CI validation


Repo

Purpose

k8s-mcp-server

Kubernetes cluster diagnostics via MCP

azure-mcp-platform

Azure resource management via MCP

grafana-mcp-observability

Grafana dashboards and alerts via MCP


License

MIT License. See LICENSE for details.


Built by Akkireddy Challa — Platform Engineer at Telia, Stockholm.

Available Tools

8 tools
get_datasetA

Fetch dataset examples for review or eval comparison.

Inputs: dataset_id (required), limit.

get_evaluation_summaryB

Get aggregated evaluation metrics (mean score, pass rate, etc.).

Inputs: eval_name (required), project_name (required).

get_spansA

Get individual spans with input, output, and latency data.

Inputs: trace_id (required), project_name (required).

get_tracesB

Retrieve LLM traces for a project with optional filters.

Inputs: project_name (required), limit, status.

list_datasetsB

List evaluation datasets in Phoenix.

Inputs: no named parameters.

list_evaluationsA

List evaluation runs and their scores for a project.

Inputs: project_name (required).

list_projectsA

List all Phoenix tracing projects.

Inputs: no named parameters.

query_tracesC

Run a structured query over trace data by time range or metadata filter.

Inputs: project_name (required), status, end_time, start_time.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.6/5.0

    Scored across 8 tools

    Disambiguation4/5

    Most tools have clearly distinct purposes: list vs get vs query operations are separated, and each targets a different entity (datasets, evaluations, traces, spans, projects). The main potential confusion is between get_traces and query_traces, though descriptions clarify query_traces as a structured filtering alternative.

    Naming Consistency5/5

    All tool names follow the verb_noun pattern consistently: get_ for single entities, list_ for collections, and query_ for the structured trace query. No style mixing or vague verbs; the naming is predictable and uniform.

    Tool Count5/5

    Eight tools provide a focused and well-scoped surface for a Phoenix evaluation/tracing server. Each tool addresses a distinct need without redundancy or bloat, fitting squarely in the ideal range.

    Completeness4/5

    The server covers the core read-only interactions with datasets, evaluations, traces, spans, and projects, including both list and detail views. Notable gaps include no tool to fetch a specific evaluation run's detailed results or a specific project's configuration, but these are minor for a query-focused eval server.

    Maintenance

    ActivityStale
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      B
      maintenance
      Phoenix MCP Server is an implementation of the Model Context Protocol for the Arize Phoenix platform. It provides a unified interface to Phoenix's capabilites. You can use Phoenix MCP Server for: Prompts Management: Create, list, update, and iterate on prompts Datasets: Explore datasets, and synte
      27
      1,272 npm
      12,325
      Apache 2.0
    • A
      license
      A
      quality
      D
      maintenance
      MCP server that gives AI agents access to your application's OpenTelemetry traces for querying, analysis, and debugging.
      5
      11 npm
      2
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server that enables AI agents to run a deterministic orchestration loop with decomposition, subagent execution, and review feedback across multiple LLM backends.
      61
      MIT