phoenix-mcp-eval
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@phoenix-mcp-evallist evaluation scores for my chatbot project"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
phoenix-mcp-eval
MCP server for Arize Phoenix — LLM tracing, evaluation, and dataset management via AI agents.
What is this?
phoenix-mcp-eval is an MCP (Model Context Protocol) server that exposes Arize Phoenix's LLM observability capabilities to AI agents. It enables AI-driven analysis of traces, evaluation of LLM outputs, and management of evaluation datasets — directly from an MCP-compatible agent.
Built for platform engineers and ML teams running LLM pipelines on AI Foundry, LangChain, or LlamaIndex who need automated quality assurance and tracing.
Related MCP server: mcp-llm-eval
Available Tools
Tool | Description |
| List all Phoenix tracing projects |
| Retrieve LLM traces for a project with filters |
| Get individual spans with input/output/latency data |
| List evaluation datasets in Phoenix |
| Fetch dataset examples for review or comparison |
| List evaluation runs and their scores |
| Get aggregated evaluation metrics (precision, recall, etc.) |
| Run structured queries over trace data |
Quick Start
Prerequisites
Python 3.11+
Arize Phoenix instance (self-hosted or cloud)
Phoenix API key or local server URL
Installation
git clone [https://github.com/akkireddy-challa/phoenix-mcp-eval.git](https://github.com/akkireddy-challa/phoenix-mcp-eval.git)
cd phoenix-mcp-eval
pip install -r requirements.txtConfiguration
export PHOENIX_HOST=http://localhost:6006
export PHOENIX_API_KEY=<your-api-key> # if using cloudRun
python server.pyMCP Client Config (Claude Desktop)
{
"mcpServers": {
"phoenix": {
"command": "python",
"args": ["/path/to/phoenix-mcp-eval/server.py"],
"env": {
"PHOENIX_HOST": "http://localhost:6006"
}
}
}
}Security Model
Connects to Phoenix via API key or local network only
All operations are read-only by default (trace/eval retrieval)
No model weights, prompts, or PII are transmitted outside Phoenix
API key stored in environment variables, never in code
Designed for internal network use within a Kubernetes cluster
Use Cases at Telia
This pattern is used to allow AI agents to:
Automatically review LLM trace quality after AI Foundry deployments
Surface failing evaluation metrics to on-call engineers without manual Phoenix access
Compare evaluation datasets across model versions
Trigger re-evaluation jobs based on trace anomaly detection
Roadmap
run_evaluation— trigger evaluation jobs programmaticallycreate_dataset— export traces to evaluation datasetsget_prompt_templates— retrieve versioned prompts from PhoenixIntegration with Azure AI Foundry deployment events
GitHub Actions workflow for CI validation
Related Projects
Repo | Purpose |
Kubernetes cluster diagnostics via MCP | |
Azure resource management via MCP | |
Grafana dashboards and alerts via MCP |
License
MIT License. See LICENSE for details.
Built by Akkireddy Challa — Platform Engineer at Telia, Stockholm.
Available Tools
8 toolsget_datasetA
Fetch dataset examples for review or eval comparison.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max examples to return (default 20) | |
| dataset_id | Yes | Dataset ID |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the burden of disclosing behavior. 'Fetch' clearly implies a read-only operation with no side effects, which is useful. However, the description does not disclose limit/pagination behavior, response shape, or edge cases. These are not heavily problematic for a simple fetch tool, but they remain undisclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with the action 'Fetch' and the resource 'dataset examples', followed by a short purpose clause. There is no filler, redundancy, or repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with no output schema, the description plus the input schema give an agent enough to understand the operation and purpose. It does not describe the exact shape of the returned examples, but the term 'examples' and the limit parameter make the return type reasonably inferable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema already documents dataset_id and limit, including the default for limit. The description adds no parameter-level semantics beyond the stated purpose. With full schema coverage, baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear action and resource: 'Fetch dataset examples'. The purpose clause 'for review or eval comparison' adds context and helps distinguish it from list_datasets, which presumably lists dataset metadata rather than examples. It does not explicitly differentiate itself from siblings like get_evaluation_summary, so it is clear but not fully differentiated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'for review or eval comparison' implies when an agent might want this tool, but it does not explicitly state when to use it over siblings such as list_datasets or get_evaluation_summary. The usage guidance is inferred from the purpose rather than stated as a decision rule.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_evaluation_summaryB
Get aggregated evaluation metrics (mean score, pass rate, etc.).
| Name | Required | Description | Default |
|---|---|---|---|
| eval_name | Yes | Evaluation name (e.g., 'hallucination', 'qa_correctness') | |
| project_name | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden of behavioral disclosure. It does add some behavioral context by revealing the kinds of aggregated metrics returned ('mean score, pass rate, etc.'), but it does not mention output structure, aggregation behavior, error conditions, or any other operational details. For a simple read-only lookup this is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence with no filler. The core action and key examples are front-loaded, and every word contributes to understanding the tool's purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool with no output schema, the description provides a reasonable baseline: it names the action, the resource, and example metrics. However, it leaves gaps around what the return payload looks like, how the two parameters interact, and when this tool should be selected over sibling tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 50%: eval_name has a helpful example but project_name has no description. The tool description does not compensate by clarifying either parameter's role or expected format, so the agent must rely on parameter names and general context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies a specific verb ('Get') and resource ('evaluation summary'), and adds meaningful detail by naming the aggregated metrics ('mean score, pass rate, etc.'). However, it does not explicitly contrast itself with sibling tools like list_evaluations, so differentiation is mostly implicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied by the phrase 'Get aggregated evaluation metrics' — an agent can infer this is for summary statistics rather than raw data. There is no explicit guidance about when to use this tool versus alternatives such as list_evaluations or get_spans, so the usage context is present but underdeveloped.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_spansA
Get individual spans with input, output, and latency data.
| Name | Required | Description | Default |
|---|---|---|---|
| trace_id | Yes | Trace ID to fetch spans for | |
| project_name | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full burden of behavioral disclosure. It reveals the return contents (input, output, latency) and 'Get' implies a safe read operation, but it does not mention ordering, pagination, errors, or the relationship between project_name and trace_id.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One compact sentence that front-loads the verb and resource and avoids any redundant or filler content. Every word contributes to stating the tool's purpose and return data.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The core purpose is clear for a simple two-parameter getter, but the description leaves project_name's role undefined and gives no guidance on selecting this tool over get_traces/query_traces. It is minimally viable but has notable contextual gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers trace_id with a description, but project_name has no schema description and the tool description does not clarify its role. With only 50% schema coverage, the description needed to compensate but instead adds no parameter meaning beyond the tool name.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get') and resource ('individual spans') and explicitly lists the returned data (input, output, latency). This clearly distinguishes get_spans from sibling tools like get_traces, list_datasets, and list_evaluations by naming a distinct entity type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit when-to-use or alternative guidance is given. The phrase 'individual spans' implies span-level retrieval, but the description never contrasts with get_traces or query_traces, so an agent must infer when this tool is the right choice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_tracesB
Retrieve LLM traces for a project with optional filters.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max traces to return (default 50) | |
| status | No | Filter by status: ok, error | |
| project_name | Yes | Phoenix project name |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral transparency burden. It only says 'Retrieve', which implies read-only, but does not disclose output shape, ordering, pagination, or how filters affect results.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single clear sentence with no waste. It front-loads the primary action and resource before mentioning filters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The input side is adequately covered by the schema, but no output schema exists and the description does not clarify return values. It also lacks sibling differentiation, so an agent may be unsure whether to choose get_traces or query_traces.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the parameters are already documented. The description adds minimal semantic value beyond noting the existence of filters, which the schema already covers.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb and resource: 'Retrieve LLM traces for a project' with optional filters. It distinguishes the tool by resource but does not explicitly differentiate it from siblings like query_traces or get_spans.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use get_traces versus alternatives such as query_traces or get_spans. The phrase 'optional filters' is too generic to serve as usage direction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_datasetsB
List evaluation datasets in Phoenix.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral disclosure burden. It states the operation but gives no information about return format, pagination, scope limitations, or any other runtime behavior beyond the literal listing action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no filler. Every word contributes meaning, and it is appropriately sized for such a simple tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter list tool, the basic operation is stated clearly enough to attempt a call. However, with no annotations, no output schema, and no guidance on how this relates to similar sibling tools, the description leaves some contextual ambiguity about what results and alternatives an agent should expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the schema coverage is 100%, so the baseline of 4 applies. The description does not need to explain parameter meanings because none exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('List') and resource ('evaluation datasets in Phoenix'), making the tool's core function apparent. It is distinguishable from sibling tools like list_evaluations and list_projects by the resource type, though it does not explicitly contrast with get_dataset.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives such as get_dataset, list_evaluations, or query_traces. The description only states what it does, leaving usage context entirely to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_evaluationsA
List evaluation runs and their scores for a project.
| Name | Required | Description | Default |
|---|---|---|---|
| project_name | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries the behavioral burden. 'List' implies a read-only operation, but the description does not explicitly confirm no side effects, ordering, pagination, or return details. It is minimally transparent but leaves key behavioral traits unstated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The entire description is a single front-loaded sentence with no redundant words. It conveys the action, object, and scope efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter list tool with no output schema or annotations, the description covers the core purpose but omits return structure, filtering behavior, and relationship to named siblings. It is adequate but has gaps an agent would need to resolve from context or by invoking the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description's 'for a project' adds semantic context that project_name is the project being queried. However, this mostly restates the parameter's name and does not explain accepted formats, required identifier types, or defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List') with a clear resource ('evaluation runs and their scores') and scope ('for a project'). This distinguishes it from siblings like get_evaluation_summary, which implies a single summary object, and list_datasets, which targets a different resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'for a project' gives contextual usage, but there is no explicit guidance on when to choose this tool over get_evaluation_summary or list_projects. Usage is implied rather than stated with alternatives or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_projectsA
List all Phoenix tracing projects.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description alone must convey safety and behavior. The verb 'List' implies a read-only operation and 'all' clarifies scope, but the description does not address pagination, return shape, or side effects; this is a modest gap for a trivial tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single short sentence that is entirely relevant and front-loaded; no filler. It is appropriately concise for a tool with no parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple and the description provides the essential 'what' and the unfiltered scope. However, with no output schema and no annotations, the description remains a bare statement and leaves return-value details to inference, so it is minimal but viable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero properties, so there are no parameter semantics to document. Per baseline for parameterless tools, the description does not need to compensate, although it adds no parameter detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('List') and a concrete resource ('Phoenix tracing projects'), making the tool's function immediately clear. It is distinct from sibling list tools because it targets projects, not datasets or evaluations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no conditions, alternatives, or when-not-to-use guidance. While 'list all' implies a use case, it never explains when to choose list_projects over sibling tools like list_datasets or list_evaluations.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
query_tracesC
Run a structured query over trace data by time range or metadata filter.
| Name | Required | Description | Default |
|---|---|---|---|
| status | No | Filter by status | |
| end_time | No | ISO8601 end time | |
| start_time | No | ISO8601 start time | |
| project_name | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It states that the tool runs a query, but does not mention whether it is read-only, whether results are paginated, whether there are limits, or what the query behavior is beyond filtering.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single focused sentence with no wasted words. The core action and key filtering dimensions are front-loaded, making it easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, no annotations, and an overlapping sibling set, the description leaves meaningful gaps: it does not explain what the tool returns, how it differs from get_traces, or what 'metadata filter' maps to. An agent would likely need to inspect sibling tools to make a reliable selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 75%, so the schema already documents start_time, end_time, and status. The description adds a useful mapping by categorizing the filters as 'time range or metadata filter,' but project_name remains undocumented and the description does not clarify its role beyond what the schema implies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Run a structured query') and resource ('trace data'), and specifies filtering by time range or metadata filter. However, it does not differentiate from the sibling tool 'get_traces', which likely serves a very similar purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for querying traces with time or metadata filters, but it provides no explicit guidance on when to choose this tool over alternatives like get_traces or get_spans. No exclusions, prerequisites, or use-case boundaries are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
8 tool updates
v0.1.0- First observed
get_dataset - First observed
get_evaluation_summary - First observed
get_spans - First observed
get_traces - First observed
list_datasets - First observed
list_evaluations - First observed
list_projects - First observed
query_traces
TDQS
Scored across 8 tools
Most tools have clearly distinct purposes: list vs get vs query operations are separated, and each targets a different entity (datasets, evaluations, traces, spans, projects). The main potential confusion is between get_traces and query_traces, though descriptions clarify query_traces as a structured filtering alternative.
All tool names follow the verb_noun pattern consistently: get_ for single entities, list_ for collections, and query_ for the structured trace query. No style mixing or vague verbs; the naming is predictable and uniform.
Eight tools provide a focused and well-scoped surface for a Phoenix evaluation/tracing server. Each tool addresses a distinct need without redundancy or bloat, fitting squarely in the ideal range.
The server covers the core read-only interactions with datasets, evaluations, traces, spans, and projects, including both list and detail views. Notable gaps include no tool to fetch a specific evaluation run's detailed results or a specific project's configuration, but these are minor for a query-focused eval server.
Maintenance
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
- SpanlyOAuthcom.spanly
MCP observability. Query live traffic, errors, duration, and alerts from your AI agent.
Related MCP Servers
AlicenseBqualityAmaintenancePhoenix MCP Server is an implementation of the Model Context Protocol for the Arize Phoenix platform. It provides a unified interface to Phoenix's capabilites. You can use Phoenix MCP Server for: Prompts Management: Create, list, update, and iterate on prompts Datasets: Explore datasets, and synte271,46311,413Apache 2.0- AlicenseAqualityCmaintenanceA local MCP server that packages LLM evaluation gates as reusable CI/CD primitives, enabling AI agents to run datasets against models, score responses, and enforce quality thresholds.10MIT
- AlicenseAqualityDmaintenanceMCP server that gives AI agents access to your application's OpenTelemetry traces for querying, analysis, and debugging.5122MIT
- AlicenseNot gradedqualityBmaintenanceMCP server that enables AI agents to run a deterministic orchestration loop with decomposition, subagent execution, and review feedback across multiple LLM backends.59MIT