The Atla MCP Server enables standardized evaluation of LLM responses using Atla's evaluation models, providing scores and textual critiques based on specific criteria.
- Evaluate LLM responses: Score and critique responses against single or multiple evaluation criteria simultaneously
- Choose evaluation models: Use flagship (
atla-selene
) or compact (atla-selene-mini
) models - Customize evaluations: Optionally include original context and expected outputs
- Integration options: Connect via OpenAI Agents SDK, Claude Desktop, or Cursor
Provides compatibility with the OpenAI Agents SDK, allowing users to connect to the Atla MCP server for LLM evaluation services.
Atla MCP Server
An MCP server implementation providing a standardized interface for LLMs to interact with the Atla API for state-of-the-art LLMJ evaluation.
Learn more about Atla here. Learn more about the Model Context Protocol here.
Available Tools
evaluate_llm_response
: Evaluate an LLM's response to a prompt using a given evaluation criteria. This function uses an Atla evaluation model under the hood to return a dictionary containing a score for the model's response and a textual critique containing feedback on the model's response.evaluate_llm_response_on_multiple_criteria
: Evaluate an LLM's response to a prompt across multiple evaluation criteria. This function uses an Atla evaluation model under the hood to return a list of dictionaries, each containing an evaluation score and critique for a given criteria.
Usage
To use the MCP server, you will need an Atla API key. You can find your existing API key here or create a new one here.
Installation
We recommend using
uv
to manage the Python environment. See here for installation instructions.
Manually running the server
Once you have uv
installed and have your Atla API key, you can manually run the MCP server using uvx
(which is provided by uv
):
Connecting to the server
Having issues or need help connecting to another client? Feel free to open an issue or contact us!
OpenAI Agents SDK
For more details on using the OpenAI Agents SDK with MCP servers, refer to the official documentation.
- Install the OpenAI Agents SDK:
- Use the OpenAI Agents SDK to connect to the server:
Claude Desktop
For more details on configuring MCP servers in Claude Desktop, refer to the official MCP quickstart guide.
- Add the following to your
claude_desktop_config.json
file:
- Restart Claude Desktop to apply the changes.
You should now see options from atla-mcp-server
in the list of available MCP tools.
Cursor
For more details on configuring MCP servers in Cursor, refer to the official documentation.
- Add the following to your
.cursor/mcp.json
file:
You should now see atla-mcp-server
in the list of available MCP servers.
Contributing
Contributions are welcome! Please see the CONTRIBUTING.md file for details.
License
This project is licensed under the MIT License. See the LICENSE file for details.
remote-capable server
The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.
The Atla MCP Server provides a standardized interface for LLMs to interact with the Atla API for state-of-the-art LLMJ evaluation.
Related MCP Servers
- AsecurityAlicenseAqualityMCP Ollama server integrates Ollama models with MCP clients, allowing users to list models, get detailed information, and interact with them through questions.Last updated -325PythonMIT License
- AsecurityAlicenseAqualityAn MCP server implementation that enables interaction with the Letta API for managing agents, memory blocks, and tools in the Letta system.Last updated -925JavaScriptMIT License
- -securityFlicense-qualityA server that enables seamless integration between local Ollama LLM instances and MCP-compatible applications, providing advanced task decomposition, evaluation, and workflow management capabilities.Last updated -6Python
- -securityFlicense-qualityAn MCP server that fetches real-time documentation for popular libraries like Langchain, Llama-Index, MCP, and OpenAI, allowing LLMs to access updated library information beyond their knowledge cut-off dates.Last updated -2Python