Confident AI MCP Server

The Confident AI MCP Server connects AI-powered tools to Confident AI, a platform to evaluate, observe, and iterate on AI quality. It gives you full control over your resources directly from your editor:

• Cloud evaluations and metric collections

• Evaluation datasets

• Prompt versioning and management

• Production tracing and observability

• Human annotations and feedback

For users of DeepEval, Confident AI is also the native backend and persistence layer for your evaluation results. This MCP server gives you the ability to iterate on your AI application by bringing all of that data directly into tools like Cursor and Claude Code.

Use Cases

Built for developers who want to iterate faster on their AI applications from inside editors like Cursor, Claude Code, and Windsurf — from simple queries to fully automated improvement workflows:

• 10x your iteration speed. Run an eval, check if a set of prompts are better — in one continuous workflow instead of scattered across tools. What used to take an hour of tab-switching now takes one conversation.

• Go from eval results to action plan automatically. Your AI assistant can pull eval results, read what failed and why, and draft a plan for what to improve next — no manual analysis needed.

• Use production traces for iteration. Pull the trace, see what went in and what came out, read what users said — and fix it before anyone else notices.

• Let human feedback drive your next iteration. Pull annotation data your team left on production traces and have your AI assistant use it to decide what to fix and how.

Every time you leave your editor to check eval results, tweak a prompt in a dashboard, or look up what your team annotated — you lose context and iteration speed.

How is this different from the platform?

Confident AI has a full web UI where you can do all of this with a mouse. This MCP server is the same platform, accessed from your editor instead. Think: AWS web console vs. AWS CLI — same resources, different interface.

The server speaks the Model Context Protocol (MCP), so any compatible client connects out of the box. The web UI isn't going anywhere. This is just another way in.

Jump Ahead

• Prerequisites — What you need before you start

• Quickstart — Get up and running in under a minute

  • Cursor · Claude Code (or Desktop) · Windsurf · Run Locally

• Configuration — Environment variables for regions, on-prem, and advanced setup

• Available Tools — Full reference of all 27 tools

• License

Related MCP server: delimit

Prerequisites

1. A Confident AI API key.

2. An MCP-compatible client — Cursor, Claude, Windsurf, or any other client that supports the Model Context Protocol.

Quickstart

Confident AI hosts the MCP server for you. Pick your region:

🖥️ Cursor

Add the following to your .cursor/mcp.json file:

{
  "mcpServers": {
    "confident-ai": {
      "url": "https://mcp.confident-ai.com/mcp",
      "headers": {
        "Authorization": "Bearer <YOUR_CONFIDENT_API_KEY>"
      }
    }
  }
}

🤖 Claude Code (or Desktop)

Claude Code — run the following command in your terminal:

claude mcp add --transport http confident-ai https://mcp.confident-ai.com/mcp --header "Authorization: Bearer <YOUR_CONFIDENT_API_KEY>"

Claude Desktop — add the following to your claude_desktop_config.json file:

{
  "mcpServers": {
    "confident-ai": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.confident-ai.com/mcp",
        "--header",
        "Authorization: Bearer <YOUR_CONFIDENT_API_KEY>"
      ]
    }
  }
}

🏄 Windsurf

Add the following to your Windsurf MCP configuration:

{
  "mcpServers": {
    "confident-ai": {
      "serverUrl": "https://mcp.confident-ai.com/mcp",
      "headers": {
        "Authorization": "Bearer <YOUR_CONFIDENT_API_KEY>"
      }
    }
  }
}

🛠️ Running the Server Locally

If you're self-hosting or contributing to this project, you can run the server from source.

Prerequisites: Python >= 3.12, Poetry

poetry install
poetry run python server.py

The server will start on http://0.0.0.0:8081. It uses the Streamable HTTP transport — a single /mcp endpoint that handles both GET and POST.

The /mcp endpoint requires a Bearer token in the Authorization header (your Confident AI API key).

When running locally, point your MCP client to http://localhost:8081/mcp instead of the hosted URLs above.

To run in stdio mode instead (for MCP clients that communicate over stdin/stdout), uncomment the relevant block at the bottom of server.py:

if __name__ == "__main__":
    mcp.run(transport="stdio")

Configuration

The server is configured through environment variables. You can set these in a .env file in the project root.

Available Tools

Manage prompt templates with full version control — pull, push, version, and interpolate.

Pull evaluation datasets for use in local test runs or agent workflows, with full version control to pin runs to immutable snapshots of goldens.

Trigger cloud evaluations and simulate multi-turn conversations.

Browse, inspect, and evaluate production observability data at every level of your LLM pipeline.

Create and manage human feedback on traces, spans, and threads.

Inspect past evaluation runs and their results.

Discover available metric collections before triggering evaluations.

Public Endpoint

License

This project is licensed under the terms of the MIT License.