Skip to main content
Glama
Coval MCP Server

Coval MCP Server

Official
by coval-ai
README.md
# Coval MCP Server

[![npm version](https://img.shields.io/npm/v/@coval/mcp-server.svg)](https://www.npmjs.com/package/@coval/mcp-server)

The official [Model Context Protocol](https://modelcontextprotocol.io/) server for [Coval](https://coval.dev) - the AI evaluation platform.

This MCP server allows AI assistants like Claude, ChatGPT, Codex, and Cursor to interact with
Coval's evaluation APIs, enabling you to:
- Launch and monitor evaluation runs
- Manage AI agents and test sets
- Retrieve evaluation metrics and results
- Inspect saved reports and recurring evaluation schedules

## Hosted connector

The hosted connector is the recommended way to use Coval from Claude, ChatGPT, and other remote
MCP clients. It uses OAuth, so users sign in to Coval and select the organization the connector
can access. No Coval API key is copied into the client.

Standard MCP endpoint: `https://mcp.coval.dev/mcp`

Claude hosted endpoint: `https://mcp.coval.dev/claude/mcp`

### Claude

1. Open **Customize > Connectors**. Team and Enterprise owners add the connector first from
   **Organization settings > Connectors**.
2. Add a custom web connector using `https://mcp.coval.dev/claude/mcp`.
3. Select **Connect**, sign in to Coval, and choose the Coval organization to authorize.
4. Enable the connector in a conversation and ask Claude to use Coval or Sofia.

See [Claude's remote MCP connector guide](https://support.claude.com/en/articles/11175166-get-started-with-custom-connectors-using-remote-mcp)
for current plan and administration requirements.

### ChatGPT

1. Enable developer mode for your ChatGPT workspace account.
2. From **Settings > Apps**, create an app using `https://mcp.coval.dev/mcp` and OAuth.
3. Select **Scan Tools**, sign in to Coval, and choose the Coval organization to authorize.
4. Create the app, then enable it in a conversation.

See [ChatGPT's MCP app guide](https://help.openai.com/en/articles/12584461-developer-mode-and-full-mcp-connectors-in-chatgpt-beta)
for current plan and workspace requirements.

### Codex

Use Codex `0.144.2` or newer. Current Codex Desktop releases include a compatible client. For the
CLI, update before connecting:

```bash
npm install -g @openai/codex@latest
codex mcp add coval --url https://mcp.coval.dev/mcp
```

When the browser opens, sign in to Coval, select the organization Codex should access, and select
**Allow**. Codex Desktop, the Codex CLI, and the IDE extension share MCP configuration on the same
machine, so the connection is available across those clients after they restart.

In Codex Desktop, you can also add Coval from **Settings > MCP servers** as a **Streamable HTTP**
server using `https://mcp.coval.dev/mcp`, then select **Authenticate**.

If OAuth returns `invalid_scope`, first confirm `codex --version` is `0.144.2` or newer. Older
clients registered a dynamic OAuth client without all the scopes they requested during consent.
After updating, clear the stale registration and reconnect:

```bash
codex mcp logout coval
codex mcp remove coval
codex mcp add coval --url https://mcp.coval.dev/mcp
```

No API key, manual scope list, or `oauth_resource` override is required. See
[Codex's MCP documentation](https://learn.chatgpt.com/docs/extend/mcp.md) for current client setup
details.

## Local installation

```bash
npx @coval/mcp-server
```

Use the local stdio server for service accounts, automation, and clients that do not support
remote OAuth.

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "coval": {
      "command": "npx",
      "args": ["-y", "@coval/mcp-server"],
      "env": {
        "COVAL_API_KEY": "your_api_key_here"
      }
    }
  }
}
```

### Cursor

Add to `.cursor/mcp.json` in your project:

```json
{
  "mcpServers": {
    "coval": {
      "command": "npx",
      "args": ["-y", "@coval/mcp-server"],
      "env": {
        "COVAL_API_KEY": "your_api_key_here"
      }
    }
  }
}
```

Get your API key from [app.coval.dev/settings](https://app.coval.dev/settings)

## Available Tools

| Tool | Description |
|------|-------------|
| `list_agents` | List all agents in your workspace |
| `get_agent` | Get details of a specific agent |
| `create_agent` | Create an agent configuration |
| `update_agent` | Update an agent configuration |
| `list_runs` | List evaluation runs |
| `get_run` | Get details of a specific run |
| `create_run` | Start a new evaluation run |
| `list_test_sets` | List available test sets |
| `get_test_set` | Get test set details |
| `create_test_set` | Create a test set |
| `list_test_cases` | List test cases in a test set |
| `get_test_case` | Get test case details |
| `create_test_case` | Add a test case to a test set |
| `update_test_case` | Update a test case |
| `list_metrics` | List evaluation metrics |
| `get_metric` | Get metric details |
| `create_python_metric_draft` | Create an immutable Python metric draft |
| `get_python_metric_draft` | Get a Python metric draft and fingerprint |
| `create_python_metric_preview` | Start an asynchronous Python metric preview |
| `get_python_metric_preview` | Poll a Python metric preview for its result |
| `cancel_python_metric_preview` | Cancel an in-progress Python metric preview |
| `publish_python_metric_draft` | Publish a draft with successful preview IDs |
| `list_personas` | List available personas |
| `get_persona` | Get persona details |
| `list_reports` | List saved evaluation reports |
| `get_report` | Get report details and a bounded page of result rows |
| `create_report` | Create an organization-private saved report |
| `list_run_templates` | List reusable evaluation configurations |
| `list_scheduled_runs` | List recurring evaluation schedules |
| `get_scheduled_run` | Get a schedule and paginated recent-run history |
| `create_scheduled_run` | Create a schedule, disabled by default |
| `update_scheduled_run` | Update selected schedule fields |
| `consult_sofia` | Delegate a read-only Coval evaluation question to Sofia |

## Example Usage

Once connected, you can ask Claude things like:

> "Show me my recent evaluation runs"
> "List all my agents"
> "Run an evaluation of my customer-support-agent against the billing-inquiries test set"
> "What are the metrics for run abc123?"
> "Create an immutable Python metric draft, preview it on a conversation, and publish it after the preview succeeds."
> "Show the first 20 rows of my regression report for the task-success metric"
> "Create a private report comparing these completed runs by agent"
> "Show my enabled evaluation schedules and the latest runs from the nightly regression"
> "Create a disabled weekday schedule from my regression run template"
> "Use `consult_sofia` to analyze my latest failed run and recommend the most useful next test."

The same tools, including `consult_sofia`, are available through every supported endpoint and
transport:

- Standard remote Streamable HTTP: `https://mcp.coval.dev/mcp` using OAuth. This is the recommended
  connection for ChatGPT, Codex, and other hosted MCP clients.
- Claude remote Streamable HTTP: `https://mcp.coval.dev/claude/mcp` using the same OAuth flow and
  tool surface, with Claude-specific risk annotations.
- Local stdio: `npx @coval/mcp-server` with `COVAL_API_KEY`, for service accounts and local
  development.

The hosted connector can access only the Coval organization selected during OAuth consent. Remove
the connector from the client or revoke its Coval access when it is no longer needed.

## Development

```bash
# Install dependencies
npm install

# Build
npm run build

# Test locally with MCP Inspector
npm run inspector

# Run tests
npm test

# Check the public health and OAuth discovery endpoints
npm run check:remote
```

The remote check accepts OAuth discovery with either DCR or CIMD with public token authentication.
It does not verify that a particular client is admitted or that an authenticated tool call succeeds.

## Complete transcript exports

For a complete run transcript request, `consult_sofia` can return a native MCP
`resource_link` after collecting every call. Read that URI with `resources/read`.
Clients without resource-link support can use `read_transcript_export`: concatenate
its `text` fragments exactly, carrying `next_offset` and `version` as
`expected_version` until `complete=true`, then JSON-decode the document. Offsets
count Unicode code points; preserve whitespace while concatenating. Each fragment
contains complete code points and can be appended as UTF-8 bytes. The document includes a final completion marker, per-call unavailable
reasons, source order, original transcript fields, timing and tool ownership.

Exports use the currently authenticated API client's Default workspace. Every
read checks current authorization; the URI contains identifiers and fingerprints,
not credentials or a public download token. The export is reconstructed from
current API reads, not stored as a historical snapshot. Source changes are reported
explicitly, and continuation refuses to mix versions. A new authenticated server
can read the same URI if it still has access to the run.

Collection supports up to 100 calls, 2 MiB of serialized content and a 30-second
deadline. Exceeding a limit fails explicitly instead of returning a sample. A
missing transcript is listed with a reason. Authentication, incomplete inventory
and mismatched run membership fail the export. Readiness comes from the collector;
Sofia's export request or prose alone does not prove delivery. Unsupported older
Sofia servers continue returning their existing consultation response.

## API Coverage Audit

`scripts/audit_api_coverage.py` compares the live public OpenAPI catalog
(`https://api.coval.dev/v1/openapi`) with what the MCP server actually calls. It
has two layers:

- **Operations:** every published operation is either reached by a registered
  tool (traced from `src/tools/` through `src/client.ts`, including private
  client helpers) or recorded in `api-coverage.toml` with a reason.
- **Request fields:** for every covered `POST`/`PATCH`/`PUT` operation, every
  published `application/json` request-body property is either declared on the
  body type the client sends or recorded with a reason. A field the client
  sends but the API does not publish is also reported, because the API may
  ignore it.

The MCP server exposes a curated tool set, so most operations are reviewed
gaps. The audit's job is to surface what the API adds or changes so each change
gets a decision.

```bash
python3 -m pip install --requirement scripts/requirements-audit.txt
python3 -m unittest discover --start-directory scripts --pattern 'test_*.py'
python3 scripts/audit_api_coverage.py --write-markdown api-coverage-report.md
```

The audit fails for new or stale gaps, a stale `[snapshot]` in the manifest, or
tool calls to client methods that do not exist. Resolve a failure by exposing
the operation or field through a tool, or by recording it in
`api-coverage.toml` with a reason, then regenerate the report.

Every Monday at 10:00 UTC, `.github/workflows/api-parity-audit.yml` refreshes
the report and opens or updates one rolling pull request from
`chore/weekly-api-parity`. This is the same mechanism and schedule the Coval CLI
uses. The live audit is strict on that pull request and advisory on others, so
an API change never blocks unrelated work. Push reconciliation commits onto the
rolling pull request. While that pull request is open, the next weekly run
leaves a branch with commits not made by the bot unchanged and comments on the
pull request instead of resetting it, so merge or rebase that pull request to
let later runs pick up new API changes. Once the pull request is merged or
closed, the next run rebuilds the branch from `main`. If the automation itself
fails, the run opens an issue.

## Container runtime

The hosted container targets Linux x86-64. Build it with
`docker build --platform linux/amd64 .`, including on Apple Silicon.
It uses the checksum-pinned official Node.js 26.11.1 musl distribution with
OpenSSL 3.5.9 and Undici 8.11.2; other container architectures fail explicitly.
The local npm package still supports the Node versions declared in `package.json`.

## Environment Variables

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `COVAL_API_KEY` | Stdio | - | Coval API key for the local stdio transport |
| `COVAL_API_BASE_URL` | No | `https://api.coval.dev/v1` | API base URL |
| `SOFIA_DELEGATION_ORIGIN` | No | Derived from `COVAL_API_BASE_URL` | Overrides the expected Sofia origin used to validate delegation URLs |
| `LOG_LEVEL` | No | `info` | Logging level |
| `MCP_ALLOWED_ORIGINS` | No | Claude and OpenAI web origins | Comma-separated exact browser origins allowed to call `/mcp` or `/claude/mcp`; clients that omit `Origin` remain supported |
| `OPENAI_APPS_CHALLENGE` | No | - | OpenAI plugin-portal domain verification token served as plain text from `/.well-known/openai-apps-challenge` |
| `DD_ENV` | No | `local` | Deployment environment included in structured hosted-request logs |
| `DD_SERVICE` | No | `coval-mcp-server` | Service name included in structured hosted-request logs |
| `DD_VERSION` | No | `COVAL_MCP_SOURCE_SHA` or `unknown` | Immutable release identifier included in structured hosted-request logs |

## Documentation

- [Coval Documentation](https://docs.coval.dev)
- [MCP Protocol](https://modelcontextprotocol.io)
- [API Reference](https://docs.coval.dev/api)
- [Privacy Policy](https://www.coval.ai/privacy-policy)
- [Connector directory submission checklist](docs/directory-submission.md)

## License

MIT

## Support

- [GitHub Issues](https://github.com/coval-ai/mcp-server/issues)
- [Coval Support](mailto:support@coval.dev)

## Package releases

A reviewed PR that increases the stable version in `package.json` and
`package-lock.json` triggers `.github/workflows/publish.yml` after CI passes
on that exact `main` commit. The workflow builds and tests the package,
publishes the verified archive with npm provenance, checks registry integrity,
and creates a matching GitHub release. Commits without a version increase do
not publish. Re-running the successful main CI run retries a failed release;
an existing npm version is accepted only when its archive integrity matches.
Hosted MCP deployment follows its separate release process.

Before the first automated package release, a package maintainer must configure
[npm trusted publishing](https://docs.npmjs.com/trusted-publishers/) for
`@coval/mcp-server`: GitHub owner `coval-ai`, repository `mcp-server`, workflow
filename `publish.yml`, no environment, and permission to publish. This uses
GitHub OIDC with Node 24 and requires no npm token secret. Until that trust is
configured, version-bump releases fail closed at npm authentication.

TDQS

B3.4/5.0

Scored across 34 tools

Disambiguation4/5

Most tools target a clearly distinct resource+action pair (agents, runs, test sets/cases, metrics, personas, reports, schedules), and descriptions make the boundaries readable. There is some crowding around the 'run' family (list_runs, list_run_templates, list_scheduled_runs, get_run, get_scheduled_run) and consult_sofia acts as a read-only catch-all that overlaps with nearly every read tool, which can invite misselection.

Naming Consistency5/5

Every tool follows a snake_case verb_noun pattern (get_agent, create_test_case, cancel_python_metric_preview, list_scheduled_runs). The only stylistic outliers (read_transcript_export, consult_sofia) still fit the verb_noun convention, so the pattern is predictable throughout.

Tool Count2/5

At 34 tools the surface is heavy for a single evaluation platform, and several tools exist only to support narrow sub-workflows (the four-step python metric draft/preview/publish lifecycle). While each tool has a real function, the set exceeds the 25-tool threshold where scoping starts to feel unwieldy for an agent.

Completeness3/5

Core create/read/update paths exist for agents, test cases, and schedules, but deletion and some update operations are missing across the board (no delete_agent, delete_test_case, delete_test_set, update_test_set, create/update_persona, delete_report, cancel_run). Metrics and personas are especially thin on the write side, leaving agents with dead ends on cleanup and persona management.

Maintenance

ActivityActive
ResponsivenessLimited data