mewcp-jev

Structured, rubric-based LLM evaluation for your agents.

A Model Context Protocol (MCP) server that exposes TypeSafe AI's Jev System One evaluation API for scoring text or structured state against typed yes/no, single-choice, and rubric questions.

Overview

The mewcp-jev MCP Server provides:

• One-call evaluation of a state (plain text or structured data) against any number of typed questions, each answered independently and in parallel

• Support for mixing yes/no (noul), single-select (choice), and rubric-scored (score) questions within the same request

• Discovery of the model aliases and versions this account can use for evaluation requests

Perfect for:

• Automated grading or QA of LLM outputs, chat transcripts, or agent state

• Evaluation pipelines that need yes/no, single-select, and rubric scoring in one pass

• Picking the right Jev model alias or pinned version before running an evaluation

Related MCP server: jev-agent-mcp

Tools

Evaluates the given state against a map of typed questions and returns one structured answer per question, keyed by the same ids used in the request. Every question is evaluated independently and in parallel against the same state, and noul (yes/no), choice (single-select), and score (rubric) questions can be freely mixed within a single call.

Inputs:

- `state` (string | object | array, required) — The content to evaluate: a plain string for text, or structured data (object/array) for chat logs, records, or app state.
- `model` (string, required) — The model that handles the request, e.g. 'jev-latest' (alias) or a pinned version like 'jev-1.13.0'. See the models group.
- `questions` (object<string, Question>, required) — A map of typed Question objects, keyed by a caller-chosen id — answers come back under the same keys, and the key itself is never sent to the model. Each Question is a discriminated union on its required 'type' field, one of 'noul', 'choice', or 'score'; all three also accept 'instructions' (required, string | object | array | null — arbitrary JSON is accepted for clarity or to pass supporting data verbatim). Noul ('type': 'noul') is a yes/no question with an optional 'criteria' object holding optional 'true'/'false' descriptions (string, object, or array) of what a yes/no answer means. Choice ('type': 'choice') picks one option from a set and requires 'criteria': a map of option name to an optional rubric description (string, object, array, or null when the option needs no extra detail) — one entry per option. Score ('type': 'score') rates the state on a rubric and requires 'criteria': an ordered array of at least two level descriptions (each a string, or an object/array for structured rubric detail). Any mix of the three types is allowed in the same map.

Output data schema:

{
  model: string;
  answers: {
    [question_id: string]: {
      type: string;
      noul?: number;
      choice?: string;
      probabilities?: { [option: string]: number };
      confidence?: number;
      score?: number;
      legend?: { [level: string]: string };
    };
  };
  usage: {
    input_tokens: number;
    output_tokens: number;
  };
}

Lists the model names and aliases this account can pass in the evaluate-state model field, with a description and release date for each. This list currently only surfaces aliases such as jev-latest and jev-preview — a versioned model ID (e.g. jev-1.13.0) is still accepted by the evaluate-state model field even when it does not appear here.

Inputs:

This tool takes no input parameters.

Output data schema:

{
  models: {
    name: string;
    description: string;
    release_date: string;
  }[];
}

API Parameters Reference

Every tool returns the same top-level envelope. Only data varies per tool.

// Success
{
  "success": true,
  "statusCode": 200,
  "retriable": false,
  "retry_after_seconds": null,
  "error": null,
  "data": { ... }
}

// Error
{
  "success": false,
  "statusCode": 400,
  "retriable": false,
  "retry_after_seconds": null,
  "error": { "code": "{ERROR_CODE}", "message": "{description}", "details": {} },
  "data": null
}

• retriable — true when it is safe to retry (rate limit, network error, 503). false for validation and auth errors.

• retry_after_seconds — seconds to wait before retrying; present only when retriable is true and the upstream specifies a delay.

• error.code — machine-readable string: VALIDATION_ERROR, AUTH_ERROR, UPSTREAM_ERROR, SERVER_ERROR.

Getting Your TypeSafe AI API Key

1. Go to the TypeSafe AI API Documentation

2. Follow the Authentication section to sign in to your TypeSafe AI account and open the API keys area

3. Click Create API Key (or equivalent)

4. Copy the generated key — you will only see it once

Troubleshooting

• Cause: API key not provided in request headers or incorrect format

• Solution:

  1. Verify Authorization: Bearer YOUR_API_KEY and X-Mewcp-Credential-Id: CREDENTIAL-ID headers are present

  2. Check API key is active in your MewCP account

• Cause: API calls have exceeded your request limits

• Solution:

  1. Check credit usage in your Curious Layer dashboard

  2. Upgrade to a paid plan or add credits for higher limits

  3. Contact support for credit adjustments

• Cause: No TypeSafe AI credential linked to your account

• Solution:

  1. Go to Credentials in your MewCP dashboard

  2. Connect your TypeSafe AI account (OAuth) or add your API key (static)

  3. Retry the request with the correct X-Mewcp-Credential-Id header

• Cause: JSON payload is invalid or missing required fields

• Solution:

  1. Validate JSON syntax before sending

  2. Ensure all required tool parameters are included

  3. Check parameter types match expected values

• Cause: Incorrect server name in the API endpoint

• Solution:

  1. Verify endpoint format: {server-name}/mcp/{tool-name}

  2. Use correct server name from documentation

  3. Check available servers in your Curious Layer account

• Cause: Upstream TypeSafe AI API returned an error

• Solution:

  1. Check the TypeSafe AI API Documentation for current service notices

  2. Verify your credential has the required permissions

  3. Review the error message for specific details

• TypeSafe AI API Documentation — Official API reference

• TypeSafe AI API Reference — Complete endpoint reference

• FastMCP Docs — FastMCP specification

• FastMCP Credentials — FastMCP Credentials package for credential handling