typesafe-mcp
Provides an evaluate tool that uses Vercel AI Gateway as a backend to answer typed questions about content, supporting yes/no probability judgments, multiple-choice selections, and rubric-based scoring.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@typesafe-mcpevaluate this review: 'fast shipping but poor packaging' for sentiment and a 1-5 score"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
system-one-mcp
Local stdio MCP server exposing TypeSafe's System One API as an MCP tool, for Claude Code and other MCP agents. No install step required -- run it with npx.
What it does
The downstream API answers one or more typed questions about a piece of content in a single call. This is exposed as one MCP tool:
Tool | Description |
| Evaluate |
Related MCP server: jev-mcp
Setup
Get a TypeSafe API key (or a credential for one of the alternate backends below).
Run it with your key in the environment:
TYPESAFE_API_KEY=your-key npx system-one-mcpThe process exits immediately with an error if no backend can be configured.
Claude Code / agent config
{
"mcpServers": {
"typesafe": {
"command": "npx",
"args": ["-y", "system-one-mcp"],
"env": {
"TYPESAFE_API_KEY": "your-typesafe-api-key"
}
}
}
}Backends
evaluate calls whichever backend is configured. Set exactly one credential and it's picked automatically, in this order:
Backend | Env var | Notes |
TypeSafe direct (default) |
| Via the official |
OpenRouter |
| Jev's alpha Decisions endpoint; |
Vercel AI Gateway |
|
|
Set JEV_BACKEND (typesafe | openrouter | vercel) to force a specific backend instead of relying on auto-detection.
Already using OPENROUTER_API_KEY, AI_GATEWAY_API_KEY, or TYPESAFE_API_KEY for something else in the same environment? Point this server at a different variable instead of renaming yours, by setting <VAR>_ENV to that variable's name:
OPENROUTER_API_KEY_ENV=MY_JEV_OPENROUTER_KEY MY_JEV_OPENROUTER_KEY=sk-... npx system-one-mcpThis works the same way for all three credentialed backends (TYPESAFE_API_KEY_ENV, OPENROUTER_API_KEY_ENV, AI_GATEWAY_API_KEY_ENV).
Development
npm install
cp .env.example .env # fill in a credential
npm run dev # run from source via tsx
npm run build # bundle to dist/index.js
npm run typecheck
npm testTests (npm test) spawn the real stdio server as a child process (via tsx, exactly as an agent would run it) for each backend, pointed at a local mock HTTP server standing in for the real provider -- they prove credentials (including redirected ones) are forwarded correctly, downstream errors propagate unmodified, and retries happen where the API docs say they should. No live calls are made.
Project layout
src/
index.ts MCP stdio server entrypoint
toolRegistry.ts Registers ./tools/* against the MCP server
backends/ One adapter per provider behind a common EvaluateBackend interface
types.ts Shared types, reusing @typesafe-ai/sdk's own request/response types
typesafe.ts Default backend, via the official SDK
openrouter.ts Same wire dialect as TypeSafe, different host
vercel.ts Translates to/from Vercel AI Gateway's dialect
http.ts Shared HTTP/retry plumbing for openrouter/vercel
index.ts Backend selection (JEV_BACKEND, auto-detect, *_API_KEY_ENV redirects)
tools/
evaluate.ts Tool handler
evaluate.json Tool schema (name, description, input schema)
tests/
evaluate.test.ts End-to-end tests, one backend at a timeAdd a new tool by creating <name>.json + <name>.ts side by side under src/tools/, then listing both in toolRegistry.ts.
Security
Your credential is read once at startup and held only in this process's memory -- it is never written to disk or sent anywhere except as the Authorization: Bearer <key> header on calls to the configured backend's host (api.typesafe.ai, openrouter.ai, or ai-gateway.vercel.sh, per the table above). Treat it like any other bearer credential. Don't paste it into chat messages, tickets, or other insecure channels.
No secrets are committed to source control: .env is gitignored, and only .env.example (no real values) is tracked.
Available Tools
1 toolevaluateA
Evaluate content against one or more typed questions using TypeSafe's System One API (model 'jev'). Supports three question primitives that can be mixed in a single call: noul (yes/no, returns a probability 0-1), choice (select the best-fitting option from a fixed set), and score (rate against a rubric with 2-10 levels). All questions in a call are answered in parallel against the same input.
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Model identifier to use for evaluation, e.g. "jev-latest". | |
| state | Yes | The content to evaluate. A plain string for text, or a structured object/array to evaluate as JSON. | |
| questions | Yes | Map of question id to question definition. Each key becomes the corresponding answer's key in the response. At least one question is required. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals key behaviors: all questions are answered in parallel against the same input, and noul returns a probability 0-1. It does not disclose potential side effects, authentication needs, or rate limits, but for an evaluation tool these are likely irrelevant. The disclosed parallel execution and probability return add meaningful transparency beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, with the core purpose front-loaded and no redundant phrases. It efficiently covers the tool's function, the three question types, and a key behavioral trait (parallel execution). Every word earns its place, and the structure is highly readable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with nested objects and three question types, the description is largely complete. It explains the primitives and the parallel behavior. The schema already details the criteria structure and the mapping of question ids to answer keys, so the description does not need to repeat those. However, there is no output schema, and the description does not describe the overall response format (beyond noul's probability), which could leave some ambiguity about what the tool returns for choice and score. This slight gap prevents a perfect score.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter is already documented. The description adds value by clarifying behavioral semantics of the question types (e.g., noul returns a probability, choice selects best-fitting option) and the parallel execution model. This goes beyond the schema, which only lists types and criteria, by explaining the expected behavior of each primitive. A baseline 3 is elevated to 4 due to this extra context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear, specific purpose: evaluating content against typed questions. It names the API (TypeSafe's System One) and model (jev), and lists the three supported question primitives (noul, choice, score). This is far from a tautology and clearly differentiates the tool's function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
While no sibling tools are provided, the description gives strong contextual guidance on when to use this tool: whenever content needs to be evaluated against structured questions. It also explains that the three question primitives can be mixed in a single call, implying flexibility. However, it does not explicitly state when not to use it or mention any alternative approaches, but given the absence of siblings, this is acceptable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v0.1.0- First observed
evaluate
TDQS
Scored across 1 tool
There is only one tool, so there is no possibility of an agent confusing it with another. The description clearly defines what 'evaluate' does and its supported question primitives.
With a single tool, naming inconsistency is impossible. 'evaluate' is a clear, action-oriented verb that accurately describes the tool's purpose.
One tool feels thin for a server named 'typesafe-mcp', even though it is not trivial because it packages multiple evaluation primitives. The minimal surface is workable but leaves little room for supporting operations.
The tool covers the full stated domain: noul, choice, and score questions can be mixed and answered in parallel. No obvious required operations are missing for this narrow evaluation-focused use case.
Related MCP Connectors
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Evidence-readiness MCP server: validate, audit, and score briefs, memos, and evidence packs.
Remote MCP for Gemini upgrade evals, prompt regressions, output diffs, and eval receipts.
Agent-driven search: build, import, tune, search, and score result quality — all over MCP.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables MCP agents to submit content for human review via LoopQuest, returning verdicts asynchronously.24 npmMIT
- AlicenseAqualityCmaintenanceEnables agents to perform typed judgments—classify, score, check, match, and screen—over closed answer sets with confidence scores, without text generation.74MIT
- AlicenseAqualityCmaintenanceEnables MCP clients to call TypeSafe's JEV classifier and receive structured, typed judgments with probabilities for binary, choice, and scoring questions.17MIT
- AlicenseNot gradedqualityBmaintenanceEnables structured, rubric-based evaluation of text or structured state using TypeSafe AI's Jev System One API, supporting yes/no, single-choice, and rubric-scored questions in parallel.Apache 2.0