Jev agent tool

A small CLI, Python interface, and local MCP server for TypeSafe's Jev model. Agents invent typed questions at runtime; Jev returns structured decisions and probabilities. One shared implementation uses the official TypeSafe Python SDK.

Public beta 0.1.0b1. An independent community integration, not an official TypeSafe product. Bring your own TypeSafe API key. No account with this project, hosted proxy, or shared API key is needed.

Install and authenticate

Requires Python 3.10 or newer. With pipx:

pipx install 'jev-agent-tool==0.1.0b1'
jev auth
jev init
jev evaluate --input jev-demo/tickets.jsonl --questions jev-demo/questions.json --output results.jsonl
python jev-demo/summarize.py results.jsonl

For the Python API, install jev-agent-tool==0.1.0b1 into your project's virtual environment using pip. To try the GitHub release before it is indexed on PyPI:

pipx install 'https://github.com/nandansrikrishna/jev-agent-tool/releases/download/v0.1.0b1/jev_agent_tool-0.1.0b1-py3-none-any.whl'

jev auth prompts without echoing your key. On macOS/Linux it saves the key to ~/.config/jev/api-key with mode 0600; on Windows it uses Windows Credential Manager through keyring. TYPESAFE_API_KEY overrides saved credentials. For automated setup, pipe a key to jev auth --stdin. Never put keys in question files, source code, or MCP arguments. Your records and questions go directly to TypeSafe and use your account's API quota. This tool has no analytics or telemetry of its own. Local result files contain model judgments and your record IDs.

jev init creates a new jev-demo directory containing the example questions, tickets, and summary script; --directory chooses a different directory. The examples below use the repository's examples/ directory; for a package-only installation, substitute jev-demo/.

Related MCP server: jev-mcp

Evaluate a dataset

jev validate --questions examples/questions.json --input examples/tickets.jsonl
jev evaluate --input examples/tickets.jsonl --questions examples/questions.json --output results.jsonl
python examples/summarize.py results.jsonl

Input is JSONL, one {"id":"unique-string", "state":"context to evaluate"} per line. State can also be a JSON object or array. The questions file maps IDs to:

• choice: instructions plus criteria mapping option names to descriptions.

• score: instructions plus an ordered criteria list, at least two levels.

• noul: instructions for a yes/no judgment; returns probability of yes.

See examples/questions.json for all three. Use jev schema for the underlying SDK JSON Schema. Jev additionally requires nonempty question IDs, instructions, and at least two criteria for choice/score; jev validate checks these rules.

Output preserves the input ID and includes answers, model, usage, and a request fingerprint. Failed records contain error instead of answers. Errors intentionally omit upstream bodies and private input. All JSONL results go to the output file or stdout; progress and diagnostics go to stderr.

# stdin/stdout pipelines
cat examples/tickets.jsonl | jev evaluate --input - --questions examples/questions.json > results-pipe.jsonl

# Continue an interrupted run, retry failed records, skip unchanged successes
jev evaluate --input examples/tickets.jsonl --questions examples/questions.json --output results.jsonl --resume

--workers 4 is the default (1–32 supported). --model jev-latest is configurable. The official SDK handles transient retries (two retries by default), rate-limit backoff, and typed response validation. Each record is one API request containing all its questions. Resume fingerprints cover record contents, questions, and model name. A change to any of these triggers another evaluation. The jev-latest alias can change upstream without changing the fingerprint: start a new output file if you want to refresh old predictions. Resume appends attempts; use the latest result per ID when aggregating. The example summary does this.

Existing output files are protected unless --resume is supplied. A malformed or partially written JSONL output fails safely; repair its final line before resuming. Exit codes: 0 success, 1 evaluation failures, 2 input/configuration error, 130 interrupted. This MVP loads records in memory and uses a thread pool; it has no hosted queue or distributed workers. Do not run concurrent writers on one output.

Python

from jev import Client, Choice, Noul, Score

with Client() as client:
    result = client.evaluate(
        state="I sign in successfully but get logged out immediately.",
        questions={
            "bug_relevance": Noul(instructions="Does the user get logged out immediately after signing in?"),
            "urgency": Score(instructions="How severe is the impact?", criteria=["Minor", "Work impaired", "Fully blocked"]),
        },
    )
    print(result.answers["bug_relevance"].noul)

Raw question dictionaries work too. Responses are the official SDK's typed models; use result.model_dump(mode="json") for plain JSON-compatible objects.

MCP

jev mcp runs a local stdio server with three tools:

• question_schema: discover the question format.

• evaluate: evaluate a single context with dynamically authored questions.

• evaluate_batch: evaluate 1–50 {id, state} records with per-record results.

For Codex, after installing with pipx and authenticating:

codex mcp add jev -- jev mcp

If your desktop app cannot find jev on PATH, use its absolute path (which jev on macOS/Linux, where jev on Windows). Other MCP clients can use:

{
  "mcpServers": {
    "jev": {
      "command": "/absolute/path/to/jev",
      "args": ["mcp"]
    }
  }
}

The server reads the same local credential file or environment variable as the CLI (including Windows Credential Manager). Restart/reconnect your agent after registration. For batches that exceed the client's tool timeout, use fewer records or run the CLI; MCP batch results are returned inline and are not resumable. No remote hosting or OAuth is included.

Agent usage

Define narrow questions with complete instructions. Group all independent questions about one record into one call. Aggregate, threshold, and rank in code; inspect representative records before drawing conclusions. Questions do not see each other's answers. Probabilities and scores are predictions, not proof. Use the CLI for large file-based jobs and MCP for direct interactive calls.

Development

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/python -m pytest -q
.venv/bin/python -m build
.venv/bin/python -m twine check dist/*

On Windows, use .venv\Scripts\python.exe and .venv\Scripts\pip.exe. CI checks Python 3.10 and 3.14 on Linux, macOS, and Windows. Tests use synthetic records and do not need an API key. See RELEASING.md for publishing instructions.

API documentation: https://docs.typesafe.ai/api