# Jev agent tool

A small CLI, Python interface, and local MCP server for TypeSafe's Jev model.
Agents invent typed questions at runtime; Jev returns structured decisions and
probabilities. One shared implementation uses the official TypeSafe Python SDK.

**Public beta 0.1.0b1.** An independent community integration, not an official
TypeSafe product. Bring your own [TypeSafe API key](https://console.typesafe.ai/).
No account with this project, hosted proxy, or shared API key is needed.

## Install and authenticate

Requires Python 3.10 or newer. With [pipx](https://pipx.pypa.io/):

```sh
pipx install 'jev-agent-tool==0.1.0b1'
jev auth
jev init
jev evaluate --input jev-demo/tickets.jsonl --questions jev-demo/questions.json --output results.jsonl
python jev-demo/summarize.py results.jsonl
```

For the Python API, install `jev-agent-tool==0.1.0b1` into your project's virtual
environment using pip. To try the GitHub release before it is indexed on PyPI:

```sh
pipx install 'https://github.com/nandansrikrishna/jev-agent-tool/releases/download/v0.1.0b1/jev_agent_tool-0.1.0b1-py3-none-any.whl'
```

`jev auth` prompts without echoing your key. On macOS/Linux it saves the key to
`~/.config/jev/api-key` with mode `0600`; on Windows it uses Windows Credential
Manager through keyring. `TYPESAFE_API_KEY` overrides saved credentials.
For automated setup, pipe a key to `jev auth --stdin`. Never put keys in question
files, source code, or MCP arguments. Your records and questions go directly to
TypeSafe and use your account's API quota. This tool has no analytics or telemetry
of its own. Local result files contain model judgments and your record IDs.

`jev init` creates a new `jev-demo` directory containing the example questions,
tickets, and summary script; `--directory` chooses a different directory.
The examples below use the repository's `examples/` directory; for a package-only
installation, substitute `jev-demo/`.

## Evaluate a dataset

```sh
jev validate --questions examples/questions.json --input examples/tickets.jsonl
jev evaluate --input examples/tickets.jsonl --questions examples/questions.json --output results.jsonl
python examples/summarize.py results.jsonl
```

Input is JSONL, one `{"id":"unique-string", "state":"context to evaluate"}` per
line. State can also be a JSON object or array. The questions file maps IDs to:

- `choice`: `instructions` plus `criteria` mapping option names to descriptions.
- `score`: `instructions` plus an ordered `criteria` list, at least two levels.
- `noul`: `instructions` for a yes/no judgment; returns probability of yes.

See `examples/questions.json` for all three. Use `jev schema` for the underlying
SDK JSON Schema. Jev additionally requires nonempty question IDs, instructions,
and at least two criteria for choice/score; `jev validate` checks these rules.

Output preserves the input ID and includes `answers`, `model`, `usage`, and a
request fingerprint. Failed records contain `error` instead of answers.
Errors intentionally omit upstream bodies and private input. All JSONL results
go to the output file or stdout; progress and diagnostics go to stderr.

```sh
# stdin/stdout pipelines
cat examples/tickets.jsonl | jev evaluate --input - --questions examples/questions.json > results-pipe.jsonl

# Continue an interrupted run, retry failed records, skip unchanged successes
jev evaluate --input examples/tickets.jsonl --questions examples/questions.json --output results.jsonl --resume
```

`--workers 4` is the default (1–32 supported). `--model jev-latest` is configurable.
The official SDK handles transient retries (two retries by default), rate-limit
backoff, and typed response validation. Each record is one API request containing
all its questions. Resume fingerprints cover record contents, questions, and model
name. A change to any of these triggers another evaluation. The `jev-latest` alias
can change upstream without changing the fingerprint: start a new output file if
you want to refresh old predictions. Resume appends attempts; use the latest result
per ID when aggregating. The example summary does this.

Existing output files are protected unless `--resume` is supplied. A malformed or
partially written JSONL output fails safely; repair its final line before resuming.
Exit codes: `0` success, `1` evaluation failures, `2` input/configuration error,
`130` interrupted. This MVP loads records in memory and uses a thread pool; it has
no hosted queue or distributed workers. Do not run concurrent writers on one output.

## Python

```python
from jev import Client, Choice, Noul, Score

with Client() as client:
    result = client.evaluate(
        state="I sign in successfully but get logged out immediately.",
        questions={
            "bug_relevance": Noul(instructions="Does the user get logged out immediately after signing in?"),
            "urgency": Score(instructions="How severe is the impact?", criteria=["Minor", "Work impaired", "Fully blocked"]),
        },
    )
    print(result.answers["bug_relevance"].noul)
```

Raw question dictionaries work too. Responses are the official SDK's typed models;
use `result.model_dump(mode="json")` for plain JSON-compatible objects.

## MCP

`jev mcp` runs a local stdio server with three tools:

- `question_schema`: discover the question format.
- `evaluate`: evaluate a single context with dynamically authored questions.
- `evaluate_batch`: evaluate 1–50 `{id, state}` records with per-record results.

For Codex, after installing with pipx and authenticating:

```sh
codex mcp add jev -- jev mcp
```

If your desktop app cannot find `jev` on PATH, use its absolute path (`which jev`
on macOS/Linux, `where jev` on Windows). Other MCP clients can use:

```json
{
  "mcpServers": {
    "jev": {
      "command": "/absolute/path/to/jev",
      "args": ["mcp"]
    }
  }
}
```

The server reads the same local credential file or environment variable as the
CLI (including Windows Credential Manager). Restart/reconnect your agent after registration. For batches that exceed the
client's tool timeout, use fewer records or run the CLI; MCP batch results are
returned inline and are not resumable. No remote hosting or OAuth is included.

## Agent usage

Define narrow questions with complete instructions. Group all independent
questions about one record into one call. Aggregate, threshold, and rank in code;
inspect representative records before drawing conclusions. Questions do not see
each other's answers. Probabilities and scores are predictions, not proof.
Use the CLI for large file-based jobs and MCP for direct interactive calls.

## Development

```sh
python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/python -m pytest -q
.venv/bin/python -m build
.venv/bin/python -m twine check dist/*
```

On Windows, use `.venv\Scripts\python.exe` and `.venv\Scripts\pip.exe`.
CI checks Python 3.10 and 3.14 on Linux, macOS, and Windows. Tests use synthetic
records and do not need an API key. See [RELEASING.md](https://github.com/nandansrikrishna/jev-agent-tool/blob/main/RELEASING.md)
for publishing instructions.

API documentation: https://docs.typesafe.ai/api