OpenJev

Typed decisions instead of prompt-and-parse: an open-source, self-hostable service that turns text plus typed questions into calibrated choice / score / noul answers.

Instead of prompting an LLM and parsing prose, you POST application state plus a set of typed questions and get back typed judgments with probabilities that ordinary code can compose. The default engine is a local sentence-embedding classifier (gte-small, int8 ONNX, CPU-only) with a lexical TF-IDF fallback that needs no model at all, so it runs anywhere Node runs. Accuracy and calibration (ECE) are measured on public datasets, not assumed.

This repository is the whole project in one place: the API server + web console, the native engine, the SDKs, the agent skills, and the otsone model toolkit used to train and serve our own model family. Licensed under Apache-2.0 (the SDK, adapter and skills subdirectories derived from TypeSafe AI's MIT SDKs keep their MIT licenses; see Licensing).

Status. OpenJev is self-host only. There is no official hosted OpenJev service, and the SDK packages are not yet published to npm or PyPI: install them from this repository (see SDKs). The openjev.com domain and the PyPI name openjev belong to unrelated projects.

Already use the TypeSafe SDK? Change one line

The official typesafe-sdk (Python) and @typesafe-ai/sdk (JavaScript) packages work against an OpenJev server unmodified. Point the base URL at your server and keep your code:

export TYPESAFE_BASE_URL=http://localhost:8080   # with or without a trailing /v1
export TYPESAFE_API_KEY=anything                 # the SDK requires a value; a self-hosted server ignores it

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="I was charged twice for order A-104, please refund the duplicate.",
        questions={"team": Choice(instructions="Which team?", criteria={"billing": None, "technical": None})},
    )
print(response.choices["team"].choice, response.request_id)

Checked on 2026-11-11 with typesafe-sdk 0.7.5 and @typesafe-ai/sdk 0.7.0 against a local server: choice, score and noul questions, several questions in one request, string and object state, models.list(), request ids, error classes for a rejected request, and the zstd-compressed request bodies both SDKs send for bodies of 1 KiB or more. The runs are examples/official-sdk.py and examples/official-sdk.mjs.

Not supported: image assets (multipart requests get a 415) and box questions (400). An unknown model name, including the SDK default jev-latest, is answered by the local system-one engine, and the response's model field says so. Details: /docs/typesafe-sdk.

Related MCP server: jev-mcp

60-second quickstart

With Docker (API + console on http://localhost:3000, plus the optional laya-onnx sidecar):

git clone https://github.com/caelum0x/openjev && cd openjev
docker compose up --build

Or with Node.js 22+ (dev server on http://localhost:8080):

git clone https://github.com/caelum0x/openjev && cd openjev
npm install
npm run embed-download           # optional: ~33 MB gte-small model; without it the lexical engine answers
npm run dev

Then ask it something (use port 3000 for Docker):

curl -s localhost:8080/api/v1/systemone -H 'content-type: application/json' -d '{
  "model": "system-one",
  "state": "I was charged twice for order A-104, please refund the duplicate",
  "questions": {
    "department": { "type": "choice", "instructions": "Which team should handle this?",
      "criteria": { "billing": "payments, refunds", "technical": "bugs, outages", "sales": "pricing" } },
    "urgency": { "type": "score", "instructions": "How urgent is this?",
      "criteria": ["low", "medium", "high", "critical"] },
    "is_refund": { "type": "noul", "instructions": "Is the customer asking for a refund?" }
  }
}'

Every response says which engine answered (engine.kind is embedding or lexical). Nothing is required to boot: no API key, no database (an in-memory PGlite is used unless you set DATABASE_URL), no model download.

No server: classify from the command line

openjev classify labels text with the engine straight from this checkout. No server, no API key, no model download. It uses Node.js built-in type stripping (Node.js 22.6+):

git clone https://github.com/caelum0x/openjev && cd openjev && npm install

# one text per line on stdin
node scripts/cli.mjs classify \
  "bug:crash error broken fails,feature:request add option,praise:love great thanks" \
  < examples/feedback.txt

# CSV or JSONL in, CSV or JSONL out; rows under the threshold are left blank for review
node scripts/cli.mjs classify bug,feature,praise --file data.csv --column text --threshold 0.6 --out results.csv
node scripts/cli.mjs classify bug,feature,praise --file data.jsonl --field text --json

Each row gets a label, a probability and the engine that answered. Labels are a comma-separated list; a description after a colon tells the engine when a label applies, which matters most for the lexical engine. With the gte-small model installed (npm run embed-download) the embedding engine answers instead. A row where the engine cannot tell the labels apart is left blank and marked for review.

Lexical-engine accuracy, measured, not claimed. Without the embedding model, the lexical engine's only signal for a bare label (bug,feature,praise, no :description) is the label word itself plus a small built-in synonym lexicon (src/lib/jev/lexicon.ts) — if a label's word isn't in that lexicon, texts that don't literally contain the word collapse to a tie and get left blank for review, and texts that happen to share a different label's vocabulary (e.g. a praise message that mentions a fixed "crash") can be scored confidently wrong. scripts/classify-eval.mjs measures this on a 65-row synthetic gold set (examples/classify-eval.jsonl, built for this repo, not scraped) across three label sets (bug,feature,praise / positive,negative,neutral / spam,sales,support), including rows tagged shared_vocab: true for ones that deliberately reuse another label's cue words.

Run it: npm run classify-eval.

Before this pass: feature and praise had zero lexicon coverage and positive / negative / neutral had none at all, so most rows either tied (positive → 32.3% accuracy counting ties as wrong, 67.7% on the rows it actually committed to) or, when a bug-shaped word like "crash" appeared anywhere, confidently mislabeled a praise or feature row as a bug. Adding two small synonym groups to the existing lexicon (positive sentiment + praise cues; feature-request cues) — no model, no training, no change to the scoring math — moved the same suite to 49.2% accuracy counting ties as wrong, 71.1% on decided rows, with the positive,negative,neutral set (previously almost all ties) at 35% and the shared_vocab: true subset at 47.5%. npm run gold (the product-wide 61-case regression suite) still passes 61/61 — the fix is additive new synonym entries, not a change to any existing one.

Real limit, not fixed here: a lexical bag-of-words engine cannot fully resolve a praise sentence that mentions "crash" ("thanks for fixing the crash") from a bug report — that needs the embedding engine or richer per-call descriptions. --threshold trades coverage for precision on the lexical engine: in this suite, --threshold 0.55 raises accuracy on the rows it still commits to from 71.1% to 83.8% by abstaining on 28/65 rows instead of 20/65 (npm run classify-eval prints the full sweep). For labels the lexicon doesn't know, a name:description (as in the stdin example above) still outperforms both, because it gives the lexical engine real words to match on instead of relying on the built-in lexicon.

MCP server

Agents that speak the Model Context Protocol can call typed decisions as tools. The running server exposes POST /v1/mcp (stateless Streamable HTTP) with four tools: choice, score, noul and decide (several typed questions in one call). No API key is needed for a self-hosted server.

# Claude Code, HTTP
claude mcp add --transport http openjev http://localhost:8080/v1/mcp

# Claude Code, stdio (forwards to the server at OPENJEV_BASE_URL)
claude mcp add openjev --env OPENJEV_BASE_URL=http://localhost:8080 -- node /path/to/openjev/scripts/cli.mjs mcp

For Cursor, add { "mcpServers": { "openjev": { "url": "http://localhost:8080/v1/mcp" } } } to .cursor/mcp.json. Tool reference and a curl check: /integrations/mcp in the console (source: src/lib/content/integrations.ts).

The three model families

The same POST /v1/systemone contract runs on any of three families:

Neural models health-gate: if their sidecar is absent, the request is answered by the local engine and the response says so (requested vs served_by). An optional escalation model (openjev-grok) routes hard decisions to OpenAI / xAI / Anthropic when you supply a provider key.

Measured accuracy of the default engine

Zero-shot (the engine sees only the class names), full test splits, from src/lib/content/leaderboard.json and rendered at /leaderboard. To reproduce (the datasets are downloaded at run time into .cache/; the embedding column needs the pinned gte-small model):

npm install
npm run embed-download
LEADERBOARD_PUBLIC=1 LEADERBOARD_PUBLIC_SAMPLE=all npm run leaderboard

The engine's constants were fitted on held-out data only (Banking77 train, CLINC150 validation, half of the in-repo gold suite); AG News was never used for fitting. On the small in-repo gold suite, which the lexical engine was tuned on, the lexical engine is ahead (85.6% vs 82.5% accuracy; 94.7% vs 61.3% of off-topic probe pairs flagged); treat the gold suite as a regression check, not a benchmark.

See docs/COMPARISON.md for how these numbers, and our otsone checkpoints, stack up against TypeSafe Jev, the open-source Kev model and the OpenAI Decisions API — with sources, and the gaps honestly flagged.

The trust layer

What a System One model alone does not give you:

• Out of scope and abstain. Every answer carries an ood score (on the embedding engine, a fitted detector over similarity to the options and their centroid plus a Mahalanobis-style distance; on other backends, distance of the state from the question's vocabulary, modulated by answer entropy), and off-topic inputs lose their confidence instead of coming back at 0.99. Set abstain: true (request or question) to get abstained: true and, for Choice, the reserved __none__ option.

• auto cascade. model: "auto" answers locally (otsone when its sidecar is up, else System One) and escalates only low-confidence or out-of-scope questions to an LLM: Groq (free tier), Ollama (local), or OpenAI/xAI/Anthropic when keyed. The response's escalation block lists what escalated and why.

• Calibration on your labels. POST /api/v1/calibrations fits a temperature or isotonic map per question from your labelled rows (CSV/JSONL upload in /calibrations, or straight from an eval run). With 20+ labels per question a stratified quarter is held out: the map is fitted on the rest and the before/after accuracy, ECE and Brier are measured on labels it never saw. Pass calibration_id to apply it. The evals page imports labelled CSV/JSONL (validated against the question schema, with a preview) and has a reliability diagram and a threshold picker.

• Shadow mode. POST /api/v1/shadow runs next to your production LLM and logs agreement; /shadow shows agreement by confidence and projected savings. Send incumbent: { kind: "label", answers } to log human ground truth instead.

• Drift alerts. Each calibration can have a drift monitor (/calibrations/<id> or POST /api/v1/drift). Shadow traffic that used the calibration is compared with its held-out numbers; accuracy drops, confidence shifts, or confidence running ahead of accuracy POST a calibration.drift event to your webhook, signed X-OTS-Signature: t=<unix>,v1=<HMAC-SHA256 of "<t>.<body>">, and calibration.drift_resolved when it clears.

• Public leaderboard. npm run leaderboard scores every available backend on the in-repo gold suite and out-of-scope probes and writes src/lib/content/leaderboard.json (rendered at /leaderboard). With LEADERBOARD_PUBLIC=1 it also scores Banking77, CLINC150 (with its out-of-scope queries) and AG News, downloaded at run time into .cache/.

• Benchmark Submissions. /submit-model lets a third party point a live HTTP endpoint or a precomputed answers file at the same gold suite and scoring engine the built-in models use (accuracy, strict pass rate, ECE, adaptive ECE, MCE, Brier, NLL, per-question-type breakdown), auth- and quota-checked through the existing API-key system and SSRF-guarded before any submitted URL is called. Every run also carries a real execution trace — steps, outbound/tool-call count, wall-clock latency p50/p95, tokens and cost when the endpoint reports them, and error/timeout counts — and an opt-in fault-injection stress test (deterministically seeded timeouts/5xx/malformed responses at configurable rates) that reports robustness: accuracy under faults vs. the same model's clean-run accuracy. Completed runs show up as extra "Community submitted" rows (with their trace + robustness columns) on /leaderboard's gold-suite tab, next to OpenJev's own models. This is the feature behind our Geek Room Global AI Hackathon entry, Umpirix — Benchmark Arena for AI Agents — see hackathons/geekroom-2026/HACKATHON-SUBMISSION.md for the full writeup, the problem-statement fit ("a benchmarking platform for AI agents, not just models"), and a recorded demo.

How Bifrost powers Umpirix

Umpirix's agent-under-test and judge LLM calls run through Bifrost, Maxim AI's open-source LLM gateway, instead of hitting a provider directly. One env switch flips every LLM-backed call in this repo onto it — the public leaderboard's llm row (scripts/leaderboard.mjs), the auto escalation cascade, and the judge stage of the router→gate→judge agent loop (src/lib/jev/loop.ts) — all funnel through the SAME choke point, src/lib/jev/llm.server.ts, so flipping the switch benchmarks the real call every one of those features already makes, not a separate demo path.

Install & run Bifrost locally (the documented quickstart):

npx -y @maximhq/bifrost -port 8099    # gateway + dashboard on :8099

Configure a provider with a file-based config.json in Bifrost's app dir (-app-dir), referencing a key from YOUR OWN environment — never Umpirix's:

{
  "providers": {
    "anthropic": {
      "keys": [{ "name": "anthropic-primary", "value": "env.ANTHROPIC_API_KEY", "models": ["*"], "weight": 1.0 }]
    }
  },
  "client": { "enforce_auth_on_inference": false, "enable_logging": true }
}

Point Umpirix at it:

export BENCH_LLM_GATEWAY=bifrost
export BIFROST_BASE_URL=http://localhost:8099
export LLM_PROVIDER=anthropic         # or groq / openai / xai / ollama — Bifrost's
export ANTHROPIC_API_KEY=...          # provider slugs match this app's LlmProvider ids 1:1

What's real, verified against a live local gateway (2026-11-11), not guessed from docs:

• Multi-provider routing, OpenAI-compatible. Every call goes to POST ${BIFROST_BASE_URL}/v1/chat/completions with {"model":"<provider>/<model>", ...} — Bifrost translates to whichever real backend (anthropic/groq/openai/xai/ollama/…) that provider needs.

• Real server-side fallback. Setting BIFROST_FALLBACK_MODEL passes it as Bifrost's own fallbacks array. Verified live: pointing the primary at a model that does not exist still returned a correct answer, with Bifrost's response reporting routing_info.is_fallback: true and the real model that actually served it — Bifrost's failover, not a client-side retry this app writes.

• Per-call telemetry straight from Bifrost's response, not recomputed: extra_fields.latency, routing_info.provider/model, and the real token usage, read by telemetryFromResponse (src/lib/jev/bifrost.server.ts) into every SystemOneResponse.bifrost and surfaced as a Gateway column/badge on /leaderboard's gold-suite table and in the /submit-model execution trace.

• A real 93-case gold-suite run through the gateway: 93/93 real calls, accuracy 0.9794, Bifrost-measured latency p50/p95 1567/2324 ms, 106,543 input + 7,350 output tokens — all read back from Bifrost's own extra_fields, not this app's timer (which independently measured 1580/2327 ms — a cross-check, not a guess). Full numbers, plus a faulted pass (67/93 real calls after timeout/server_error faults short-circuited the rest, robustness 0.621) and the fallback demo above, are committed at hackathons/geekroom-2026/bifrost-run-results.json and reproducible with hackathons/geekroom-2026/verify-bifrost.mjs.

• Not yet wired to Bifrost's own /api/logs/rate-limit/governance features — those need a dashboard admin account (BIFROST_SETUP_TOKEN), which this integration deliberately skips for a local/CI gateway; every number above comes from the inference response itself, which needs no extra auth. Honest gap, not hidden.

• Signed audit reports. GET /v1/reports/:calibrationId (?format=html for a printable copy) bundles a calibration's maps, raw vs calibrated ECE / Brier, the recommended threshold, drift history and shadow agreement, signed with HMAC-SHA256; POST /v1/reports/verify checks a report was issued by the server and not edited.

Set OPENJEV_SITE_URL to your deployment's public origin; without it the app publishes no sitemap and asks crawlers not to index it.

Layout

src/                 API server + web console (TanStack Start / Vite)
  routes/api/v1/     POST /v1/systemone, batch, models, keys, usage, evals, ...
  lib/jev/           the typed-decision engine (TS), backends, validation, cookbooks
engine/              native engine: C++ kernel, Rust HTTP server, Python binding, gold suite
migrations/          SQL migrations (PGlite in-memory by default, Postgres optional)
scripts/             CLI (`openjev serve`, `openjev mcp`, `openjev classify`), Laya sidecar/download/distill helpers
otsone/              otsone model toolkit (Python): train / calibrate / export / serve
                     otsone-1 and run Laya on MLX or ONNX (formerly laya-mlx)
sdk/js/              `@openjev/sdk` — full TypeScript SDK
sdk/ai-sdk/          `@openjev/ai-sdk` — Vercel AI SDK integration
sdk/js-lite/         `@openjev/lite` — zero-dependency client + decision modules
sdk/python/          Python SDK (import `openjev`)
adapters/python/     `openjev-adapters` (import `openjev_adapters`) — run the same API on top of OpenAI/Anthropic/...
skills/              agent skills (Claude Code, Codex, ...) for building with typed decisions
examples/            runnable examples (see examples/README.md)
docs/                weights provenance, screenshots

SDKs

The HTTP API is the primary interface; any language can call it. The SDKs below live in this repository. Registry names are reserved/claimed and verified free as of 2026-11-09 (@openjev/sdk and @openjev/lite are unclaimed on npm; the plain openjev PyPI name belongs to an unrelated project, as do the openjev-sdk and pyjev fallbacks, so the Python package publishes as openjev-client instead) — publish to the real registries is pending owner npm/PyPI account auth, so install from a clone until then:

Do not pip install openjev from PyPI: that name belongs to an unrelated project. Once published, install the Python SDK with pip install openjev-client (the importable module stays openjev).

Examples

Runnable scripts in examples/ — rank ad-copy hooks (Node.js) or triage support tickets (Python) by batching Choice + Score + Noul per item and applying a policy in code over the returned probabilities, plus the official TypeSafe SDKs pointed at OpenJev (official-sdk.py, official-sdk.mjs). See examples/README.md.

Self-hosting

Docker (Node service + laya-onnx sidecar):

cp .env.example .env             # optional — nothing is required to boot
docker compose up --build        # → http://localhost:3000

See SELF-HOSTING.md for the non-Docker Node server, Postgres, LLM escalation and every environment variable (.env.example).

Serving otsone-1 locally

cd otsone
uv sync --extra onnx --extra train
uv run otsone distill --backbone otsone-base --out models/otsone-base   # train + export
OTSONE_BACKEND=onnx OTSONE_MODEL_PATH=./models/otsone-base uv run otsone-serve

Then point the server at it with OTSONE_MODEL_URL (see .env.example). Model weights are never committed — see docs/WEIGHTS.md.

Development

See PUBLISHING.md for how each package ships a release.

Renamed from opentypesafe

This project was previously called opentypesafe. The HTTP API is unchanged (POST /v1/systemone, /v1/batch, ... and ots_live_ keys), and old names keep working. Package names below are the names the packages are built under; none are published to a registry yet.

Origins & licenses

OpenJev began as an open, wire-compatible reimplementation of the TypeSafe AI System One v1 API, and it is honest about where its pieces come from:

• The MIT-licensed sdk/js, sdk/python, adapters/python and skills directories are derived from TypeSafe AI's SDKs; each keeps its original LICENSE and NOTICE.

• "Jev" is the name of TypeSafe AI's model family, and "System One" is the name of TypeSafe AI's API. OpenJev is an independent project and is not affiliated with, sponsored by or endorsed by TypeSafe AI; those names are used only to describe wire compatibility, and the jev-* model ids exist only as compatibility aliases. OpenJev does not ship or reproduce TypeSafe AI's models. Other unrelated open-source projects also use the name "OpenJev".

• The otsone-1 model is our own; the Laya models are Convai Innovations' (Apache-2.0).

Licensing

• Root project, engine/, otsone/, sdk/js-lite/: Apache-2.0 (LICENSE).

• sdk/js, sdk/python, adapters/python, skills: MIT, derived from the TypeSafe AI SDKs (each directory keeps its original LICENSE and NOTICE).

• otsone/ includes code adapted from Laya (Convai Innovations, Apache-2.0) — see otsone/NOTICE.

• Model weights are not distributed in this repository.

See NOTICE for attributions.

Roadmap

The live roadmap is rendered at /roadmap (source: src/lib/content/roadmap.ts). Near-term priorities:

• Publish the SDKs to npm and PyPI (the Python distribution needs a new name; openjev is taken on PyPI) and a prebuilt Docker image.

• Improve the default engine's accuracy on the public leaderboard, and add more public datasets.

• Go client SDK, decision export (JSONL / CSV), OpenTelemetry trace export.

Suggestions are welcome as GitHub issues.

Found this useful?

If OpenJev saved you a prompt-and-parse hack, a star helps the next person wrestling with unreliable LLM output find it. If you know someone building agents or routing/triage logic who'd rather get a typed answer back than parse prose, a share or a mention helps more than the star — no worries if not, just offering since it's free either way.

Contributing

Contributions are welcome: bug reports, datasets for the leaderboard, engine improvements, SDKs for more languages. Start with CONTRIBUTING.md; this project follows the Code of Conduct. Please report security issues privately as described in SECURITY.md, not in public issues.