daf-jev

Modular, composable Python client and decision toolkit for the TypeSafe Jev (System One) API. One HTTP endpoint, three question primitives, and a set of pure-logic composition patterns built on top of the answers — plus a concurrent batch evaluation harness, an MCP server, a figure registry, and a reproducible manuscript pipeline. An explicit benchmark backend interface adds local priors/classifiers, local model servers and OpenRouter decision candidates, with prepared datasets, durable attempt receipts and separate quality/cost evidence.

Contents: What it provides · Architecture at a glance · How a decision flows · Quickstart · Examples · Evaluating a corpus · Usage accounting and resilience · Decision-point decider · Graphical models · Jev to RxInfer.jl pipeline · CLI · MCP server · Configuration · Providers · Figures and manuscript · Decision-model benchmarks · Tests and benchmarks · Map · Documentation

What it provides

• Primitives — noul (yes/no), choice (pick an option from a probability distribution), score (rated on ordered levels). Build questions with noul() / choice() / score() and group them in a QuestionSet; batch questions within the selected backend's capacity.

• Client — JevClient / AsyncJevClient wrapping POST https://api.typesafe.ai/v1/systemone, with retries (429/529, exponential backoff, Retry-After), typed error mapping, and a models() listing. Retry policy, default timeout, and default model resolve from the environment (see Configuration); every ask / models call also accepts a per-call timeout override and extra request_headers (merged over the defaults for that call only; models() retries per the same policy as ask).

• Providers — one wire contract, several backends: hosted jev (TypeSafe), self-hosted jeff / kev / localjev / openthai-systemone, and the openrouter proxy. A global --provider CLI flag, a keyless providers registry listing, and for_provider() / open_client(provider, ...) constructors dispatch across them (see Providers).

• Composition patterns — pure functions over answers: composite_score (probability-weighted expected value over score levels), confidence_gate (auto-escalate low-confidence answers), route / pick (intent routing by choice). route, composite_score, and confidence_gate are exported from the package root; pick and tiered_gate live only in daf_jev.compose.

• Evaluation — Evaluator runs a fixed question set over many states concurrently (thread pool for JevClient, asyncio semaphore for AsyncJevClient) without aborting the batch: per-state failures are captured in EvaluationRecord.error. summary() aggregates per-question means plus batch-level latency mean/p95 and token usage; to_json() serializes records.

• Usage accounting — UsageLedger (thread-safe) accumulates request counts and token totals across any loop of ask calls; it accepts Usage objects, full responses, or None for error paths, and snapshot().to_dict() is JSON-safe.

• Resilience — an opt-in, composable CircuitBreaker: after failure_threshold consecutive failures it fails fast for cooldown_seconds, then admits a single recovery probe. It wraps any callable, never sleeps, and takes an injectable clock; it complements the per-request retry policy.

• Decider — the decision-point loop as one reusable class: observe a state, compose a batched ask, gate the answers, and fail open to a deterministic fallback, with a call/token budget, a per-decision cache, a consecutive-failure latch, and JSON-safe event receipts.

• Graphical models — Jev as a factor source for discrete Bayes nets: elicit_cpts elicits every CPT of a network in one batched ask, propose_structure proposes the topology via pairwise choices, and BayesNet runs exact local inference (pure-Python variable elimination) plus MPE, ancestral sampling, and what-if scenario sweeps; to_json() emits the dafjev.bayesnet/1 GraphSpec interchange for GNN / RxInfer.jl / GTSAM-style engines (see Graphical models).

• Calibration — pure reliability statistics in daf_jev.calibration (bucket_index, reliability_table, expected_calibration_error, brier_score) over (confidence, correct) pairs, plus a live calibration benchmark (benchmarks/bench_calibration.py).

• Model jaggedness — pure uniformity statistics in daf_jev.jaggedness (uniform_chi2, uniform_deviation, runs_test_z, max_streak, position_slope, noul_choice_delta) over repeated stochastic prompts (coin, d6), plus a live multi-provider jaggedness benchmark (benchmarks/bench_jaggedness.py).

• Figures & manuscript — a matplotlib registry with eight core figures, six selected empirical figures and an optional native capability figure, plus figure_registry.json and a {{TOKEN}} variable pipeline that keep the 10-section manuscript in manuscript/ free of hardcoded results.

• MCP server — daf-jev serve exposes the toolkit as nine MCP tools (jev_ask, jev_evaluate, jev_models, jev_composite_score, jev_confidence_gate, jev_tiered_gate, jev_docs_verify, jev_posteriors_load, jev_reask_plan) plus a jev://docs/snapshot resource, over stdio (see MCP server).

Dependencies: Python >= 3.10, httpx, pyyaml, and tomli on Python 3.10 (plus matplotlib + pillow for figures). Managed with uv.

Related MCP server: TypeSafe MCP

Architecture at a glance

Every box maps to a file — follow it into the Map for source and contract links. The per-module contract lives in Package layout in docs/ARCHITECTURE.md.

flowchart TB
    subgraph wire["wire layer — one endpoint, typed questions"]
        direction LR
        PRV["primitives.py<br/>noul · choice · score"]
        PR["providers.py<br/>registry · dispatch"]
        CF["config.py<br/>env → settings"]
        CL["client.py<br/>JevClient / AsyncJevClient"]
        SUP["_types · _http · _retry · _errors<br/>wire plumbing"]
        PR --> CF
        CF --> CL
        PR --> CL
        SUP --> CL
    end
    subgraph pure["composition and orchestration — provider-agnostic"]
        direction LR
        CP["compose.py<br/>composite_score · gates · route"]
        EV["evaluate.py<br/>Evaluator"]
        DC["decider.py<br/>Decider · Budget"]
        CB["calibration · resilience · ledger"]
    end
    subgraph gfx["graphical models — Jev as factor source"]
        direction LR
        GE["graphical_elicitation.py<br/>propose_structure · elicit_cpts"]
        GA["graphical.py<br/>BayesNet · exact VE"]
        GV["graphical_viz.py<br/>mermaid + PNG"]
    end
    subgraph surf["surfaces"]
        direction LR
        CI["cli.py — daf-jev"]
        MC["mcp_server.py — serve"]
        QU["questions.py · docs_verify.py<br/>shared CLI/MCP helpers"]
        EX["examples/ · skills/"]
    end
    PRV --> CL
    CL --> CP
    CL --> EV
    EV --> DC
    CL --> GE
    GE --> GA
    GA --> GV
    CI --> CL
    MC --> CL
    EX --> CL

    classDef wire fill:#0e7490,color:#fff;
    classDef pure fill:#4c1d95,color:#fff;
    classDef gfx fill:#166534,color:#fff;
    classDef surf fill:#9a3412,color:#fff;
    class PR,CF,CL,SUP,PRV wire;
    class CP,EV,DC,CB pure;
    class GE,GA,GV gfx;
    class CI,MC,QU,EX surf;

The core uses httpx and pyyaml, with tomli for TOML parsing on Python 3.10; Python 3.11 and newer use standard-library tomllib. matplotlib is optional (the figures extra). Inference in graphical.py is pure-stdlib variable elimination — no numpy, no network at query time.

How a decision flows

flowchart LR
    S["state<br/>(text / JSON)"] --> Q["typed questions<br/>noul · choice · score"]
    Q --> C["JevClient / AsyncJevClient<br/>POST /v1/systemone"]
    C --> A["typed answers<br/>probability · distribution · rubric"]
    A --> P["compose layer<br/>composite_score · confidence_gate · route/pick"]
    P --> G{"tiered_gate"}
    G -- "≥ high" --> AUTO["automate"]
    G -- "between" --> REV["human review"]
    G -- "< low" --> ESC["escalate"]
    A --> EV["Evaluator<br/>batch over many states"]
    EV --> CAL["calibration<br/>ECE · Brier · reliability"]

The batched call carries all questions at once. The retained September receipt measured up to 18.55× shorter wall time and a 4.22× total-token ratio for its specific task and model alias. It did not retain paired answers or reported USD charges, so these results establish timing and token measurements for that run. See Tests and benchmarks for the selected receipt.

Quickstart

uv sync
export JEV_API_KEY="sk-..."   # or TYPESAFE_API_KEY, or put it in .env

.env at the project root is auto-loaded (there is a .env.example to copy from; .env itself is gitignored).

from daf_jev import JevClient, choice, noul, score

with JevClient() as client:  # api_key resolved from env / .env
    resp = client.ask(
        "Customer message: I was charged twice this month and nobody has responded.",
        {
            "billing": noul("Is this about a billing problem?"),
            "tone": choice("What is the tone?", {"calm": None, "frustrated": "annoyed but civil"}),
            "severity": score("How severe?", ["minor", "noticeable", "blocking"]),
        },
    )

print(resp.nouls["billing"].noul)          # 0.0 (no) .. 1.0 (yes)
print(resp.choices["tone"].choice, resp.choices["tone"].confidence)
print(resp.scores["severity"].score)       # probability-weighted, may be fractional

Composing decisions:

from daf_jev import confidence_gate, composite_score

value = composite_score(resp.scores["severity"])   # expected value over level indices
verdict = confidence_gate(resp.choices["tone"], threshold=0.6, below="review")

Examples

Thirteen runnable scripts live in examples/ (walkthrough per script in examples/README.md). Each resolves the API key from the environment or .env and — when no key is found — prints SKIP: JEV_API_KEY not set and exits 0. The provider example uses an injected transport and makes no network call:

python examples/quickstart.py         # one mixed ask call; answers, usage, request id
python examples/triage_router.py      # tiered_gate + route over one choice answer
python examples/composite_scoring.py  # composite_score + confidence_gate
python examples/evaluate_corpus.py    # Evaluator over an inline four-state corpus
python examples/gated_fallback.py     # heuristic-first: model called only when it adds value
python examples/decider_loop.py       # decision-point loop: gate, budget, fail-open fallback
python examples/providers_example.py  # provider registry + dispatch; injected transport, no network
python examples/asia_bayes.py         # Bayes net from Jev factors: CPT elicitation, structure proposal, posterior walkthrough
python examples/decider_resilience.py # decision loop with a circuit breaker
python examples/evaluate_async.py     # evaluation from an existing async loop
python examples/calibration_walkthrough.py # calibration statistics walkthrough
python examples/retry_policies.py     # explicit retry policies
python examples/reask_policy.py       # posterior plan wired into a provider ask

Model-backed walkthroughs take --model NAME (default: provider-resolved — for the default jev provider: JEV_MODEL, then TYPESAFE_DEFAULT_MODEL, then jev-latest); asia_bayes.py also takes --provider KEY (default jev); evaluate_corpus.py also takes --concurrency N (default 2).

Evaluating a corpus

Run a fixed question set over many states, concurrently, with per-state error capture (evaluate never aborts the batch on one bad state).

CLI — --questions-file is a YAML mapping of id to a spec string or a native question mapping; --states-file is one state per line (blank lines skipped) or a JSON array of strings:

uv run daf-jev evaluate \
  --questions-file questions.yaml \
  --states-file states.txt \
  --concurrency 8 \
  --include-records

Python — pass a sync or async client; bare str states get state_0000-style ids. An AsyncJevClient is single-use through evaluate(): the Evaluator closes the async session when the batch completes (its keep-alive connections are bound to the private event loop).

Inside a running event loop, await evaluator.evaluate_async([("id", state), ...]) is the public async entry point: the same batch on your own loop (TypeError unless the client is an AsyncJevClient).

from daf_jev import Evaluator, JevClient, QuestionSet, noul, score

questions = QuestionSet().add(
    "billing", noul("Is this about a billing problem?")
).add(
    "severity", score("How severe?", ["minor", "noticeable", "blocking"]),
)

with JevClient() as client:
    evaluator = Evaluator(client, questions, concurrency=8)
    records = evaluator.evaluate(["state one text", "state two text", ...])

summary = evaluator.summary(records)      # per-question means + batch latency mean/p95 + usage
print(evaluator.to_json(records))         # per-state records incl. errors/latency

Usage accounting and resilience

Long-running consumers need receipts and failure isolation beyond the per-request retry policy. Both are small, opt-in, client-side helpers:

from daf_jev import UsageLedger

ledger = UsageLedger()
for state in states:
    try:
        response = client.ask(state, questions)
    except TypeSafeError:
        response = None               # error paths produce no usage
    ledger.record(response)
print(ledger.snapshot().to_dict())    # {"requests": .., "input_tokens": .., ...}

UsageLedger accumulates request counts and token totals across any loop of ask calls; Evaluator.summary() remains the aggregator for batch evaluation runs. reset() returns the pre-reset totals and zeroes the ledger.

from daf_jev import CircuitBreaker, CircuitOpenError

breaker = CircuitBreaker(failure_threshold=5, cooldown_seconds=30.0)
try:
    response = breaker.call(client.ask, state, questions)
except CircuitOpenError as exc:
    ...                               # fail fast while the circuit is open

await breaker.call_async(async_callable, ...) awaits async completion before recording success; cancellation and any BaseException record failure and re-raise, including recovery probes.

CircuitBreaker wraps a synchronous callable: failure_threshold consecutive failures open the circuit for cooldown_seconds, after which a single probe is admitted. It never sleeps — wait out the cooldown in your own loop (exc.remaining_seconds reports what is left) — and composes with the per-request retry policy.

Decision-point decider

Decider distills the recurring decision-point loop — observe a state, compose a batched ask, gate the answers, fail open to a deterministic fallback — into one reusable class over injected I/O:

flowchart LR
    S["state"] --> D["Decider"]
    D --> Q["questions(state)"]
    Q --> A["ask (once, behind<br/>breaker + budget)"]
    A --> G["gate(answers)"]
    G -->|accepted| M["map_answers -> action"]
    G -->|rejected| F["fallback(state)"]
    A -->|error / no key / budget| F
    F --> OUT["action"]
    M --> OUT

from daf_jev import Budget, ConfidenceGate, Decider, choice

def fallback(state: str) -> str:
    return "hold"                      # deterministic floor action

decider = Decider(
    client,                             # any object with .ask(state, questions, timeout=)
    render_state=str,
    questions=lambda state: {
        "route": choice("Pick an action.", {"act": None, "hold": "wait"})
    },
    map_answers=lambda state, resp: resp.choices["route"].choice,
    fallback=fallback,
    gate=ConfidenceGate("route", threshold=0.7),
    budget=Budget(max_calls=100),
    on_event=lambda event: print(event.to_dict()),   # JSON-safe receipt
)
action = decider.decide("state text")   # never raises

decide() never raises: every failure — no API key, client construction, budget exhaustion, compose/ask/gate/mapping errors, an open circuit, or too many consecutive failures — falls back to the floor action and is classified into a closed reason taxonomy on the emitted DecisionEvent (not_asked, no_key, client_error, latched, budget, breaker, compose_error, ask_error, gate, mapping_error). The default client is single-attempt per ask (JevClient(env=env, retry=RetryPolicy( max_attempts=1))), so worst-case blocking is one timeout, never timeout x retries; consumers wanting retries pass their own client_factory. With a ConfidenceGate configured, decider.calibration_pairs() accumulates (declared confidence, gate-accepted) pairs — a self-consistency proxy to feed into daf_jev.calibration.

Graphical models

Jev doubles as a factor source for graphical models: zero-shot probabilistic factors that a Bayes-net engine turns into reusable inference (per Frank Dellaert's Jev+GTSAM experiments). Two batched requests cover a whole discrete network — one asks for every CPT at once (elicit_cpts: an 8-variable binary net is 18 rows in a single call), the other proposes the topology itself via pairwise three-way choices (propose_structure: a->b / b->a / no-edge over all n(n-1)/2 pairs, scored into a DAG by log(edge/no-edge) gain and an edge penalty). Inference is local and exact: BayesNet.query / .posterior run pure-Python variable elimination over the elicited factors — stdlib only, no numpy, no API calls at query time.

Jev appears at three seams:

• Upstream — the net's structure and CPTs are elicited from a Jev provider (any of them; the provider is chosen where the client is built).

• Within — the factors themselves are Jev probabilities.

• Downstream — evidence queries run locally and exactly; a posterior can drive a follow-up ask (re-asking) in the same workflow.

from daf_jev import Variable, elicit_cpts, propose_structure

variables = [...]                        # e.g. the 8 Asia variables
proposal = propose_structure(variables, client=client)      # edges only
net = elicit_cpts(variables, proposal.edges, client=client) # fills every CPT
p_tub = net.query("tub", evidence={"xray": "true"})[1]      # exact marginal
spec = net.to_json()                     # GraphSpec interchange

to_json() emits GraphSpec ("format": "dafjev.bayesnet/1"), the interchange between this client and downstream graphical-model engines: the GNN bridge consumes/produces it and the RxInfer.jl example reads it; GTSAM-style engines map the same elicited factors onto their own discrete types. The format string is a cross-repo contract — it changes only together with the consuming bridges in the same wave (see AGENTS.md).

End-to-end walkthrough: examples/asia_bayes.py — a runnable example (keyless skip; --provider / --model flags) builds the Asia variables, runs both batched requests, walks the posterior trajectory from the experiment (asia=false, then +xray=true, then +dysp=true, printing the tub/lung/bronc marginals at each step), and writes asia_graphspec.json.

The same surfaces ship with visualization: to_mermaid(net) renders a zero-dependency mermaid graph TD diagram of the DAG (one labeled node per variable, one arrow per edge, deterministic order), and the two matplotlib plotters write artifacts — plot_network(net, path) (layered layout: topological generations top-to-bottom, deterministic coordinates) and plot_posterior_trajectory(net, query_keys, steps, path, labels=...) (grouped bars of P(state=true) per query variable across cumulative evidence steps, where "true" is the last state of each variable's states tuple and values come from net.posterior). animate_posterior(net, query_keys, evidence_steps, path, *, labels=None, fps=1, dpi=110) and animate_network(net, evidence_steps, path, *, fps=1) — both in graphical_animation.py — render the same walkthrough as animated GIFs: grouped P(true) bars growing per evidence step (same P(true)-is-last-state convention) and the network with node fills shaded by P(true) at each step (coolwarm 0..1); PillowWriter writes the frames (pillow ships in the figures extra). Matplotlib imports lazily inside the plotters and animators; install the figures extra with uv sync --extra figures.

The Dellaert-style Asia experiment runs end-to-end through one thin orchestrator:

uv run python scripts/bayes_experiment.py [--provider KEY] [--model NAME] \
    [--edge-penalty FLOAT] [--propose-structure] [--animate] [--out-dir PATH]

It prints the mermaid diagram, elicits the Asia CPTs in one batched ask, walks the posterior trajectory (priors -> asia=false -> +xray=true -> +dysp=true) as a P(true) table for tub/lung/bronc, and writes five artifacts into --out-dir (default output/experiments/asia): asia_graphspec.json, network.png, posterior_trajectory.png, mermaid.txt, and receipts.json (see Live receipts). With --animate the runner additionally writes posterior_animation.gif and network_animation.gif into --out-dir and records them under animations in receipts.json. Keyless runs print SKIP: JEV_API_KEY not set and exit 0 before any network use.

Live receipts

The committed artifacts come from a real end-to-end run (provider openrouter, model jev-latest, two batched asks: topology proposal plus the full 8-CPT elicitation). Exact posteriors under cumulative evidence — P(tub) climbs 0.120 → 0.371 → 0.434 as x-ray and dyspnea arrive:

Committed run artifacts (reproduce with the command above; keyless runs skip cleanly):

• asia_graphspec.json — the elicited net as GraphSpec dafjev.bayesnet/1

• network.png — layered DAG layout

• posterior_trajectory.png — the P(true) bars behind the table

• mermaid.txt — the to_mermaid render of the DAG

• receipts.json — provider, model, proposed edges, elicited CPTs, and the trajectory itself

Jev to RxInfer.jl pipeline

GraphSpec (dafjev.bayesnet/1) is the interchange between daf-jev and the GNN bridge — PR #165 on branch feat/rxinfer-bridge. daf-jev elicits structure and factors, the GNN bridge emits a RxInfer.jl @model, Julia runs the inference, and the marginals come back to daf-jev for calibration and re-asking:

flowchart LR
    subgraph jev["daf-jev (this repo)"]
        A["propose_structure + elicit_cpts<br/>2 batched asks"]
        G["calibration · re-ask<br/>decider.py · evaluate.py"]
    end
    subgraph art["output/experiments/asia"]
        C["asia_graphspec.json<br/>dafjev.bayesnet/1"]
    end
    subgraph gnn["GNN repo (PR #165)"]
        D["rxinfer_bridge.py<br/>GraphSpec → @model"]
        E["examples/rxinfer/<br/>asia_model.jl"]
    end
    subgraph jl["Julia (RxInfer.jl)"]
        F["marginals<br/>evidence updates stay local"]
    end
    A --> C
    C --> D
    D --> E
    E --> F
    F --> G

Runnable end to end — the live receipts above are step 1 with --provider openrouter:

# 1. daf-jev (this repo) — elicit the Asia net, write the five artifacts
uv run python scripts/bayes_experiment.py --provider openrouter --propose-structure

# 2. GNN repo — clone, check out the bridge branch, emit the Julia model
git clone https://github.com/ActiveInferenceInstitute/Generalized_Notation_Notation
cd Generalized_Notation_Notation && git checkout feat/rxinfer-bridge
python -m gnn.rxinfer_bridge emit /path/to/daf-jev/output/experiments/asia/asia_graphspec.json

# 3. Julia — run the generated model over the same GraphSpec
julia --project=examples/rxinfer examples/rxinfer/asia_model.jl \
      examples/rxinfer/asia_graphspec.json

Cross-repo note: the bridge (src/gnn/rxinfer_bridge.py), its 39 tests, and the GNN-side walkthrough (examples/rxinfer/README.md, with the verified capability/gap matrix) live in the GNN repository — a relative link across repos does not resolve on GitHub, so PR #165 is the entry point (its Files list shows all five bridge files). Verified capability status: single-parent nets run end-to-end with correct posteriors; multi-parent DiscreteTransition nodes hit a ReactiveMP structured-rule limitation (documented in the PR).

Downstream is plain Python and needs no bridge: feed the marginals back into Decider / Evaluator or the calibration functions, re-asking Jev with the new evidence when a posterior warrants it.

CLI

uv run daf-jev ask \
  --state "I was charged twice this month." \
  --question billing=noul:Is this about a billing problem? \
  --question tone=choice:What is the tone?:calm=,angry=hostile \
  --question severity=score:How severe?:minor,noticeable,blocking \
  --pretty

uv run daf-jev models                       # list available models
uv run daf-jev models --pick latest         # pick one (latest|first|last)
uv run daf-jev models --pick latest --contains jev

uv run daf-jev providers                    # provider registry listing (keyless, no network)
uv run daf-jev --provider kev models        # route any command through a registered provider

uv run daf-jev evaluate \
  --questions-file questions.yaml --states-file states.txt \
  --concurrency 8 --include-records

uv run daf-jev docs-verify     # re-hash docs/reference/ against MANIFEST.json; reports missing/drifted/added

All commands print JSON to stdout (errors go to stderr as JSON); exit 0 on success, 2 on usage error, 1 on runtime error.

MCP server

daf-jev serve runs the toolkit as a Model Context Protocol (MCP) server over stdio — the default and only supported transport. The server needs the official mcp SDK, shipped in the optional mcp dependency group. Every surface below calls the same core; there is no second implementation:

flowchart TB
    subgraph CORE["daf-jev core"]
        CLI["CLI<br/>ask · evaluate · models · providers · docs-verify"]
        MCP["MCP server (stdio)<br/>jev_ask · jev_evaluate · jev_models<br/>jev_composite_score · jev_confidence_gate<br/>jev_tiered_gate · jev_docs_verify"]
        SKILL["agent skill<br/>skills/daf-jev/SKILL.md"]
        EX["examples/<br/>8 runnable scripts"]
    end
    CLI --> K["JevClient / compose / calibration"]
    MCP --> K
    SKILL --> K
    EX --> K
    K --> API["TypeSafe Jev API<br/>POST /v1/systemone"]

uv sync --extra mcp
uv run daf-jev serve          # stdio; --transport stdio is the only choice

Tools (each returns JSON-safe values; keys resolve per call from env or .env, and a missing API key surfaces as a tool error):

Schema note: jev_evaluate accepts string, JSON-object, or JSON-array states (str | dict | list); the CLI evaluate --states-file JSON mode accepts strings only. jev_docs_verify returns {manifest, pages, missing, drifted, added, ok} — added lists unlisted .md files on disk.

Point any MCP client at the server with a stdio config, e.g.:

{
  "mcpServers": {
    "daf-jev": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/daf-jev", "daf-jev", "serve"],
      "env": { "JEV_API_KEY": "sk-..." }
    }
  }
}

env may be omitted when a .env file is present in the server's working directory; credentials are never returned in tool output.

Configuration

Everything resolves from the environment (injected env mapping > process env

.env file); unset or invalid values fall back to the defaults below.

Naming note (verified): the official TypeSafe SDK convention is TYPESAFE_API_KEY / TYPESAFE_BASE_URL / TYPESAFE_DEFAULT_MODEL; daf-jev keeps the JEV_* names as its own primaries with the TYPESAFE_* names as fallbacks. Every provider's key and base-URL env vars additionally fall back to TYPESAFE_API_KEY / TYPESAFE_BASE_URL last.

Per-field: a bad value keeps only that field's default. Per-call timeout= and request_headers= on ask() / models() win over all of the above for that call.

Providers

One wire contract — POST /v1/systemone — many backends. daf-jev ships a provider registry (src/daf_jev/providers.py) that parameterizes config resolution, client construction, and CLI/MCP dispatch. Composition and calibration are pure; evaluator and decider orchestrate injected I/O. All are provider-agnostic. daf-jev providers prints the registry as JSON (keyless, no network):

The jev provider additionally falls back to TYPESAFE_DEFAULT_MODEL for its model (see Configuration for the naming convention).

Selecting a provider:

uv run daf-jev providers                 # registry listing, keyless, exit 0
uv run daf-jev --provider kev models     # any command; the flag precedes the subcommand

The global --provider flag selects the backend; precedence: --provider

DAF_JEV_PROVIDER env var > jev. Invalid keys are usage errors (exit 2). The MCP tools accept the same choice through an optional provider argument (unknown keys return a JSON-safe error listing the available providers).

Python selection — per-provider settings resolution plus for_provider / open_client constructors (explicit arguments always win over the provider's env resolution):

from daf_jev import open_client, load_settings

settings = load_settings(provider="kev")   # KEV_API_KEY / KEV_BASE_URL / KEV_MODEL
with open_client("kev", api_key=settings.api_key) as client:
    response = client.ask(state, questions)

Extending: a third-party backend registers at runtime with register_provider(ProviderSpec(key="acme", ...)) and becomes first-class everywhere (CLI, MCP, clients, settings). For a backend needing custom HTTP behavior, inject a Transport / AsyncTransport implementation — the seam between daf-jev's wire layer and anything upstream or downstream. Provider keys are stable API: adding a built-in updates this section, the architecture contract, and the agent skill in the same commit.

For new OpenRouter experiments use the explicit DecisionBackend profiles, not the legacy registry defaults. OpenRouter documents native Jev at POST https://openrouter.ai/api/alpha/decisions with canonical IDs such as typesafe/jev-1.13; Tev candidates use chat completions and return letters. Native endpoint, Jev integration guide (reviewed 2026-10-07). The provider guide covers output semantics, pinned local setup evidence, discovery, limits and cost receipts.

Decision-model benchmarks

The benchmark commands separate data preparation, public catalog discovery, immutable planning, actual inference, resume and offline reporting. They compare complete task vocabularies across native probability arms and training-only priors/classifiers. Explicit chat adapters support separately named comparator experiments; they are excluded from the current native hosted cohort. Reports retain failures, unsupported and unattempted arms, unresolved attempts and unknown cost. Dataset configuration IDs, inventory indices and prepared input hashes distinguish cohorts sharing a family name. Reports keep wholly unavailable arms and separate coverage among attempted outcomes from coverage over all planned decisions.

The full-study guide documents five-fold validation, separate final refits and leakage sensitivity, nine portable CPU recipes and native/hosted admission. The recorded CPU comparison contains 68,894 completed cells (68,870 quality decisions and 24 probes) and 24,152 unsupported cells across nine jobs. Full native and hosted studies remain unfinished; their proposal and continuation limits are stated alongside the results. The separate first hosted probe report records an HTTP 404 and unknown billing under the shared USD 25 allocation. It supplies no hosted quality result. A separately reviewed bounded-UNKNOWN exception can permit continuation while retaining that unknown charge and its upper bound; it does not rewrite the historical receipt or create another budget. New hosted execution requires an existing allocation bound into its frozen manifest. The shared-allocation guide explains explicit initialization, legacy imports, cross-run reservations and recovery. Older unbound plans remain reportable and cannot execute hosted requests.

uv sync --extra benchmark
mkdir -p .benchmarks/data .benchmarks/runs
uv run daf-jev benchmark dataset synthetic --kind binary --samples 150 \
  --seed 20261007 --output .benchmarks/data/binary-v3.json
uv run daf-jev benchmark dataset synthetic --kind categorical --samples 150 \
  --seed 20261007 --output .benchmarks/data/categorical-v3.json
uv run daf-jev benchmark dataset synthetic --kind ordinal --samples 150 \
  --seed 20261007 --output .benchmarks/data/ordinal-v3.json
uv run daf-jev benchmark dataset synthetic --kind bayes --samples 150 \
  --seed 20261007 --output .benchmarks/data/bayes-v3.json
uv run daf-jev benchmark plan --config benchmarks/configs/synthetic.yaml \
  --out-dir .benchmarks/runs
uv run daf-jev benchmark run .benchmarks/runs/RUN_UUID
uv run daf-jev benchmark report .benchmarks/runs/RUN_UUID \
  --output .benchmarks/report.json

Replace RUN_UUID with the returned run directory. The offline profile tests the labeled harness; it is not evidence of local language-model inference. The four prepared pools supply enough independent ordinary test groups for the global 100-example timing pack. The committed synthetic.yaml freezes uniform and exact rule controls for all four tasks. Save new prepared files rather than replacing retained inputs from earlier runs. Exact rules use state fields only; hard labels/levels have no fabricated probabilities or confidence, while the Bayes reference reports its analytical distribution. Local/hosted model claims require exact execution receipts. Catalog listings are declared candidates, generated labels have no invented probability vectors, confidence concentration is not P(correct), and unknown monetary cost is not free. A refreshed discovery snapshot does not extend an earlier frozen executable cohort or establish inference access for newly listed models.

The requested repeated-timing protocol uses 100 shared test examples overall and five additional rounds. The current default timing_sampling_scope: cohort freezes an exact target-free global ID pack; insufficient input groups fail before planning. Matched cyclic Choice controls remain a separate quality cohort. The retained expanded Mac study used a legacy per-dataset selector and therefore an expanded 368-ID timing cohort; it stopped at a clean boundary before later rounds. Preserve that evidence and its future unattempted rows separately. Correcting scope or ordered prompt presentation requires a new accepted source and manifest, without resetting the cumulative profile allowance. See the timing scope and compatibility rules. Probability provenance identifies where values came from; it does not establish posterior meaning or calibration. The semantic reference distinguishes normalized compatibility scores, class estimates, training frequencies, analytical conditionals and generated judgments.

Read the protocol and commands, synthetic and real dataset contracts, provider abstraction, local serving and Mac execution, reproducibility gates and reproduced methods review. The existing four scripts in benchmarks/ preserve their historical receipt lane.

Figures and manuscript

The repo renders its own paper: 10 manuscript sections under manuscript/, with every measured number injected as a {{TOKEN}} placeholder — nothing is hardcoded in the prose.

uv sync --extra dev --extra figures --extra benchmark
uv run python scripts/generate_figures.py --include-study  # selected 13-figure manuscript registry
uv run python scripts/generate_figures.py --only batching   # single figure by name

Figures architecture, primitives, and confidence are drawn from code; batching, latency, calibration, and graphical_abstract read the reviewed benchmark evidence selected for the manuscript. Source identities must match the variable pipeline; an independently chosen newest file is not a coherent cohort.

uv run python scripts/capture_verification.py --out-dir .benchmarks/verification-new
# Deliberately select verification.json path + SHA-256 in manuscript/evidence.json.
uv run python scripts/z_generate_manuscript_variables.py
# tokens -> output/data/manuscript_variables.json
uv run python scripts/render_pdf.py --output output/pdf/reproduction-new.pdf \
    --artifacts-dir .benchmarks/reproduction-build-new

The capture runs unit tests with coverage/JUnit and collects live tests without executing them. Its retained, source/test/script/config-bound selection permits offline token regeneration without raw coverage or a new pytest collection. Malformed/stale explicit verification fails even in draft mode. It does not select itself or publish anything; keep the historical benchmark selection intact. See retained verification for the record and selection format. The legacy unselected raw-coverage path remains available; retain its fresh raw data through token custody. A layout-only rerender can reuse the unchanged validated saved map.

The in-repository renderer consumes manuscript sources and the generated token map, uses Pandoc/TeX, and checks unresolved citations, references, images and overfull vertical boxes. Retain intermediate TeX/log inputs with --artifacts-dir and inspect all headings/prose and pages; marker/bounds checks alone can miss clipped content. The reproducible build timestamp is separate from render time. It writes the selected fresh PDF path and preserves existing outputs; --install explicitly refreshes the root PDF. The optional external template integration has additional path and provenance requirements and is not needed for standalone rendering. See reproducibility for evidence selection and final checks.

Tests and benchmarks

uv sync --extra dev --extra figures --extra benchmark   # matches CI; optional comparators/resources included
uv run pytest tests/unit --cov=src          # unit tests — counts live in output/data/manuscript_variables.json (refresh: uv run python scripts/z_generate_manuscript_variables.py); coverage gate >= 90%
JEV_API_KEY=... uv run pytest tests/live    # live tests against the real API

Unit tests need no key: they run against a real local HTTP stub server (tests/conftest.py). Live tests and benchmarks hit the real API and are skipped with a SKIP: message when JEV_API_KEY is absent.

uv run python benchmarks/bench_batching.py --runs 3   # 1 call with N questions vs N calls
uv run python benchmarks/bench_patterns.py --runs 10  # composite-score / routing latency

Historical receipts (2026-09-16, output/benchmarks/): batching is 4.0x–18.6x faster (N=5→20) and uses 2.8x–4.2x fewer total tokens; decision-pattern pipelines run at ~0.13 s p50. These receipts do not establish dollar savings, paired answer equivalence, isolated composition overhead or a current cross-model result.

Calibration benchmark

benchmarks/bench_calibration.py measures how well the live model's reported confidence tracks its behavior, plus noul answer stability:

JEV_API_KEY=... uv run python benchmarks/bench_calibration.py
# flags: --states N (default 6), --repeats N (default 5), --model NAME

It repeats one three-option classification question per state and treats agreement with the modal (majority) choice across repeats as a self-consistency correctness proxy — not ground-truth accuracy — so the resulting error figures quantify confidence-vs-self-consistency, not confidence-vs-correctness. The (confidence, correct) pairs feed the pure daf_jev.calibration functions; a noul question repeated the same way yields a mean pairwise |Δnoul| stability metric. Results land in output/benchmarks/calibration_<YYYYMMDD>.json (latest recorded: historical 2026-09-16, jev-latest, 6 states x 5 repeats — ECE 0.0730, Brier 0.0252, mean pairwise noul gap 0.0050). Without an API key (env or project .env) it prints SKIP: JEV_API_KEY not set and exits 0; a failing call drops that state's repeats into n_errors instead of aborting the batch.

Jaggedness benchmark

benchmarks/bench_jaggedness.py measures how far repeated answers to stochastic prompts (coin flips, six-sided rolls) deviate from their stated uniform distributions:

uv run python benchmarks/bench_jaggedness.py --providers jeff,kev
# flags: --providers 'jeff,kev,jev', --fixtures, --repeats 50,
# --concurrent 32, --timeout, --model

Per provider and fixture it reports the uniformity chi-square (with p-value) and total variation, chosen-label degeneracy over identical repeats (deterministic servers make the choice a point mass, which is itself the finding), option-order rotation sensitivity (position bias, argmax flips), concurrent-batch wobble, and the noul-vs-choice cross-instrument delta on the coin fixture. Keys resolve per provider (JEV_API_KEY / JEFF_API_KEY / KEV_API_KEY; registry defaults localhost:8000 jeff, localhost:8009 kev — see docs/models.md §10); a provider without its key prints SKIP[<provider>] and the run continues. Results land in output/benchmarks/jaggedness_<YYYYMMDD>.json. Caveat: the numbers quantify stated-distribution deviation, NOT accuracy; local self-hosted servers (jeff: normalized sigmoid compatibility scores; kev: discriminative pointer head) behave differently from the hosted jev endpoint.

Package and releases

The native-only continuation is documented in the hosted study guide. It uses complete decision vocabularies, TypeSafe-only Jev routing and the same shared allocation. New successful capability observations keep a separate selected journal cut; the larger quality run remains in progress. The retained v0.7.1 release and its PDF continue to describe their original evidence.

The v0.7.1 source adds shared hosted allocation admission and the scoped Mac/Python 3.10 benchmark-runtime repair to the modular decision-evaluation release: typed native and constrained-chat adapters, frozen datasets and policies, exact graphical inference, explicit execution coverage and cost accounting, and an evidence-selected manuscript. It preserves the native client API and separates completed CPU studies from partial local observations and the failed hosted capability probe. Publication status is established by the remote tag and published Zenodo record; a local version number is not that confirmation.

Install the reviewed wheel or a tagged source checkout. Core dependencies are small; optional benchmark, figures and mcp extras enable their respective surfaces. The wheel includes the PEP 561 py.typed marker. Dataset/model weights, private run journals and local server runtimes are not bundled. Benchmark execution and manuscript reproduction require the documented source checkout and its selected inputs; installing the core wheel does not establish model readiness or ship the documentation snapshot.

• Concept DOI (all versions, stable): 10.5281/zenodo.22816187

• v0.7.1 published version (verified 2026-10-09): 23266335, version DOI 10.5281/zenodo.23266335. Its source, wheel, sdist, manuscript, selected evidence, release manifest and checksums are the seven verified GitHub v0.7.1 assets. New source changes described in the 0.7.2 changelog are separate from that archived tag.

• v0.6.0 historical version record (recorded 2026-09-23): 22921974, version DOI 10.5281/zenodo.22921974; earlier versions remain in the same concept family. identifies this historical archive. A new version DOI is added only after the corresponding version record is created and verified.

• Public repository: https://github.com/docxology/daf-jev

• Reviewed scholarship PDF: Native hosted capability evidence, v0.7.2 documents the updated source and its selected terminal capability evidence. Billing recovery and bibliography follow-up remains retained with its original source and selected verification evidence. The archived release PDF is Modular Decision Models, v0.7.1. The v0.7.0 GitHub release and its reviewed PDF remain retained. The prior scholarship edition remains retained separately. The historical root PDF remains a separate artifact; release rendering chooses a fresh output and retains the prior bytes.

• Release protocol and asset boundaries: docs/release.md.

• Machine-readable metadata: CITATION.cff and .zenodo.json.

The concept citation below identifies the software family. A version-specific citation must use the DOI confirmed for the published version:

@software{friedman2026dafjev,
  title   = {daf-jev: Modular Decision Models, Orchestration, and Reproducible Evaluation},
  author  = {Friedman, Daniel Ari},
  year    = {2026},
  doi     = {10.5281/zenodo.22816187},
  url     = {https://github.com/docxology/daf-jev},
  version = {0.7.2}
}

New releases are new version records in the same Zenodo concept. Preserve historical receipts and distinguish the software version from each observation's inference source, dataset and runtime. The selected empirical status reports remain descriptive; no release turns an unattempted study or unknown charge into a completed experiment.

Stack integration

daf-jev verification-status --root PATH inspects selected, source-bound unit execution and live-test collection without model calls or new test execution. daf-jev stack-member FILE consumes daf-stack's daf-stack-doctor roster JSON and audits this checkout against the member entry daf-stack declares (required files only; no gate, git, Sidecar or subprocess execution). See the integration contract for status/exit codes, the native offline gate, Sidecar evidence boundaries and remaining consumer work. DAF Dot carries the real-time voice protocol; DAF Operator spatializes information.

Map

Every module, example, script, and receipt — one click from prose to source to contract. Module contracts live in Package layout.

Source modules

Example scripts

Per-script walkthroughs: examples/README.md.

Scripts

Benchmarks

Methodology and the committed-receipts policy: benchmarks/README.md.

Skills, tests, docs, artifacts

• Agent skill: skills/daf-jev/SKILL.md (install notes in skills/README.md).

• Test suite: the shared stub server tests/conftest.py, dynamically collected unit modules under tests/unit/ — incl. test_graphical_methods.py and test_graphical_animation.py — 2 live tests in tests/live/test_live_api.py.

• Docs: contract docs/ARCHITECTURE.md, model reference docs/models.md, index docs/README.md, manifest-bound hashed snapshot under docs/reference/.

• Manuscript: 10 sections under manuscript/ + config.yaml; rendered PDF daf-jev_combined.pdf.

• Generated figures: output/figures/figure_registry.json (+ 7 PNGs alongside).

• Release metadata: CITATION.cff, .zenodo.json; dev environment template .env.example.

Documentation

• docs/ARCHITECTURE.md — the authoritative design contract (wire facts, module signatures, test and benchmark conventions).

• docs/models.md — sourced technical reference on System One models and Jev, with primary vs third-party claims flagged.

• docs/ — index, including the manifest-bound hashed snapshot of docs.typesafe.ai in docs/reference/.

• skills/daf-jev/SKILL.md — the agent skill for this toolkit (when-to-use, API surface, CLI, MCP server, pitfalls). To use it with an agent outside the repo, copy the whole skills/daf-jev/ directory into the agent's skills location — see skills/README.md.

• Map — deep links to every module, example, script, and receipt.