semantic-if

A semantic classifier for LLM agents and pipelines. Give it a state (text or JSON), one question and 2–16 mutually exclusive options ("which queue should handle this request?", "does the counterparty depend on public contracts?"). It returns the chosen option and a probability for every option, in 0.15–0.5 s per decision.

The idea is Jev, TypeSafe's closed, hosted model for semantic decisions: the question and its permitted answers are defined at call time, and what comes back is a typed answer with probabilities, not text. semantic-if is an open, self-hosted counterpart: an independent implementation of the direct method from SemIf, served by your own vLLM through LiteLLM. The model generates no text: one prefill and one decoding step, and the answer is read from the next-token logprobs of the option letters. There is nothing to parse, and uncertain cases show up in the probabilities. The spec the project was built against is in docs/spec.md.

Features

• the prompt is byte-identical to SemIf direct-options-v1 (checked by prompt_sha256), thinking is off;

• probabilities of all 2–16 options (a float64 softmax over the logprobs of the letter labels A…P); the service also returns confidence, margin and low_confidence;

• direct and shared modes; in shared, decisions about the same state reuse the vLLM prefix cache;

• a library (sync and async), the semantic-if CLI with JSONL in the SemIf format, and scripts for comparison, quality and speed;

• a service: REST API and MCP server in one process. It holds no key of its own; each caller sends their own LiteLLM key;

• rows are never dropped or truncated silently: every error is explicit. The key never reaches output, logs or metrics;

• the model is switched with -m alone: tokenizers at pinned revisions are kept in a registry.

Related MCP server: jev-mcp

How it works

1. The decision {state, question, options} is validated by the SemIf rules and rendered into its prompt (Qwen ChatML).

2. A local Hugging Face tokenizer checks the label tokens, the tokenization boundary and the prompt length. The prompt is never truncated.

3. One POST /v1/completions goes to LiteLLM: max_tokens=1, temperature=0, allowed_token_ids = the labels.

4. The letters of the options in use are taken from the raw next-token logprobs and normalized with a softmax.

5. Labels outside the top-k are fetched with a follow-up request. Retries happen only on network errors, 429 and 5xx.

Details are in docs/method.md.

Requirements

• Python 3.10+ (development uses 3.13 and uv);

• a LiteLLM proxy in front of a vLLM server with an OpenAI-compatible /v1/completions that accepts allowed_token_ids and logprobs;

• a model from the registry (currently Qwen: qwen-3-8-27b → Qwen/Qwen3.8-27B-FP8, qwen3.5-9b → Qwen/Qwen3.5-9B), served under that alias, or any other Qwen ChatML model with --tokenizer (docs/method.md).

Quick start

git clone https://github.com/rinat-amanbekov/semantic-if.git && cd semantic-if

The whole chain on one machine with an NVIDIA GPU (24 GB; 16 GB with FP8): semantic-if, LiteLLM and vLLM serving qwen3.5-9b, from compose.yaml (docs/deployment.md).

cp .env.example .env               # set LITELLM_MASTER_KEY (and LLM_API_KEY to the same value)
docker compose up -d --wait        # the first start downloads ~19 GB of weights
curl -s http://127.0.0.1:8000/semantic-if/health

The library and the CLI, against that stack or a LiteLLM of your own:

uv sync --extra test     # Python 3.13, venv in .venv; the test extra includes the service (REST + MCP)

They take their settings from environment variables or from the nearest .env (kept out of git):

LITELLM_BASE_URL=http://localhost:4000/v1
LLM_API_KEY=sk-...
LLM_MODEL=qwen-3-8-27b

CLI: JSONL in and out, one line per decision.

uv run semantic-if -i tests/data/decisions.jsonl -o runs/decisions.jsonl --force

Library:

from semantic_if import Decision, classify

row = {
    "id": "route-1",
    "state": "I can't log in after changing my password",
    "question": "Which queue should handle this request?",
    "options": [{"id": "access", "description": "Account access"}, {"id": "other", "description": "Other"}],
}
for result in classify([row], "qwen-3-8-27b"):
    if isinstance(result, Decision):
        print(result.choice, dict(zip(result.option_ids, result.probabilities)))
    else:  # RowError
        print("error:", result.kind, result.error)

Service: REST and MCP in one process, with Swagger UI at /semantic-if/docs. It reads only environment variables, and the caller's key comes with every request in Authorization: Bearer or x-litellm-api-key. The key must be issued by the LiteLLM instance the service calls (docs/service.md).

LITELLM_BASE_URL=http://localhost:4000/v1 SEMANTIC_IF_ROOT_PATH=/semantic-if uv run semantic-if-service
curl -s http://127.0.0.1:8000/semantic-if/v1/decide \
  -H "Authorization: Bearer $LITELLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"state": "I cannot log in after changing my password", "question": "Which queue should handle this request?",
       "options": [{"id": "access", "description": "Account access"}, {"id": "other", "description": "Other"}]}'

The MCP server (tools semantic_decision and semantic_decisions) connects to agents such as Claude Code and OpenCode, directly or through the LiteLLM MCP gateway; see docs/mcp.md. The Docker image, the Compose stack and the Helm chart are described in docs/deployment.md.

Documentation

Status

Layout

semantic_if/          library, CLI (cli.py), service (service.py, mcp_server.py), model registry (config.py)
scripts/              compare.py, evaluate.py, bench.py
tests/                unit tests, fake vLLM (fakes.py), live test (test_live.py), tests/data/decisions.jsonl
data/ru_custom.jsonl  our own labeled set in Russian (90 decisions)
docs/                 documentation
Dockerfile, docker/   service image
compose.yaml          local stack: semantic-if, LiteLLM, Postgres, vLLM (.env.example holds its variables)
.helm/                Helm chart
.github/workflows/    CI
external/SemIf/       the SemIf reference for comparisons and benchmarks (not in git)
runs/                 run results (not in git)

Development

uv run pytest -q                                              # unit tests, no network (tokenizers are downloaded from HF once)
SEMANTIC_IF_LIVE=1 uv run pytest -q -m live                   # live test against the LiteLLM from .env
uvx ruff@0.16.7 check . && uvx ruff@0.16.7 format --check .   # as in CI

The SemIf reference for the comparison tests, the code layout and the conventions are in docs/development.md.

Acknowledgements

• Jev by TypeSafe: the interface this project follows (questions and answers defined at call time, typed answers with probabilities). semantic-if is not affiliated with or endorsed by TypeSafe and does not reproduce Jev's model; Jev and TypeSafe are the property of their owners.

• SemIf (MIT): the method and the prompt this project reproduces. tests/data/decisions.jsonl is a copy of its examples/decisions.jsonl, and the benchmarks use its datasets from a local checkout.

• Swagger UI (Apache-2.0) is vendored in semantic_if/static/swagger-ui/ with its license and notice.

License

MIT, see LICENSE.