# semantic-if

A semantic classifier for LLM agents and pipelines. Give it a state (text or JSON), one question and 2–16 mutually
exclusive options ("which queue should handle this request?", "does the counterparty depend on public
contracts?"). It returns the chosen option and a probability for every option, in 0.15–0.5 s per decision.

The idea is [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev), TypeSafe's closed, hosted model for
semantic decisions: the question and its permitted answers are defined at call time, and what comes back is a typed
answer with probabilities, not text. semantic-if is an open, self-hosted counterpart: an independent implementation of
the direct method from [SemIf](https://github.com/TheoLeeCJ/SemIf), served by your own vLLM through LiteLLM. The model
generates no text: one prefill and one decoding step, and the answer is read from the next-token logprobs of the
option letters. There is nothing to parse, and uncertain cases show up in the probabilities. The spec the project was
built against is in [docs/spec.md](docs/spec.md).

> [!NOTE]
> semantic-if was written by an AI coding agent from that spec. It is covered by tests and was measured against a live
> vLLM ([docs/benchmarks.md](docs/benchmarks.md)), but the agent may have missed something. Review it before you rely on
> it, and please open an issue for anything that looks wrong.

## Features

- the prompt is byte-identical to SemIf `direct-options-v1` (checked by `prompt_sha256`), thinking is off;
- probabilities of all 2–16 options (a float64 softmax over the logprobs of the letter labels `A…P`); the service
  also returns `confidence`, `margin` and `low_confidence`;
- `direct` and `shared` modes; in `shared`, decisions about the same `state` reuse the vLLM prefix cache;
- a library (sync and async), the `semantic-if` CLI with JSONL in the SemIf format, and scripts for comparison,
  quality and speed;
- a service: REST API and MCP server in one process. It holds no key of its own; each caller sends their own
  LiteLLM key;
- rows are never dropped or truncated silently: every error is explicit. The key never reaches output, logs or
  metrics;
- the model is switched with `-m` alone: tokenizers at pinned revisions are kept in a registry.

## How it works

1. The decision `{state, question, options}` is validated by the SemIf rules and rendered into its prompt (Qwen
   ChatML).
2. A local Hugging Face tokenizer checks the label tokens, the tokenization boundary and the prompt length. The
   prompt is never truncated.
3. One `POST /v1/completions` goes to LiteLLM: `max_tokens=1`, `temperature=0`, `allowed_token_ids` = the labels.
4. The letters of the options in use are taken from the raw next-token logprobs and normalized with a softmax.
5. Labels outside the top-k are fetched with a follow-up request. Retries happen only on network errors, 429 and
   5xx.

Details are in [docs/method.md](docs/method.md).

## Requirements

- Python 3.10+ (development uses 3.13 and [uv](https://docs.astral.sh/uv/));
- a [LiteLLM](https://github.com/BerriAI/litellm) proxy in front of a [vLLM](https://github.com/vllm-project/vllm)
  server with an OpenAI-compatible `/v1/completions` that accepts `allowed_token_ids` and `logprobs`;
- a model from the registry (currently Qwen: `qwen-3-8-27b` → `Qwen/Qwen3.8-27B-FP8`, `qwen3.5-9b` →
  `Qwen/Qwen3.5-9B`), served under that alias, or any other Qwen ChatML model with `--tokenizer`
  ([docs/method.md](docs/method.md#8-model-registry)).

## Quick start

```bash
git clone https://github.com/rinat-amanbekov/semantic-if.git && cd semantic-if
```

The whole chain on one machine with an NVIDIA GPU (24 GB; 16 GB with FP8): semantic-if, LiteLLM and vLLM serving
`qwen3.5-9b`, from [compose.yaml](compose.yaml) ([docs/deployment.md](docs/deployment.md#docker-compose)).

```bash
cp .env.example .env               # set LITELLM_MASTER_KEY (and LLM_API_KEY to the same value)
docker compose up -d --wait        # the first start downloads ~19 GB of weights
curl -s http://127.0.0.1:8000/semantic-if/health
```

The library and the CLI, against that stack or a LiteLLM of your own:

```bash
uv sync --extra test     # Python 3.13, venv in .venv; the test extra includes the service (REST + MCP)
```

They take their settings from environment variables or from the nearest `.env` (kept out of git):

```dotenv
LITELLM_BASE_URL=http://localhost:4000/v1
LLM_API_KEY=sk-...
LLM_MODEL=qwen-3-8-27b
```

CLI: JSONL in and out, one line per decision.

```bash
uv run semantic-if -i tests/data/decisions.jsonl -o runs/decisions.jsonl --force
```

Library:

```python
from semantic_if import Decision, classify

row = {
    "id": "route-1",
    "state": "I can't log in after changing my password",
    "question": "Which queue should handle this request?",
    "options": [{"id": "access", "description": "Account access"}, {"id": "other", "description": "Other"}],
}
for result in classify([row], "qwen-3-8-27b"):
    if isinstance(result, Decision):
        print(result.choice, dict(zip(result.option_ids, result.probabilities)))
    else:  # RowError
        print("error:", result.kind, result.error)
```

Service: REST and MCP in one process, with Swagger UI at `/semantic-if/docs`. It reads only environment variables, and
the caller's key comes with every request in `Authorization: Bearer` or `x-litellm-api-key`. The key must be issued
by the LiteLLM instance the service calls ([docs/service.md](docs/service.md)).

```bash
LITELLM_BASE_URL=http://localhost:4000/v1 SEMANTIC_IF_ROOT_PATH=/semantic-if uv run semantic-if-service
curl -s http://127.0.0.1:8000/semantic-if/v1/decide \
  -H "Authorization: Bearer $LITELLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"state": "I cannot log in after changing my password", "question": "Which queue should handle this request?",
       "options": [{"id": "access", "description": "Account access"}, {"id": "other", "description": "Other"}]}'
```

The MCP server (tools `semantic_decision` and `semantic_decisions`) connects to agents such as Claude Code and
OpenCode, directly or through the LiteLLM MCP gateway; see [docs/mcp.md](docs/mcp.md). The Docker image, the Compose
stack and the Helm chart are described in [docs/deployment.md](docs/deployment.md).

## Documentation

| Document | Contents |
| --- | --- |
| [docs/method.md](docs/method.md) | the method: prompt, tokenization and checks, request and readout, model registry |
| [docs/usage.md](docs/usage.md) | CLI and library: flags, `.env`, input and output, errors, the `compare`/`evaluate`/`bench` scripts |
| [docs/service.md](docs/service.md) | REST API: authorization, endpoints, status codes, environment variables, limits, metrics |
| [docs/mcp.md](docs/mcp.md) | MCP: tools, connecting through the LiteLLM gateway and directly, Claude Code, OpenCode, registering the server |
| [docs/benchmarks.md](docs/benchmarks.md) | quality and speed report, vLLM determinism, prefix cache, reproduction |
| [docs/calibration.md](docs/calibration.md) | confidence calibration and option order: a review of AnyJev, the `low_confidence` threshold, when to revisit |
| [docs/development.md](docs/development.md) | development: environment, the SemIf reference, tests, linting, code layout |
| [docs/deployment.md](docs/deployment.md) | Docker image, Docker Compose stack, Helm chart, CI |
| [docs/spec.md](docs/spec.md) | the spec: requirements, stages and acceptance criteria |
| [AGENTS.md](AGENTS.md) | rules for AI agents working on the repository |

## Status

| What | State |
| --- | --- |
| `qwen-3-8-27b` | stages 1–3 and 5 of the spec are done; stage 4 ⚠️: criterion 3.5 (max \|Δp\| ≤ 1e-3) is not met because vLLM is nondeterministic under batching. Mean-family balanced accuracy 0.93–0.98, 0.15–0.5 s per decision ([docs/benchmarks.md](docs/benchmarks.md)) |
| `qwen3.5-9b` | tokenizer in the registry; not benchmarked yet |

## Layout

```text
semantic_if/          library, CLI (cli.py), service (service.py, mcp_server.py), model registry (config.py)
scripts/              compare.py, evaluate.py, bench.py
tests/                unit tests, fake vLLM (fakes.py), live test (test_live.py), tests/data/decisions.jsonl
data/ru_custom.jsonl  our own labeled set in Russian (90 decisions)
docs/                 documentation
Dockerfile, docker/   service image
compose.yaml          local stack: semantic-if, LiteLLM, Postgres, vLLM (.env.example holds its variables)
.helm/                Helm chart
.github/workflows/    CI
external/SemIf/       the SemIf reference for comparisons and benchmarks (not in git)
runs/                 run results (not in git)
```

## Development

```bash
uv run pytest -q                                              # unit tests, no network (tokenizers are downloaded from HF once)
SEMANTIC_IF_LIVE=1 uv run pytest -q -m live                   # live test against the LiteLLM from .env
uvx ruff@0.16.7 check . && uvx ruff@0.16.7 format --check .   # as in CI
```

The SemIf reference for the comparison tests, the code layout and the conventions are in
[docs/development.md](docs/development.md).

## Acknowledgements

- [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) by [TypeSafe](https://typesafe.ai/): the
  interface this project follows (questions and answers defined at call time, typed answers with probabilities).
  semantic-if is not affiliated with or endorsed by TypeSafe and does not reproduce Jev's model; Jev and TypeSafe are
  the property of their owners.
- [SemIf](https://github.com/TheoLeeCJ/SemIf) (MIT): the method and the prompt this project reproduces.
  `tests/data/decisions.jsonl` is a copy of its `examples/decisions.jsonl`, and the benchmarks use its datasets from
  a local checkout.
- [Swagger UI](https://github.com/swagger-api/swagger-ui) (Apache-2.0) is vendored in `semantic_if/static/swagger-ui/`
  with its license and notice.

## License

MIT, see [LICENSE](LICENSE).