prism

Prism puts ONNX Runtime GenAI (CUDA or CPU) and Ollama / llama.cpp (GGUF) behind one command line, one /v1/chat/completions endpoint on a fixed port, and ready-made connectors for Cursor, Cline and MCP.

📖 Documentation: https://senssei.github.io/prism-local/

Status: alpha (v0.2.0). It works, it is tested (~300 tests, no GPU needed), and its defaults are safe (loopback-only, no CORS). APIs and flags may still change. See Known limitations.

Why

Prism began as an evaluation of Microsoft Foundry Local on WSL2 (see the research notes). In that evaluation the official CLI (0.10.3) detected no GPU under WSL2, ran on the CPU, and served on a random port. Prism keeps the useful part, running ONNX GenAI models locally, and adds:

Related MCP server: Local LLM MCP Server

Quickstart

python3 -m venv .venv && source .venv/bin/activate
pip install "prism-local[cuda,pull]"   # GPU stack (needs Python 3.11+) + huggingface_hub; both optional
prism doctor                       # checks NVML, ONNX Runtime GenAI, the CUDA provider, Ollama
prism pull phi-4-mini              # downloads to ~/.prism/models
prism run phi-4-mini "Write a Fibonacci function in Python."
prism serve                        # http://127.0.0.1:5272/v1

Without a GPU, pip install prism-local is enough for CPU and Ollama use. You can also run from a checkout without installing: ./bin/prism …. The Python it uses comes from $PRISM_PYTHON, then ./.venv, then the active virtualenv, then python3.

curl -s http://127.0.0.1:5272/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "phi-4-mini", "stream": true,
  "messages": [{"role": "user", "content": "Count from 1 to 5."}]
}'

Commands

run, chat, serve and benchmark accept --device auto|cuda|cpu (or $PRISM_DEVICE). auto tries CUDA and falls back to CPU with a warning that says why; cuda fails instead of falling back.

Configuration

Models in the Foundry Local cache (~/.foundry/cache/models) are also discovered.

Security defaults

prism serve binds to 127.0.0.1, sends no CORS headers, rejects non-loopback Host headers (DNS-rebinding defence) and caps request bodies at 10 MB. To expose it on a network, set a key: prism serve --host 0.0.0.0 --api-key "$(openssl rand -hex 16)". Details: Security.

IDE and agent integration

prism connect cursor --test --export-rules --export-mcp   # Cursor: provider settings, .cursorrules, MCP
prism connect cline --test --export-mcp                   # Cline
prism connect mcp --target claude --write                 # Claude Desktop, Cursor, Antigravity

The MCP server exposes prism_ask_coder, prism_code_review, prism_list_models, prism_get_status and prism_benchmark. See Integrations.

Performance

Speed depends almost entirely on which execution provider runs. Measured with prism benchmark on 2026-09-19 (RTX 5070 12 GB, WSL2, driver 615.71, Phi-4-mini INT4, onnxruntime-genai-cuda 0.16.0, onnxruntime-gpu 1.30.0, CUDA 13 libraries, two runs each):

Earlier evaluation figures of 118–130 tok/s and a ~56 ms TTFT (see the reproducibility note) were not reproduced; the GPU here was shared with about 6.7 GB of other applications and the prompt differs. Measure your own machine with prism benchmark <model>; it prints the provider it really used, and prism doctor shows a CUDA library mismatch.

Known limitations

• CUDA needs matching libraries and Python 3.11+. pip install "prism-local[cuda]" installs a matched stack (ONNX Runtime GenAI, ONNX Runtime GPU and its CUDA 13 / cuDNN libraries, about 2.5 GB). If you bring your own environment, prism doctor names any missing library.

• One ONNX model is resident at a time and requests are serialized (a lock), so this is a single-user local server, not a high-concurrency one.

• No embeddings from ONNX models (/v1/embeddings is served by Ollama), and tool calling on ONNX models needs prism-local[jinja] and a chat template that takes tools.

• Chat templates are detected from the model name and genai_config.json (Phi, Qwen/ChatML, Llama 3, DeepSeek); unknown families fall back to ChatML and may need a template added.

• Linux and WSL2 only.

Development

pip install -e ".[dev]"
PYTHONPATH=. python3 -m unittest discover -s tests -v    # ~300 tests, ~20 s, no GPU/network/models needed
pip install -e ".[docs]" && mkdocs serve                  # docs site at http://127.0.0.1:8000

See CONTRIBUTING.md. Repository layout: prism/ (the product), tests/, docs/, and foundry_wsl/, the earlier WSL2 bridge toolkit kept for reference.

License

Apache-2.0