local-models

Run decision models and LLMs locally on an Apple Silicon Mac with MLX. Use them from Claude Code over MCP, the command line, HTTP, or a Mac app.

Servers listen only on 127.0.0.1 and have no authentication.

Models

Decision models return calibrated probabilities instead of generated text. They answer yes/no, choice, and score questions. The Kev format is used.

LLMs are for drafting, translating, summarizing, or getting a second opinion. They accept text and image input.

Memory is the measured peak, with a long prompt for the LLMs. Each section runs one model at a time.

Related MCP server: claude-token-saver-mcp

Install

You need an Apple Silicon Mac and uv.

git clone https://github.com/phykn/local-models.git && cd local-models
git clone https://github.com/jaredpalmer/kev.git vendor/kev && git -C vendor/kev checkout kev-1.0
uv sync --python 3.13
.venv/bin/llmctl pull qwen3.8-27b                 # weights go to weight/
HF_HUB_OFFLINE=0 .venv/bin/decisionctl start kev-4b    # the first start downloads the model

Use

Mac app

scripts/build-app.sh && open LOCAL-MODELS.app

Claude Code: register the MCP servers, then ask something like "Use decision to decide whether this ticket is urgent".

claude mcp add decision --scope user -- "$PWD/.venv/bin/decision-mcp"
claude mcp add llm-local --scope user -- "$PWD/.venv/bin/llm-mcp"

• decision provides choose, yes_no, score, decide, and list_models. Check answers marked low_confidence yourself.

• llm-local provides ask and list_models. Pass think: true to ask for harder problems.

Command line: decisionctl and llmctl take the same commands: start, stop, status, list, and port.

.venv/bin/decisionctl start kev-4b

HTTP

• LLMs serve an OpenAI-compatible API at /v1/chat/completions on port 8082. Set model to the id from /v1/models. Send "enable_thinking": false to turn thinking off.

• Decision models serve POST /v1/systemone on port 8081. See the Kev README for the request format.

Configure

Models are registered in decision.toml and llm.toml. Each entry sets the weights, port, memory, and sampling. Pin run to a commit. Unknown keys are an error.

Test

.venv/bin/pytest                                                   # fake servers
.venv/bin/pytest -m live                                           # running models
.venv/bin/python scripts/evaluate.py --data evals/ko/smoke.jsonl   # accuracy and calibration

License

This project is MIT licensed. Kev, Qwen, MiniCPM-V, and Gemma 4 are Apache-2.0. Malkuth-2B is CC BY-NC 4.0, which allows non-commercial use only.