# llm-local

Run [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) on an Apple Silicon Mac with [MLX](https://github.com/ml-explore/mlx), and use it from:

- **Claude Code**, through an MCP server (`llm-mcp`)
- **any program**, through an OpenAI-compatible API on `http://127.0.0.1:<port>/v1`
- **a Mac app** that starts and stops the server

The server listens only on `127.0.0.1`. Nothing is exposed to the network, and there is no authentication.

## Install

You need an Apple Silicon Mac and [uv](https://docs.astral.sh/uv/). The default model is the 4-bit [mxfp4 build](https://huggingface.co/mlx-community/Qwen3.8-27B-mxfp4) (about 15 GB of weights). With a 32K-token context it needs about 21 GB of free memory.

A second, optional model is [Qwen3.6-35B-A3B](https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-mxfp4), a mixture-of-experts model that uses about 3B parameters per token. It generates and reads prompts about six times faster than Qwen3.8-27B but is weaker at coding and reasoning, so it suits short tasks such as translation and summaries. It needs about 22 GB of free memory, so run one model at a time on a 32 GB Mac.

```bash
git clone https://github.com/phykn/llm-local.git && cd llm-local
uv sync
.venv/bin/llmctl pull qwen3.8-27b      # downloads the weights into weight/ once
.venv/bin/llmctl pull qwen3.6-35b-a3b  # optional
```

## Use

```bash
.venv/bin/llmctl start qwen3.8-27b     # uses only weights already in weight/
claude mcp add llm-local --scope user -- "$PWD/.venv/bin/llm-mcp"
```

Start a new Claude Code session and ask, for example: "Use llm-local to translate this paragraph into Korean: ...".

- **As a Mac app:** run `scripts/build-app.sh` once, then open `LLM-LOCAL.app`, pick a model and press Start. Quitting the app, even with Force Quit, stops the server.
- **Stop the model:** `llmctl stop qwen3.8-27b`
- **Check what is running:** `llmctl status`

### MCP tools

- **ask**: sends a prompt (and an optional system prompt) to the local model and returns the answer text. Thinking is off by default; pass `think: true` for harder problems. It is slower, and the reasoning itself is not returned.
- **list_models**: shows the registered models, whether each is running, and the command to start it.

### HTTP API

With mlx-vlm, `model` must be the id of the loaded model: the entry with `"loaded": true` in `curl -s http://127.0.0.1:8020/v1/models` (its snapshot path under `weight/`).

```bash
curl http://127.0.0.1:8020/v1/chat/completions -H 'content-type: application/json' -d '{
  "model": "<id of the loaded model>",
  "messages": [{"role": "user", "content": "Hello"}],
  "enable_thinking": false,
  "max_tokens": 256
}'
```

Qwen3.8 thinks before answering by default. Send `"enable_thinking": false` (mlx-vlm) or `"chat_template_kwargs": {"enable_thinking": false}` (mlx-lm) to turn it off. Any OpenAI client works with `base_url="http://127.0.0.1:8020/v1"`.

## Models

`models.toml` lists the models. `llmctl`, the MCP server and the app all read this file.

```toml
[models."qwen3.8-27b"]
server = "mlx-vlm"          # mlx-vlm or mlx-lm
run = "mlx-community/Qwen3.8-27B-mxfp4@97ab0819817ab1c61d7d39f9169fc71999915641"   # Hugging Face id@commit, or a local directory
port = 8020
memory_gb = 21              # llmctl warns when less memory than this is free
max_kv_size = 32768         # context limit in tokens (mlx-vlm only)
think = { temperature = 1.0, top_p = 0.95, top_k = 20 }
nothink = { temperature = 0.7, top_p = 0.8, top_k = 20, presence_penalty = 1.5 }
```

To add a model, add an entry with its own port, then run `llmctl pull <name>`. Pin `run` to a commit so the weights do not change under you.

## Layout

- `src/llm_local/`: the Python package. `registry.py` reads `models.toml`, `backends.py` calls a running server, `ctl.py` is `llmctl` and `mcp_server.py` is `llm-mcp`.
- `app/`: the Mac app in Swift. `scripts/build-app.sh` builds it into `LLM-LOCAL.app`.
- `weight/`: downloaded weights (`HF_HOME`). `logs/`: server logs and pid files.

## Test

```bash
.venv/bin/pytest                # uses a fake server
.venv/bin/pytest -m live        # against a running model
```

## License

MIT. Qwen3.8 weights are licensed separately under Apache-2.0.