Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@llm-mcp ask Qwen to translate this into Korean: Good morning"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Run Qwen3.8-27B on an Apple Silicon Mac with MLX, and use it from:
Claude Code, through an MCP server (llm-mcp)
any program, through an OpenAI-compatible API on http://127.0.0.1:<port>/v1
a Mac app that starts and stops the server
The server listens only on 127.0.0.1. Nothing is exposed to the network, and there is no authentication.
You need an Apple Silicon Mac and uv. The default model is the 4-bit mxfp4 build (about 15 GB of weights). With a 32K-token context it needs about 21 GB of free memory.
A second, optional model is Qwen3.6-35B-A3B, a mixture-of-experts model that uses about 3B parameters per token. It generates and reads prompts about six times faster than Qwen3.8-27B but is weaker at coding and reasoning, so it suits short tasks such as translation and summaries. It needs about 22 GB of free memory, so run one model at a time on a 32 GB Mac.
git clone https://github.com/phykn/llm-local.git && cd llm-local
uv sync
.venv/bin/llmctl pull qwen3.8-27b # downloads the weights into weight/ once
.venv/bin/llmctl pull qwen3.6-35b-a3b # optionalRelated MCP server: Ollama MCP Server
.venv/bin/llmctl start qwen3.8-27b # uses only weights already in weight/
claude mcp add llm-local --scope user -- "$PWD/.venv/bin/llm-mcp"Start a new Claude Code session and ask, for example: "Use llm-local to translate this paragraph into Korean: ...".
As a Mac app: run scripts/build-app.sh once, then open LLM-LOCAL.app, pick a model and press Start. Quitting the app, even with Force Quit, stops the server.
Stop the model: llmctl stop qwen3.8-27b
Check what is running: llmctl status
ask: sends a prompt (and an optional system prompt) to the local model and returns the answer text. Thinking is off by default; pass think: true for harder problems. It is slower, and the reasoning itself is not returned.
list_models: shows the registered models, whether each is running, and the command to start it.
With mlx-vlm, model must be the id of the loaded model: the entry with "loaded": true in curl -s http://127.0.0.1:8020/v1/models (its snapshot path under weight/).
curl http://127.0.0.1:8020/v1/chat/completions -H 'content-type: application/json' -d '{
"model": "<id of the loaded model>",
"messages": [{"role": "user", "content": "Hello"}],
"enable_thinking": false,
"max_tokens": 256
}'Qwen3.8 thinks before answering by default. Send "enable_thinking": false (mlx-vlm) or "chat_template_kwargs": {"enable_thinking": false} (mlx-lm) to turn it off. Any OpenAI client works with base_url="http://127.0.0.1:8020/v1".
models.toml lists the models. llmctl, the MCP server and the app all read this file.
[models."qwen3.8-27b"]
server = "mlx-vlm" # mlx-vlm or mlx-lm
run = "mlx-community/Qwen3.8-27B-mxfp4@97ab0819817ab1c61d7d39f9169fc71999915641" # Hugging Face id@commit, or a local directory
port = 8020
memory_gb = 21 # llmctl warns when less memory than this is free
max_kv_size = 32768 # context limit in tokens (mlx-vlm only)
think = { temperature = 1.0, top_p = 0.95, top_k = 20 }
nothink = { temperature = 0.7, top_p = 0.8, top_k = 20, presence_penalty = 1.5 }To add a model, add an entry with its own port, then run llmctl pull <name>. Pin run to a commit so the weights do not change under you.
src/llm_local/: the Python package. registry.py reads models.toml, backends.py calls a running server, ctl.py is llmctl and mcp_server.py is llm-mcp.
app/: the Mac app in Swift. scripts/build-app.sh builds it into LLM-LOCAL.app.
weight/: downloaded weights (HF_HOME). logs/: server logs and pid files.
.venv/bin/pytest # uses a fake server
.venv/bin/pytest -m live # against a running modelMIT. Qwen3.8 weights are licensed separately under Apache-2.0.
Send one prompt to a local model and return its answer text.
system: optional system prompt. model: a name from list_models; omit for the default. think: let the model reason before answering. Better on hard problems, but often several times slower; the reasoning itself is not returned. max_tokens: upper bound on generated tokens, reasoning included. finish_reason "length" means the answer was cut off.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| think | No | ||
| prompt | Yes | ||
| system | No | ||
| max_tokens | No |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover the safety profile (readOnlyHint=true, openWorldHint=false), and the description adds genuinely useful behavior beyond them: the model is local, reasoning tokens are consumed but 'not returned', and finish_reason 'length' signals truncation. It stops short of describing concurrency, timeout, or error behavior, so 4 rather than 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core action is front-loaded in one sentence, then parameters are listed one per line with no filler. Every clause adds information (defaults, trade-offs, truncation semantics) rather than restating the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description need not enumerate return fields, and it correctly focuses on the one return nuance an agent must act on (finish_reason 'length'). Nothing critical is missing for a 5-parameter inference call, though latency/cost hints beyond the 'think' note would round it out.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description carries the full burden and does so: it explains system, model (source and default), think (behavioral trade-off plus the fact that reasoning is not returned), and max_tokens ('reasoning included' plus the length cutoff meaning). This is meaning the bare schema types and defaults cannot convey.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Send one prompt to a local model and return its answer text'), and the parameter note 'a name from list_models' implicitly establishes the division of labor with its only sibling. An agent can tell immediately that this performs inference while list_models enumerates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear operational guidance for the optional parameters, e.g. 'omit for the default' and 'think: better on hard problems, but often several times slower'. It does not state exclusions or explicitly contrast with list_models, but the usage context is unambiguous for a single-prompt inference call.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The registered local models, which one is the default, and whether each is running (with its start command if not).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, and an output schema exists, so the safety and return-shape burden is largely carried elsewhere. The description adds a modest value: that unstarted models include a start command, hinting at actionable follow-up. It does not discuss refresh, caching, or whether the list is dynamic.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single well-formed sentence that front-loads the primary content and never wastes a word. Nothing is repeated from the title or schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, read-only listing tool with an output schema and annotations covering safety, the description is sufficient; return values need not be restated because the output schema covers them.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is no parameter semantics to document; the baseline for a no-parameter tool applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the resource concretely and enumerates the three things it surfaces: registered local models, which is the default, and each one's running state. It is clearly distinguishable from the only sibling, 'ask', but it never names the action (list) in the description itself and offers no explicit sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to call this tool versus the 'ask' sibling, nor any prerequisites or exclusions. Usage is only implied by the content it returns.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0asklist_modelsScored across 2 tools
The two tools occupy entirely disjoint roles: list_models is a read-only discovery/inventory call, while ask performs inference. An agent can unambiguously pick between them, and all the optional knobs (system, model, think, max_tokens) are correctly folded into ask as parameters rather than separate tools.
list_models follows a clean verb_noun pattern, but ask is a bare verb with no noun, so the set is not fully uniform (ask_model or prompt_model would have matched). With only two tools this is a minor, still-readable deviation.
Two tools is thin for a server whose apparent scope is managing and using local models. The surface is defensible because most capability is expressed as ask parameters, but it sits at the borderline of feeling under-specified.
Discover models and send a single prompt covers the core inference loop, but there is no model lifecycle control (starting/stopping a model, despite list_models reporting start commands and running state), no multi-turn chat, no streaming, and no embeddings. These are notable gaps an agent will hit quickly.
Source-checked CLI guides and model-aware planning for Claude Code, Codex, and Grok Build.
Use your Mac, Windows or Linux computer from ChatGPT, Claude or Codex: files, commands, documents.
Use AI models for chat, image, and video generation from Claude Code and other MCP hosts.
Work on your own computers from Claude or ChatGPT: run commands, edit and search files.
