jetson-llm-mcp

An MCP server that exposes local LLMs running on a Jetson AGX Xavier to Claude Code.

Two tools: ask_jetson (send a prompt to a local model) and check_jetson_status (see what is available and what is loaded).

The point is to offload small, cheap work — quick classifications, short code review, sanity checks — to hardware you already own, instead of spending API tokens on it.

Target

The server talks to jetson-xav2 over HTTP, to a stock ollama install on port 11434. Nothing needs to be installed on the Jetson beyond ollama itself, and ollama.service is enabled there, so it comes back on its own after a reboot:

$ ssh [name]@[host] 'systemctl is-enabled ollama; systemctl is-active ollama'
enabled
active

Available models:

Related MCP server: LMStudio-MCP

Performance

Measured on jetson-xav2 with qwen3-coder:30b:

$ curl -s http://jetson-xav2:11434/api/generate -d '{...,"options":{"num_predict":60}}'
load      : 0.3 s
eval      : 24 tokens on 1.3 s  => 18.92 tok/s

So roughly 19 tok/s once warm, plus about 30 s the first time a model is loaded after an idle period. Budget max_tokens accordingly — the default of 150 is a few seconds of generation on a warm model.

Two constraints worth knowing

Only one 30b model fits at a time. The Xavier has unified memory — the GPU shares the system's ~30 GiB rather than having its own VRAM — and a loaded 30b model occupies ~25 GB of it. Asking for the other one evicts the first, and asking while the first is still resident can fail outright:

{"error":{"message":"model failed to load, this may be due to resource
limitations or an internal error, check ollama server logs for details"}}

check_jetson_status reports what is currently resident, so you can tell a slow first call from a genuine problem.

Thinking models put their reasoning in a separate field. With qwen3:30b-thinking, ollama's OpenAI-compatible endpoint returns the chain of thought in reasoning and leaves content empty when max_tokens runs out mid-thought. Reading content alone yields a silent empty string — exactly what low max_tokens provokes. ask_jetson falls back to reasoning and labels it:

[no answer — only reasoning, length]
We are to determine if 17 is a prime number.
 A prime number is a natural number greater than 1 that has no positive divisors

If you see that, raise max_tokens or use the default non-thinking model.

Setup

Requires Python 3.10+, mcp>=1.0 and httpx>=0.27. Register it with Claude Code as a stdio server:

{
  "mcpServers": {
    "jetson-llm": {
      "type": "stdio",
      "command": "/path/to/python3",
      "args": ["/path/to/jetson-llm-mcp/server.py"]
    }
  }
}

After editing server.py, reconnect the server (/mcp → jetson-llm) — for a stdio server that spawns a fresh process and re-reads the file. A full restart of Claude Code is only needed if you change the configuration above, which is read at session start.

Why not jetson-xav1

jetson-xav1 also exists and also runs ollama, hosting deepseek-r1:32b, deepseek-r1:14b and a deepseek32b-slim derivative. It is deliberately not used:

• deepseek-r1 is older, and the distilled qwen3 models on xav2 do the same jobs better.

• Its 32b path ran through a hand-built llama.cpp on port 11435 at ~1.4 tok/s, about 13× slower than xav2.

• That llama-server has no systemd unit. It was started by hand via ~/start_llama_server.sh, so it does not survive a reboot — which is why this server pointed at a dead port until it was retargeted.

• ollama and llama-server cannot share the Xavier's NvMap CUDA carveout, so running both on one machine means fighting over memory. The start script works around that with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 and -ngl 32 (32 layers on GPU, 32 on CPU, against an NvMap limit of ~9–13 GB).

If xav1 is ever brought back, it needs a host argument here and a systemd unit there.