# llama-mcp-server

MCP server bridging Claude Code to local llama.cpp. Run local LLMs alongside Claude for experimentation, testing, and cost-effective inference.

## Requirements

- Node.js 18+
- [llama.cpp](https://github.com/ggerganov/llama.cpp) with `llama-server` built
- A GGUF model file

## Installation

```bash
npm install llama-mcp-server
```

Or clone and build from source:

```bash
git clone https://github.com/ahays248/llama-mcp-server
cd llama-mcp-server
npm install
npm run build
```

## Configuration

Configure via environment variables:

| Variable | Description | Default |
|----------|-------------|---------|
| `LLAMA_SERVER_URL` | URL of llama-server | `http://localhost:8080` |
| `LLAMA_SERVER_TIMEOUT` | Request timeout in ms | `30000` |
| `LLAMA_MODEL_PATH` | Path to GGUF model file | (none) |
| `LLAMA_SERVER_PATH` | Path to llama-server binary | `llama-server` |

## Usage with Claude Code

### Option 1: Plugin Installation (Recommended)

Due to a [known bug](https://github.com/anthropics/claude-code/issues/12164) in Claude Code, non-plugin MCP servers may connect but not expose their tools. The workaround is to install llama-mcp-server as a plugin via a local marketplace.

**Step 1: Create the marketplace structure**

```
llama-marketplace/
├── .claude-plugin/
│   └── marketplace.json
└── plugins/
    └── llama/
        ├── .claude-plugin/
        │   └── plugin.json
        └── .mcp.json
```

**Step 2: Create marketplace.json**

```json
// llama-marketplace/.claude-plugin/marketplace.json
{
  "name": "llama-marketplace",
  "description": "Local marketplace for llama.cpp MCP plugin",
  "owner": {
    "name": "Your Name"
  },
  "plugins": [
    {
      "name": "llama",
      "description": "llama.cpp MCP server for local LLM inference",
      "source": "./plugins/llama"
    }
  ]
}
```

**Step 3: Create plugin.json**

```json
// llama-marketplace/plugins/llama/.claude-plugin/plugin.json
{
  "name": "llama",
  "version": "0.1.0",
  "description": "llama.cpp MCP server for local LLM inference"
}
```

**Step 4: Create .mcp.json**

```json
// llama-marketplace/plugins/llama/.mcp.json
{
  "mcpServers": {
    "llama": {
      "command": "npx",
      "args": ["-y", "llama-mcp-server"],
      "env": {
        "LLAMA_SERVER_URL": "http://localhost:8080",
        "LLAMA_MODEL_PATH": "/path/to/your/model.gguf",
        "LLAMA_SERVER_PATH": "/path/to/llama-server"
      }
    }
  }
}
```

**Step 5: Install the plugin**

```bash
# Add the local marketplace
claude plugin marketplace add /path/to/llama-marketplace

# Install the plugin
claude plugin install llama@llama-marketplace

# Restart Claude Code
```

After restart, tools will appear as `mcp__plugin_llama_llama__*`.

### Option 2: Direct MCP Configuration

> **Note:** This method may not work due to the bug mentioned above. If tools don't appear after adding the server, use Option 1.

Add to your Claude Code MCP configuration:

```bash
claude mcp add llama -e LLAMA_SERVER_URL=http://localhost:8080 -e LLAMA_MODEL_PATH=/path/to/model.gguf -e LLAMA_SERVER_PATH=/path/to/llama-server -- npx -y llama-mcp-server
```

Or add manually to `~/.claude.json`:

```json
{
  "mcpServers": {
    "llama": {
      "command": "npx",
      "args": ["-y", "llama-mcp-server"],
      "env": {
        "LLAMA_SERVER_URL": "http://localhost:8080",
        "LLAMA_MODEL_PATH": "/path/to/your/model.gguf",
        "LLAMA_SERVER_PATH": "/path/to/llama-server"
      }
    }
  }
}
```

## Tools

### Server Tools

| Tool | Description |
|------|-------------|
| `llama_health` | Check if llama-server is running and get status |
| `llama_props` | Get or set server properties |
| `llama_models` | List available/loaded models |
| `llama_slots` | View current slot processing state |
| `llama_metrics` | Get Prometheus-compatible metrics |

### Token Tools

| Tool | Description |
|------|-------------|
| `llama_tokenize` | Convert text to token IDs |
| `llama_detokenize` | Convert token IDs back to text |
| `llama_apply_template` | Format chat messages using model's template |

### Inference Tools

| Tool | Description |
|------|-------------|
| `llama_complete` | Generate text completion from a prompt |
| `llama_chat` | Chat completion (OpenAI-compatible) |
| `llama_embed` | Generate embeddings for text |
| `llama_infill` | Code completion with prefix and suffix context |
| `llama_rerank` | Rerank documents by relevance to a query |

### Model Management Tools

| Tool | Description |
|------|-------------|
| `llama_load_model` | Load a model (router mode only) |
| `llama_unload_model` | Unload the current model (router mode only) |

### LoRA Tools

| Tool | Description |
|------|-------------|
| `llama_lora_list` | List loaded LoRA adapters |
| `llama_lora_set` | Set LoRA adapter scales |

### Process Control Tools

| Tool | Description |
|------|-------------|
| `llama_start` | Start llama-server as a child process |
| `llama_stop` | Stop the llama-server process |

## Example: Starting llama-server and Running Inference

```
User: Start llama-server with my local model

Claude: I'll start llama-server for you.
[Uses llama_start tool with model path]

User: Generate a haiku about coding

Claude: Let me use the local model for that.
[Uses llama_complete tool]

Result:
Lines of code cascade
Through the silent morning hours
Bugs flee from the light
```

## Development

```bash
# Run tests
npm test

# Type check
npm run typecheck

# Build
npm run build

# Watch mode for development
npm run dev
```

## Troubleshooting

### Tools don't appear in Claude Code

**Symptom:** Server shows "Connected" in `claude mcp list` but no `llama_*` tools are available.

**Cause:** Known bug in Claude Code where non-plugin MCP servers don't expose tools ([#12164](https://github.com/anthropics/claude-code/issues/12164)).

**Solution:** Use the plugin installation method (Option 1 above).

### HTTP 501 errors for certain tools

Some tools require specific server configurations:

| Tool | Requirement |
|------|-------------|
| `llama_metrics` | Start llama-server with `--metrics` flag |
| `llama_embed` | Start llama-server with `--embedding` flag or use an embedding model |
| `llama_infill` | Use a model with fill-in-middle support (e.g., CodeLlama, DeepSeek Coder) |
| `llama_rerank` | Use a reranker model |
| `llama_load_model` / `llama_unload_model` | llama-server must be in router mode |

### Connection refused errors

**Symptom:** `Cannot connect to llama-server at http://localhost:8080`

**Solutions:**
1. Use `llama_start` to start the server, or
2. Start llama-server manually: `llama-server -m /path/to/model.gguf`
3. Check that `LLAMA_SERVER_URL` matches where llama-server is running

### WSL/Windows path issues

When running in WSL, ensure paths use Linux format:
- ✓ `/home/user/models/model.gguf`
- ✗ `C:\Users\user\models\model.gguf`

## License

MIT