Skip to main content
Glama
llama-mcp-server

llama-mcp-server

by ahays248

llama-mcp-server

MCP server bridging Claude Code to local llama.cpp. Run local LLMs alongside Claude for experimentation, testing, and cost-effective inference.

Requirements

  • Node.js 18+

  • llama.cpp with llama-server built

  • A GGUF model file

Related MCP server: MCP-Demo

Installation

npm install llama-mcp-server

Or clone and build from source:

git clone https://github.com/ahays248/llama-mcp-server
cd llama-mcp-server
npm install
npm run build

Configuration

Configure via environment variables:

Variable

Description

Default

LLAMA_SERVER_URL

URL of llama-server

http://localhost:8080

LLAMA_SERVER_TIMEOUT

Request timeout in ms

30000

LLAMA_MODEL_PATH

Path to GGUF model file

(none)

LLAMA_SERVER_PATH

Path to llama-server binary

llama-server

Usage with Claude Code

Due to a known bug in Claude Code, non-plugin MCP servers may connect but not expose their tools. The workaround is to install llama-mcp-server as a plugin via a local marketplace.

Step 1: Create the marketplace structure

llama-marketplace/
├── .claude-plugin/
│   └── marketplace.json
└── plugins/
    └── llama/
        ├── .claude-plugin/
        │   └── plugin.json
        └── .mcp.json

Step 2: Create marketplace.json

// llama-marketplace/.claude-plugin/marketplace.json
{
  "name": "llama-marketplace",
  "description": "Local marketplace for llama.cpp MCP plugin",
  "owner": {
    "name": "Your Name"
  },
  "plugins": [
    {
      "name": "llama",
      "description": "llama.cpp MCP server for local LLM inference",
      "source": "./plugins/llama"
    }
  ]
}

Step 3: Create plugin.json

// llama-marketplace/plugins/llama/.claude-plugin/plugin.json
{
  "name": "llama",
  "version": "0.1.0",
  "description": "llama.cpp MCP server for local LLM inference"
}

Step 4: Create .mcp.json

// llama-marketplace/plugins/llama/.mcp.json
{
  "mcpServers": {
    "llama": {
      "command": "npx",
      "args": ["-y", "llama-mcp-server"],
      "env": {
        "LLAMA_SERVER_URL": "http://localhost:8080",
        "LLAMA_MODEL_PATH": "/path/to/your/model.gguf",
        "LLAMA_SERVER_PATH": "/path/to/llama-server"
      }
    }
  }
}

Step 5: Install the plugin

# Add the local marketplace
claude plugin marketplace add /path/to/llama-marketplace

# Install the plugin
claude plugin install llama@llama-marketplace

# Restart Claude Code

After restart, tools will appear as mcp__plugin_llama_llama__*.

Option 2: Direct MCP Configuration

Note: This method may not work due to the bug mentioned above. If tools don't appear after adding the server, use Option 1.

Add to your Claude Code MCP configuration:

claude mcp add llama -e LLAMA_SERVER_URL=http://localhost:8080 -e LLAMA_MODEL_PATH=/path/to/model.gguf -e LLAMA_SERVER_PATH=/path/to/llama-server -- npx -y llama-mcp-server

Or add manually to ~/.claude.json:

{
  "mcpServers": {
    "llama": {
      "command": "npx",
      "args": ["-y", "llama-mcp-server"],
      "env": {
        "LLAMA_SERVER_URL": "http://localhost:8080",
        "LLAMA_MODEL_PATH": "/path/to/your/model.gguf",
        "LLAMA_SERVER_PATH": "/path/to/llama-server"
      }
    }
  }
}

Tools

Server Tools

Tool

Description

llama_health

Check if llama-server is running and get status

llama_props

Get or set server properties

llama_models

List available/loaded models

llama_slots

View current slot processing state

llama_metrics

Get Prometheus-compatible metrics

Token Tools

Tool

Description

llama_tokenize

Convert text to token IDs

llama_detokenize

Convert token IDs back to text

llama_apply_template

Format chat messages using model's template

Inference Tools

Tool

Description

llama_complete

Generate text completion from a prompt

llama_chat

Chat completion (OpenAI-compatible)

llama_embed

Generate embeddings for text

llama_infill

Code completion with prefix and suffix context

llama_rerank

Rerank documents by relevance to a query

Model Management Tools

Tool

Description

llama_load_model

Load a model (router mode only)

llama_unload_model

Unload the current model (router mode only)

LoRA Tools

Tool

Description

llama_lora_list

List loaded LoRA adapters

llama_lora_set

Set LoRA adapter scales

Process Control Tools

Tool

Description

llama_start

Start llama-server as a child process

llama_stop

Stop the llama-server process

Example: Starting llama-server and Running Inference

User: Start llama-server with my local model

Claude: I'll start llama-server for you.
[Uses llama_start tool with model path]

User: Generate a haiku about coding

Claude: Let me use the local model for that.
[Uses llama_complete tool]

Result:
Lines of code cascade
Through the silent morning hours
Bugs flee from the light

Development

# Run tests
npm test

# Type check
npm run typecheck

# Build
npm run build

# Watch mode for development
npm run dev

Troubleshooting

Tools don't appear in Claude Code

Symptom: Server shows "Connected" in claude mcp list but no llama_* tools are available.

Cause: Known bug in Claude Code where non-plugin MCP servers don't expose tools (#12164).

Solution: Use the plugin installation method (Option 1 above).

HTTP 501 errors for certain tools

Some tools require specific server configurations:

Tool

Requirement

llama_metrics

Start llama-server with --metrics flag

llama_embed

Start llama-server with --embedding flag or use an embedding model

llama_infill

Use a model with fill-in-middle support (e.g., CodeLlama, DeepSeek Coder)

llama_rerank

Use a reranker model

llama_load_model / llama_unload_model

llama-server must be in router mode

Connection refused errors

Symptom: Cannot connect to llama-server at http://localhost:8080

Solutions:

  1. Use llama_start to start the server, or

  2. Start llama-server manually: llama-server -m /path/to/model.gguf

  3. Check that LLAMA_SERVER_URL matches where llama-server is running

WSL/Windows path issues

When running in WSL, ensure paths use Linux format:

  • ✓ /home/user/models/model.gguf

  • ✗ C:\Users\user\models\model.gguf

License

MIT

Available Tools

19 tools
llama_apply_templateA

Format chat messages using model's template without inference

Inputs: messages (required).

llama_chatC

Chat completion (OpenAI-compatible format)

Inputs: messages (required), max_tokens, temperature, top_p, +2 more.

llama_completeB

Generate text completion from a prompt

Inputs: prompt (required), max_tokens, temperature, top_p, +3 more.

llama_detokenizeB

Convert token IDs back to text

Inputs: tokens (required).

llama_embedC

Generate embeddings for text

Inputs: content (required).

llama_healthB

Check if llama-server is running and get status

Inputs: no named parameters.

llama_infillC

Code completion with prefix and suffix context (fill-in-middle)

Inputs: input_prefix (required), input_suffix (required), max_tokens, temperature, +1 more.

llama_load_modelC

Load a model (router mode only)

Inputs: model (required).

llama_lora_listA

List loaded LoRA adapters

Inputs: no named parameters.

llama_lora_setC

Set LoRA adapter scales

Inputs: adapters (required).

llama_metricsA

Get Prometheus-compatible metrics (tokens processed, latency, etc.)

Inputs: no named parameters.

llama_modelsA

List available/loaded models

Inputs: no named parameters.

llama_propsB

Get or set server properties and default generation settings

Inputs: default_generation_settings.

llama_rerankB

Rerank documents by relevance to a query

Inputs: query (required), documents (required).

llama_slotsA

View current slot processing state

Inputs: no named parameters.

llama_startB

Start llama-server as a child process with the specified model

Inputs: model (required), port, ctx_size, n_gpu_layers, +1 more.

llama_stopA

Stop the running llama-server process

Inputs: no named parameters.

llama_tokenizeC

Convert text to token IDs

Inputs: content (required), add_special, with_pieces.

llama_unload_modelB

Unload the current model (router mode only)

Inputs: model (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 19 tools

    Disambiguation5/5

    Each tool targets a distinct operation or resource: chat vs. completion vs. infill vs. embed vs. rerank, plus model and server management. No ambiguous overlaps.

    Naming Consistency5/5

    All tools follow the 'llama_' prefix with lowercase snake_case names, using verbs or verb_noun patterns consistently.

    Tool Count4/5

    19 tools is slightly above the typical 3-15 range, but each tool serves a clear purpose covering model interaction, management, and server control, so it remains reasonable.

    Completeness5/5

    The tool set covers the full lifecycle: chat/completion/infill/embed/rerank, tokenization, model loading/management, server health and metrics, and LoRA adapter control. No obvious gaps.

    Related MCP Connectors

    Related MCP Servers