start_vllm
Launch a local OpenAI-compatible vLLM server per model, bind it to 127.0.0.1, wait for health, and register it as a backend.
Instructions
Launch a local OpenAI-compatible server bound to 127.0.0.1 only (never 0.0.0.0). Each catalog model gets its own Docker container, port in 8000–8099, backend id (vllm- from the catalog id, e.g. vllm-qwen25-7b-instruct), and specialist. Tensor-parallels across every GPU only when the catalog fit needs more than one card; models that fit a single GPU stay at --tensor-parallel-size 1. Pass use_all_gpus=false to pin to one GPU even when the model is larger. Does not stop other orchestrator vLLM containers unless replace=true (restarts this model only). On intel-xpu, if intel/llm-scaler-vllm or intel/vllm:*xpu is local, starts that container (API published as 127.0.0.1:port:8000). Model must already be downloaded. Waits until GET /v1/models is healthy, upserts that backend + specialist, and stores a dummy loopback Bearer in gitignored GUI secrets if needed (never copy a key from vLLM). Cloud agents still cannot reach this server.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| host | No | Must be 127.0.0.1 or localhost; any other value is rejected | |
| port | No | ||
| image | No | Local Docker image override, e.g. intel/vllm:0.17.0-xpu or intel/llm-scaler-vllm:0.21.0-b3 | |
| replace | No | If true, stop the existing instance of this model before starting. Other running vLLM models are left alone. | |
| runtime | No | Force Docker (Intel images) or host vllm. Default on intel-xpu is docker when a matching image exists. | |
| model_id | Yes | Catalog id or Hugging Face repo already downloaded | |
| timeout_ms | No | ||
| quantization | No | ||
| use_all_gpus | No | Allow using every GPU when the catalog fit needs tensor parallel. False: pin to one GPU. Models that already fit one card stay on one GPU even when this is true. |