start_late_infer
Start a local late-infer server on 127.0.0.1:8010 by spawning the binary and loading a Hugging Face Instruct model. Attach idle GPUs, and wait for approval before running further requests.
Instructions
Start late-infer on this computer (OpenAI /v1 on 127.0.0.1:8010 only, never 0.0.0.0 or LAN). Spawns the late-infer binary (PATH or sibling Late target/release). First start loads Hugging Face safetensors (not packed). Extra calls wait for Late Approve — this is not auto-run.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | Hugging Face Instruct id (default Qwen/Qwen2.5-0.5B-Instruct) | |
| use_all_gpus | No | Override on your computer. Default Start pins the idle (non-display) GPU at full VRAM. Display GPUs stay at 70% — this runtime cannot apply that cap, so they are not attached. Does not start vLLM. |