set_placement
Assign specific GPUs and optional tensor splits to roles like chat, embed, rerank, or vision, then persist the manual multi-GPU placement to config.
Instructions
Set and apply a manual multi-GPU placement spec (persists to config).
Maps a role ("chat"/"embed"/"rerank"/"vision") to a placement, e.g.
{"chat": {"devices": [0, 1], "tensor_split": [1, 1]}}. devices is the
GPU indices (get_gpus lists them); tensor_split is optional per-device
weights (omit for an even split). A role left out is auto-placed.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| spec | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||