estimate_model_fit
Determine whether an LLM fits in a GPU's VRAM by estimating weights, runtime overhead, and KV cache before loading the model.
Instructions
Estimate whether an LLM fits in one GPU's VRAM. Rough estimate: weights + runtime overhead + KV cache. Pass 'architecture' (e.g. llama-3-8b, mistral-7b) for an architecture-aware KV cache; without it, default GQA assumptions are used and flagged.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"gpu_id": "<gpu_id>",
"model_size_b": "<model_size_b>"
},
"name": "estimate_model_fit"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| gpu_id | Yes | ||
| architecture | No | ||
| model_size_b | Yes | ||
| quantization | No | q4 | |
| context_length | No | ||
| parallel_requests | No |