cost_per_token
Computes cost per million tokens from live vLLM throughput using GPU hourly cost and number of GPUs. Returns insufficient-data forecast when throughput is missing.
Instructions
[READ] Attribute a $/1M-token unit cost from live vLLM throughput.
Multiplies the current generation-throughput gauge by the supplied GPU
hourly cost to derive the cost of serving 1M tokens; degrades to an
insufficient-data forecast when no throughput metric is present.
Args: gpu_hourly_cost: Hourly cost of a single GPU (e.g. cloud on-demand rate). num_gpus: Number of GPUs backing the deployment; defaults to 1. target: Inference target name from config; omit for the default.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| target | No | ||
| num_gpus | No | ||
| gpu_hourly_cost | Yes |