dio_predict_latency
Predict queue delay, execution latency, and optimal backend before sending requests to choose the most cost-effective LLM endpoint.
Instructions
Get latency and cost predictions before sending requests. Predicts queue delay, execution latency, and optimal backend using DIO's dual-timescale NLMS filter.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | Target model name (optional) | |
| prompt | Yes | The prompt text or query to estimate | |
| tokens | No | Estimated token count (optional) |