Checkpoint Interval Advisor
recommend_checkpoint_intervalRecommends how often to checkpoint a training job using the Young and Daly formulas, from GPU count, blocking checkpoint write time, restart time and job MTTF (estimated from Meta's published 2024 reliability data when not given). Returns the recommended interval, expected failures per day, GPU time wasted per day at the recommended and current intervals, and the daily saving.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| gpu_count | Yes | GPUs in the training job. | |
| job_mttf_hours | No | Measured mean time between job-interrupting failures for a job this size, in hours. If omitted, it is estimated from Meta's published 2024 cluster reliability data. | |
| restart_minutes | No | Time from a failure to training resuming: detection, rescheduling, process start and checkpoint load, in minutes. Default 20. | |
| gpu_hour_cost_usd | No | Cost per GPU-hour in USD. Default 2.75. | |
| checkpoint_write_minutes | Yes | Time training is blocked while a checkpoint is written, in minutes. With asynchronous checkpointing, use only the blocking part. | |
| current_interval_minutes | No | Current checkpoint interval in minutes, to compare against the recommendation. |