plan_dataset
Validate a local shard manifest and size a bounded cache for large datasets by reading only the manifest, not the corpus.
Instructions
Validate a local immutable shard manifest and size a bounded cache for large datasets.
Reads only the manifest, not the corpus. Does not provision a VM, contact Drive, or confirm source files, VM free disk, Drive quota, throughput or GPU model fit. Training scripts use colab_persist.datasets.ShardCache on the VM.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| cache_gib | No | ||
| reserve_gib | No | ||
| manifest_path | Yes |