split_dataset

Split a dataset into training, testing, and optionally validation subsets.

Creates a new dataset hierarchy with full provenance tracking:

• Split (parent, type: "Split")

  • Training (child, type: "Training" + training_types)

  • Validation (child, type: "Validation" + validation_types) # if val_size

  • Testing (child, type: "Testing" + testing_types)

The API follows scikit-learn's train_test_split conventions for test_size, train_size, val_size, shuffle, and seed parameters.

Splitting strategies:

• Random (default): Shuffles members and splits at the boundary. No denormalization needed. Fast for any dataset size.

• Stratified: Maintains class distribution across splits. Requires stratify_by_column and include_tables. Uses scikit-learn internally.

Column naming for stratification:

When using stratify_by_column, the column name must match the denormalized DataFrame format: {TableName}_{ColumnName}. For example, to stratify by the Image_Class column from the Image_Classification feature table, use Image_Classification_Image_Class.

Derive the column name from the table schema (via the deriva://catalog/schema or deriva://catalog/features resource) rather than calling denormalize_dataset().

Args: source_dataset_rid: RID of the source dataset to split. test_size: Test set size as a fraction (0-1) or absolute count. Default: 0.2 (20% of data). train_size: Train set size as a fraction (0-1) or absolute count. Default: None (complement of test_size and val_size). val_size: Validation set size as a fraction (0-1) or absolute count. Default: None (no validation split, two-way only). When provided, creates a three-way train/val/test split. seed: Random seed for reproducibility. Default: 42. shuffle: Whether to shuffle before splitting. Default: True. stratify_by_column: Column name in the denormalized DataFrame for stratified splitting. Maintains class distribution across all partitions. Requires include_tables. Example: "Image_Classification_Image_Class". stratify_missing: Policy for null values in the stratify column. "error" (default): raise if any nulls exist, reporting count and percentage. "drop": exclude rows with null values from the split. "include": treat nulls as a separate class. Only used when stratify_by_column is set. element_table: Element table to split (e.g., "Image"). If not specified, auto-detected from the dataset's members. include_tables: Tables to include when denormalizing. Required when using stratify_by_column. Example: ["Image", "Image_Classification"]. training_types: Additional dataset types for the training set beyond "Training". Example: ["Labeled"]. testing_types: Additional dataset types for the testing set beyond "Testing". Example: ["Labeled"]. validation_types: Additional dataset types for the validation set beyond "Validation". Example: ["Labeled"]. Ignored when val_size is None. split_description: Description for the parent Split dataset. dry_run: If True, return what would happen without modifying the catalog. Useful for previewing split sizes.

Returns: JSON with split results including: - split: RID, version, and count of the parent Split dataset - training: RID, version, and count of the Training dataset - validation: RID, version, and count of the Validation dataset (if val_size) - testing: RID, version, and count of the Testing dataset - source: RID of the source dataset

Example: # Random 80/20 split split_dataset("28D0", test_size=0.2, seed=42)

# Three-way train/val/test split
split_dataset("28D0", test_size=0.2, val_size=0.1, seed=42)

# Stratified split maintaining class balance
split_dataset("28D0", test_size=0.2,
             stratify_by_column="Image_Classification_Image_Class",
             include_tables=["Image", "Image_Classification"])

# Fixed-count split with labeled types
split_dataset("28D0", train_size=400, test_size=100,
             training_types=["Labeled"], testing_types=["Labeled"])

# Dry run to preview
split_dataset("28D0", test_size=0.2, dry_run=True)