drop_duplicates
Removes duplicate rows from a DataFrame, optionally targeting specific columns and choosing which duplicates to retain (first, last, or none). Use early to prevent biased statistics and models.
Instructions
Remove duplicate rows. subset: columns to check (None = all). keep: 'first', 'last', or 'none' (drop all duplicates). Run early in pipeline, right after EDA. Duplicates inflate statistics and bias models. Example: drop_duplicates(subset=["col1","col2"], keep="first")
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| keep | No | first | |
| subset | No | ||
| df_name | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |