Load a CSV file into memory. The name defaults to the filename. This becomes the current active dataframe. Always the first step in any pipeline. After loading, immediately run quality_report to understand the data.

Save a dataframe to a CSV file (without row index). Use at the end of a pipeline to export cleaned/processed data.

List all loaded dataframes with their shapes. Check what datasets are available before switching or merging.

Switch the current active dataframe. Switch active dataset when working with multiple dataframes.

Create a deep copy of a dataframe under a new name. Use BEFORE any destructive operation (dropping, filtering, encoding) if you may need the original data later.

Merge two dataframes (like SQL JOIN). How: 'inner', 'left', 'right', 'outer'. Result is stored as a new dataframe. Use when data is split across files. Always verify join keys exist in both dataframes with get_info first.

Create a pivot table. Stores result as a new dataframe. Creates summary tables. Use for reporting or reshaping data from long to wide format. Example: pivot_table(index=["City"], columns="Category", values="Revenue", agg_func="mean")

Unpivot (wide to long format). id_vars are columns to keep, value_vars are columns to melt. Reverse of pivot. Use when data is in wide format but tools expect long format. Example: melt_dataframe(id_vars=["Name","City"], value_vars=["Score","Revenue"])

Concatenate multiple dataframes along rows (axis=0) or columns (axis=1). Combine train and test sets back together, or append new data. Example: concat_dataframes(names=["data_train", "data_test"], axis=0)

Load an Excel file (.xlsx, .xls) into memory. If sheet_name is empty, loads the first sheet. The name defaults to the filename. Requires openpyxl package for .xlsx files. Example: load_excel(file_path="data.xlsx", sheet_name="Sheet1")

Load a Parquet file into memory. Parquet is columnar and much faster than CSV for large files. The name defaults to the filename. Example: load_parquet(file_path="data.parquet")

Return the first N rows of the dataframe as a formatted table. Quick first look after loading. Verify data loaded correctly and understand structure. Example: get_head(n=10)

Return the last N rows of the dataframe as a formatted table. Check the last rows. Useful for time-series to see the most recent data. Example: get_tail(n=10)

Get column dtypes, non-null counts, and memory usage for the dataframe. Similar to pandas df.info() output. Check column types and non-null counts. Run when a tool reports 'column not found' to verify names.

Get descriptive statistics for all columns (numeric and categorical). Returns count, mean, std, min, quartiles, max for numeric; count, unique, top, freq for categorical. Understand distributions. Run after quality_report to decide cleaning strategies.

Get the number of rows and columns in the dataframe. Quick dimension check. Always run after filtering or dropping to verify not too many rows lost.

Comprehensive data quality report: dtype per column, missing value count and %, unique value count, and total duplicate rows. ESSENTIAL FIRST STEP after loading data. Run before any cleaning or transformation. Reveals the full data quality landscape in one call.

List unique values with their frequencies for a column, sorted by count descending. Understand categorical columns before encoding. Reveals rare categories that may need grouping. Example: get_unique_values(column="CargoType", top_n=20)

Detailed single-column analysis: dtype, nulls, unique count, statistics (if numeric: min/max/mean/median/std/skew/kurtosis), and top 10 value frequencies. Deep-dive into one column. Use when quality_report flags something. Provides skewness for deciding if log_transform is needed. Example: get_column_profile(column="Revenue")

Return a random sample of N rows. Use when head/tail are not representative. Random sampling reveals data variety better. Example: sample_data(n=5)

Remove duplicate rows. subset: columns to check (None = all). keep: 'first', 'last', or 'none' (drop all duplicates). Run early in pipeline, right after EDA. Duplicates inflate statistics and bias models. Example: drop_duplicates(subset=["col1","col2"], keep="first")

Drop specified columns from the dataframe. Remove useless columns (constants, IDs, leaked features). Always understand a column before dropping it. Example: drop_columns(columns=["col1","col2"])

Drop rows with missing values. how: 'any' (any null) or 'all' (all null). subset: limit check to specific columns (None = all). Use when entire rows are invalid. For partial missingness, prefer fill_missing instead. Example: drop_missing(subset=["col1"], how="any")

Fill missing values. Strategies: 'value' (literal), 'mean', 'median', 'mode', 'ffill', 'bfill'. columns: which columns to fill (None = all columns with nulls). Choose strategy based on distribution: 'median' for skewed numeric, 'mean' for normal, 'mode' for categorical. 'ffill'/'bfill' only for time-series. Example: fill_missing(columns=["Revenue"], strategy="median")

Filter rows by condition. Operators: ==, !=, >, <, >=, <=, contains, isin. For isin: provide comma-separated values. Value is auto-cast for numeric columns. Use to subset data or remove invalid rows. WARNING: modifies in-place. Use copy_dataframe first if you need the full dataset later. Example: filter_rows(column="Revenue", operator=">", value="1000") Example: filter_rows(column="CargoType", operator="isin", value="GCR,SCR")

Rename columns using an old->new mapping. Standardize column names early in the pipeline (snake_case, no spaces, explicit names). Prevents reference errors throughout the pipeline. Example: rename_columns(mapping={"old_name": "new_name"})

Clip outlier values in a numeric column. Methods: 'iqr' (Q1 - multiplierIQR, Q3 + multiplierIQR) or 'quantile' (lower/upper quantile). Use AFTER detect_outliers confirms outliers exist. Only clip if outliers are errors; genuine extreme values should be kept. Example: clip_outliers(column="Revenue", method="iqr", iqr_multiplier=1.5)

Sort the dataframe by one or more columns. Sort before visual inspection or time-series analysis. Useful before plot_line. Example: sort_values(columns=["Revenue"], ascending=False)

Discretize a numeric column into bins/categories. Methods: 'quantile' (equal-frequency bins), 'uniform' (equal-width bins), 'custom' (provide bin edges as list). For custom bins, pass bins as a list of edges, e.g. [0, 100, 1000, float('inf')]. Example: bin_column(column="Revenue", bins=3, method="quantile", labels=["low","medium","high"]) Example: bin_column(column="Revenue", bins=[0,500,5000,1e9], method="custom", labels=["small","medium","large"])

Create a new column using a pandas eval expression. Columns are referenced by name. For domain-relevant feature engineering: ratios, differences, interactions. Always add epsilon (1e-5) to denominators to avoid division by zero. Example: create_column(new_column="Yield", expression="Revenue / (ChargeableWeight + 1e-5)") Example: create_column(new_column="WeightPerPiece", expression="ChargeableWeight / (Pieces + 1e-5)")

Apply log transform to columns. Creates new columns named Log_{column}. Methods: 'log1p' (recommended, handles zeros), 'log' (natural log), 'log10'. Use on right-skewed distributions (|skewness| > 1). 'log1p' is safest (handles zeros). Check result with plot_histogram. Example: log_transform(columns=["Revenue","ChargeableWeight"], method="log1p")

Scale numeric columns in-place. Methods: 'minmax' (0-1), 'standard' (z-score), 'robust' (IQR-based). Only needed for distance-based models (linear, logistic). Tree-based models do NOT need normalization. Use 'standard' by default, 'robust' if outliers remain. Example: normalize(columns=["Revenue","Weight"], method="standard")

Map values in a column using a dictionary. Unmapped values stay as-is. Values are auto-converted to match the column's type. Map categorical values to new values. Useful for translating codes, merging similar categories, or creating ordinal mappings. Example: apply_mapping(column="FlownMonth", mapping={"SEPTEMBER":"9","OCTOBER":"10"})

Convert column types. Dtypes: 'int', 'float', 'str', 'category', 'datetime', 'bool'. Fix incorrect types (numeric stored as string). Do this BEFORE statistical analysis or encoding. Example: convert_dtype(columns=["age","score"], dtype="float")

Replace a specific value with another in a column. Types are auto-cast. Fix known data entry errors or standardize values. Use after identifying issues with get_unique_values. Example: replace_values(column="Status", old_value="N/A", new_value="Unknown")

Keep only columns matching specified dtypes. Drops non-matching columns. Filter columns by type. Use include=['number'] before modeling to keep only numeric features. Example: select_dtypes(include=["number"]) — keeps only numeric columns. Example: select_dtypes(exclude=["object"]) — drops string columns.

Clean a string column. Operations applied in order: 'strip', 'lower', 'upper', 'title', 'replace'. For 'replace': uses replace_old and replace_new parameters. Run on text columns BEFORE any encoding. Inconsistent casing or whitespace creates spurious categories. Example: string_clean(column="Name", operations=["strip","lower"])

Create polynomial and interaction features from numeric columns. degree=2 creates x^2 and x1*x2 terms. interaction_only=True skips powers (x^2). Use when linear models underfit: polynomial features capture non-linear relationships. Example: polynomial_features(columns=["Revenue","Weight"], degree=2)

One-hot encode categorical columns (creates binary columns). drop_first=True avoids multicollinearity (recommended for modeling). Best for low-cardinality (2-10 categories). Always drop_first=True for modeling. WARNING: high-cardinality columns create too many columns — use target_encode instead. Example: one_hot_encode(columns=["CargoType"], drop_first=True)

Target-encode high-cardinality categorical columns. Each category is replaced by the smoothed mean of the target variable. Good for columns with many unique values. Best for high-cardinality (>10 categories). IMPORTANT: to avoid data leakage, split data first with train_test_split, then encode training set only. Example: target_encode(columns=["AgentCode","OriginCode"], target_column="Log_Revenue", smoothing=10.0)

Label-encode categorical columns: map each unique value to an integer (0, 1, 2, ...). Use ONLY for ordinal data with natural order (e.g., low/medium/high, small/large). For nominal categories without order, use one_hot_encode or target_encode. Example: label_encode(columns=["City","Category"])

Replace each category with its frequency (count / total rows). Quick encoding that preserves frequency information. Useful for initial exploration or when no clear target variable exists. Example: frequency_encode(columns=["City","Category"])

Plot a histogram for a numeric column with optional KDE overlay and log-scaled x-axis. Use during EDA to understand numeric distributions. Set log_scale=True for skewed data (revenue, prices) — otherwise histogram is unreadable. Check for bimodality indicating subgroups. Example: plot_histogram(column="Revenue", bins=50, kde=True, log_scale=True)

Bar plot of value counts (top N categories). Vertical or horizontal. Understand categorical distributions during EDA. Reveals rare categories for potential grouping and dominant categories for stratified sampling. Example: plot_bar(column="CargoType", top_n=10, orientation="horizontal")

Scatter plot between two numeric columns. Optional color grouping by hue column. Explore relationships between two numeric variables. Add hue for categorical grouping to reveal subgroup patterns. Example: plot_scatter(x="Revenue", y="Weight", hue="CargoType")

Box plot for outlier visualization. Optional grouping by a categorical column. Visualize outliers and compare distributions across groups. Complements detect_outliers with a visual perspective. Example: plot_box(column="Revenue", by="CargoType")

Correlation heatmap with lower triangle mask and annotations. Methods: pearson, spearman, kendall. Empty columns = all numeric. Run after all numeric features are prepared. Identifies multicollinearity (features correlated with each other) and features correlated with the target. Example: plot_correlation_matrix(method="pearson")

Seaborn pairplot for up to 6 columns. Shows distributions on diagonal and scatter plots on off-diagonal. Limit to 6 columns for readability. Expensive computation. Use after feature_importance to focus on top features. Example: plot_pairplot(columns=["Revenue","Weight","Score"], hue="Category")

Heatmap showing missing value patterns across all columns. White = present, colored = missing. Use when quality_report shows significant missingness. Patterns in the heatmap reveal if data is missing randomly or systematically. Example: plot_missing_values()

Line plot. Supports multiple y columns overlaid. Essential for time-series and trends. For time-series data and trends. Sort data by x-axis column first for meaningful results. Supports multiple y columns overlaid. Example: plot_line(x="date", y="Revenue") Example: plot_line(x="month", y=["Revenue", "Cost"], hue="Category")

Violin plot: combines box plot with KDE to show full distribution shape. Better than boxplot for skewed or multimodal distributions. Optional grouping. Example: plot_violin(column="Revenue", by="CargoType")

Q-Q plot to visually assess if a column follows a normal distribution. Points on the diagonal = normal. Deviations show skewness or heavy tails. Use alongside normality_test for visual confirmation. Example: plot_qq(column="Revenue")

Stacked bar chart showing composition of one categorical within another. normalize=True shows percentages (100% stacked), False shows raw counts. Example: plot_stacked_bar(column="ProductCode", by="CargoType", normalize=True)

Heatmap of aggregated values in a pivot table format. Shows a color-coded matrix of one metric across two categorical dimensions. Example: plot_heatmap(index_col="OriginCode", columns_col="FlownMonth", values_col="Revenue", agg_func="sum")

Overlay KDE distributions of a numeric column for different groups. Better than separate histograms for comparing distribution shapes across categories. Example: plot_distribution_comparison(column="Revenue", by="CargoType")

Cumulative distribution function (CDF) plot. Shows what percentage of data falls below each value. Useful for understanding thresholds. Example: plot_cumulative(column="Revenue")

Compute pairwise correlation between two numeric columns. Methods: pearson, spearman, kendall. Returns value and interpretation. Measure relationship between two numeric columns. Use for targeted investigation after plot_correlation_matrix reveals interesting pairs. Example: get_correlation(col_a="Revenue", col_b="Weight", method="pearson")

Value counts with percentages, sorted descending. Shows top_n values. Detailed frequency analysis with percentages and cumulative percentages. More detailed than plot_bar. Example: get_value_counts(column="City", top_n=10)

Detect outliers in a numeric column. Methods: 'iqr' (threshold=IQR multiplier) or 'zscore' (threshold=z-score cutoff). Returns count, %, and boundary values. Run BEFORE clip_outliers to quantify outliers and determine if they are errors or genuine. Do not automatically clip without this check. Example: detect_outliers(column="Revenue", method="iqr", threshold=1.5)

GroupBy aggregation. Functions: mean, sum, count, min, max, median, std. For segmented analysis: understanding how metrics differ across groups. Useful for business questions like 'average revenue by city'. Example: group_aggregate(group_by=["City"], agg_column="Revenue", agg_func="mean")

Cross-tabulation of two categorical columns. Optional normalization (percentages). Understand relationships between two categorical variables. Set normalize=True to see percentages instead of raw counts. Example: crosstab(index_col="City", columns_col="Category", normalize=True)

Add or reset a numeric row index as a column. Add a numeric identifier column. Useful after filtering to create unique IDs or reset row numbering. Example: add_row_index(column_name="row_id", start=1)

GroupBy with multiple columns and multiple aggregation functions in one call. Much more efficient than calling group_aggregate repeatedly. Functions: mean, sum, count, min, max, median, std. Example: group_aggregate_multi(group_by=["CargoType"], aggregations={"Revenue": ["mean","sum","count"], "Weight": ["mean","median"]})

Descriptive statistics per group: mean, median, std, min, max for each numeric column within each group. One call replaces many separate group_aggregate calls. If numeric_columns is empty, uses all numeric columns. Example: describe_by_group(group_column="CargoType", numeric_columns=["Revenue","ChargeableWeight"])

Split dataframe into train/test sets. Stores them as '{name}_train' and '{name}_test'. stratify=True preserves target class proportions (classification only). Run BEFORE training any model. Always use fixed random_state for reproducibility. Use stratify=True for imbalanced classification datasets. Example: train_test_split(target_column="Revenue", test_size=0.2)

Train a model. Types: 'linear_regression', 'logistic_regression', 'random_forest', 'gradient_boosting', 'decision_tree'. The model is stored for later predict/evaluate. Start with simple model (linear/logistic) as baseline. Only move to complex models (random_forest, gradient_boosting) if baseline is insufficient. Uses ALL numeric columns as features. Example: train_model(target_column="Revenue", model_type="random_forest")

Generate predictions on a dataframe using a stored model. Stores predictions as a new column. Generate predictions on new data. The dataframe must contain the same feature columns used during training. Example: predict(model_name="random_forest_data_train", df_name="data_test")

Evaluate a model on test data. Returns classification metrics (accuracy, precision, recall, F1, confusion matrix) or regression metrics (MAE, MSE, RMSE, R²). Always evaluate on TEST set, never training set. Compare metrics against baseline. Test score much worse than training = overfitting. Example: evaluate_model(model_name="random_forest_data_train", test_df_name="data_test")

List all trained models with their type and training metadata. Check which models have been trained and their training scores before deciding next steps.

K-fold cross-validation. Returns mean and std of scores across folds. More reliable than a single train/test split. Detects overfitting when train >> test scores. Example: cross_validate(target_column="Revenue", model_type="random_forest", n_folds=5)

Compare multiple model types using cross-validation. Returns a ranked table. Quick way to find the best model type before fine-tuning hyperparameters. Default models: linear_regression, random_forest, gradient_boosting (or classifiers if target is categorical). Example: compare_models(target_column="Revenue", model_types=["linear_regression","random_forest","gradient_boosting"])

Grid search for hyperparameter tuning. Tests all combinations and stores the best model. Returns the best parameters and score. The best model is saved for later predict/evaluate. Example: grid_search(target_column="Revenue", model_type="random_forest", param_grid={"n_estimators": [50,100,200], "max_depth": [5,10,20]})

Identify features with correlation to target below threshold. Returns list of low-importance columns with their correlation values. Does NOT auto-drop. Non-destructive: identifies low-correlation features but does NOT drop them. Review results before using drop_low_importance. Example: correlation_filter(target_column="Revenue", threshold=0.05, method="pearson")

Identify columns with variance at or below threshold (constant/near-constant). Returns list of low-variance columns. Does NOT auto-drop. Non-destructive: identifies constant/near-constant columns. These provide no useful information for modeling. Example: variance_filter(threshold=0.01)

Compute feature importance using tree-based model or mutual information. Methods: 'random_forest', 'mutual_info_classif', 'mutual_info_regression'. Returns ranked list of features with importance scores. Run after all encoding and feature engineering. Ranks features by predictive power. Helps focus modeling on most important features. Example: feature_importance(target_column="Revenue", method="random_forest", top_n=20)

Drop columns below a computed importance threshold. Methods: 'variance' (drop cols with variance <= threshold), 'correlation' (drop cols with abs correlation to target <= threshold). Destructive: actually drops columns. Use AFTER reviewing results from correlation_filter or variance_filter. Example: drop_low_importance(target_column="Revenue", method="correlation", threshold=0.05)

Extract datetime components as new columns ({column}_year, {column}_month, etc.). Parts: year, month, day, dayofweek, hour, minute, weekofyear, quarter, is_weekend. Use early in feature engineering when datetime columns exist. year, month, dayofweek, is_weekend are often highly predictive. Original column is preserved. Example: extract_datetime_parts(column="date", parts=["year", "month", "dayofweek", "is_weekend"])

Compute time difference between two datetime columns (column_a - column_b). Units: days, hours, minutes, seconds. Creates a new numeric column. Create duration features between two dates (e.g., delivery time, age, tenure). Result is a numeric column ready for modeling. Example: datetime_diff(column_a="end_date", column_b="start_date", new_column="duration_days", unit="days")

Filter rows by datetime range. Start/end as ISO strings (e.g., '2024-01-01'). Leave start or end empty for open-ended range. Subset data to a specific time period. WARNING: modifies in-place. Use copy_dataframe first if you need the full date range later. Example: datetime_filter(column="date", start="2024-01-01", end="2024-06-30")

Set a datetime column as the DataFrame index. Set datetime column as DataFrame index. Required for time-series resampling. Sorts by index automatically. Example: set_datetime_index(column="date")

Independent two-sample t-test. Compares means of a numeric column across two groups. Use to determine if the difference between two group means is statistically significant. Requires exactly 2 groups in group_column. For 3+ groups, use anova_test instead. Example: ttest_independent(column="Revenue", group_column="CargoType")

One-way ANOVA test. Compares means of a numeric column across 3+ groups. Use to determine if at least one group mean differs significantly from the others. For exactly 2 groups, ttest_independent is more appropriate. Example: anova_test(column="Revenue", group_column="FlownMonth")

Chi-square test of independence between two categorical columns. Tests whether two categorical variables are statistically associated or independent. Example: chi_square_test(col_a="CargoType", col_b="ProductCode")

Test if a numeric column follows a normal distribution. Methods: 'shapiro' (best for n < 5000), 'ks' (Kolmogorov-Smirnov, any sample size), 'dagostino' (D'Agostino-Pearson, n >= 20). Use before deciding on parametric vs non-parametric tests. If p < 0.05, data is NOT normal. Example: normality_test(column="Revenue", method="shapiro")

Mann-Whitney U test (non-parametric alternative to t-test). Compares distributions of a numeric column across two groups without assuming normality. Use when normality_test indicates non-normal data. Requires exactly 2 groups. Example: mann_whitney_test(column="Revenue", group_column="CargoType")

Kruskal-Wallis H test (non-parametric alternative to ANOVA). Compares distributions of a numeric column across 3+ groups without assuming normality. Use when normality_test indicates non-normal data. For 2 groups, use mann_whitney_test. Example: kruskal_wallis_test(column="Revenue", group_column="FlownMonth")

Bar plot of feature importance from a trained model. Works with tree-based models (random_forest, gradient_boosting, decision_tree). Use after train_model to understand which features drive predictions. Example: plot_feature_importance_model(model_name="random_forest_data_train")

Residuals vs predicted values plot for regression models. Use to diagnose model quality: random scatter = good, patterns = systematic error. Also shows a histogram of residuals to check normality. Example: plot_residuals(model_name="linear_regression_data_train", test_df_name="data_test")

Visual confusion matrix heatmap for classification models. Shows true vs predicted labels with counts. Use after evaluate_model for deeper diagnosis. Example: plot_confusion_matrix(model_name="rf_classifier", test_df_name="data_test")

ROC curve with AUC score for binary classification models. Plots the trade-off between true positive rate and false positive rate. AUC = 0.5 means random, AUC = 1.0 means perfect. Example: plot_roc_curve(model_name="logistic_data_train", test_df_name="data_test")

Precision-Recall curve for binary classification models. Better than ROC for imbalanced datasets. Shows trade-off between precision and recall. Example: plot_precision_recall_curve(model_name="logistic_data_train", test_df_name="data_test")

Learning curve: training and validation scores vs training set size. Diagnoses overfitting (gap between train/val) and underfitting (both scores low). Example: plot_learning_curve(model_name="rf_model", train_df_name="data_train")

Permutation importance: model-agnostic feature importance measured on test data. More reliable than built-in feature_importances_ because it measures actual impact on predictions. Shows importance with error bars from multiple random shuffles. Example: permutation_importance(model_name="rf_model", test_df_name="data_test")

K-Means clustering. Assigns each row to one of n_clusters groups based on numeric columns. Features are auto-standardized before clustering. Result is stored as a new column. Use for customer segmentation, anomaly grouping, or discovering natural data groups. Example: kmeans_cluster(columns=["Revenue","ChargeableWeight","Pieces"], n_clusters=4)

DBSCAN clustering. Density-based clustering that finds arbitrarily shaped clusters. Does not require specifying n_clusters. Labels outliers as -1. Features are auto-standardized. Use when clusters have irregular shapes or when you need outlier detection. Example: dbscan_cluster(columns=["Revenue","Weight"], eps=0.5, min_samples=5)

Elbow plot to find optimal number of clusters for K-Means. Plots inertia (within-cluster sum of squares) vs number of clusters. The 'elbow' point where the curve bends is the optimal K. Example: elbow_plot(columns=["Revenue","Weight"], max_k=10)

Compute silhouette score to evaluate clustering quality. Score ranges from -1 to 1: higher is better. >0.5 = good, >0.7 = excellent, <0.25 = poor. Run after kmeans_cluster or dbscan_cluster. Example: silhouette_score(cluster_column="cluster", feature_columns=["Revenue","Weight"])

Descriptive statistics per cluster: mean of each feature for each cluster. Use to understand what characterizes each cluster. If feature_columns is empty, uses all numeric columns. Example: cluster_profile(cluster_column="cluster", feature_columns=["Revenue","Weight","Pieces"])

PCA dimensionality reduction. Projects numeric columns onto principal components. New columns PC1, PC2, ... are added to the dataframe. Features are auto-standardized. If plot=True and n_components>=2, also stores a 2D scatter plot (viewable via save_report). Use to visualize high-dimensional data, reduce multicollinearity, or compress features. Example: pca_transform(columns=["Revenue","Weight","Pieces"], n_components=2, hue="CargoType")

t-SNE visualization. Non-linear dimensionality reduction for 2D visualization. Better than PCA for revealing clusters and local structure. Features are auto-standardized. Slower than PCA — best on datasets < 10,000 rows or use sample_data first. Example: tsne_plot(columns=["Revenue","Weight","Pieces"], perplexity=30, hue="CargoType")