List datasets currently held in memory: name, kind (table|tensor), shape, memory, parent, how they were made.

Load a file into memory as a table (rows x columns) or tensor (N-d labelled array). Returns a schema summary, never the data.

Formats by extension (or format): csv tsv txt parquet feather json jsonl xlsx npy npz mat h5/hdf5 nc zarr. options: pandas/reader keyword arguments, e.g. {"sep": ";", "index_col": 0, "parse_dates": ["date"]}, {"sheet": "Sheet2"}, {"key": "array_name"} for npz/mat/h5/nc with several arrays. Relative paths resolve against the first HIDIM_ROOTS entry (home by default). npz files written by save_dataset come back as tensors with their dims and coordinates intact.

Write a dataset to disk under HIDIM_ROOTS. Tables: csv parquet feather json npy xlsx mat. Tensors: npz (keeps dims+coords, reloadable), npy, nc (needs xarray), mat.

Remove a dataset from memory.

Create a synthetic dataset for demos and sanity checks.

kinds: blobs (k Gaussian clusters in d dims, label column), swiss_roll / s_curve (2-D manifold embedded in d dims, colour column t), gaussian (iid noise: the null model for the diagnostics), lowrank (rank-k matrix plus noise), options_surface (long table date x strike x maturity_days -> iv, spot, volume; for tensorize demos).

Budgeted profile of a dataset. For tables: schema, label columns, identifier-like and label-like integer columns, per-feature stats, missingness, constant and duplicated features, plus high-dimensional geometry on a sample: strongest correlations, effective rank, condition number, PCA components needed for 50/80/90/95/99% variance (on the chosen scaling AND on the other one), the number of correlation eigenvalues above the Marchenko-Pastur noise edge (how many factors carry more than noise), intrinsic dimension (TwoNN and MLE), distance concentration and hubness. standardize: 'auto' (default) keeps raw scales for data in one shared unit such as pixels / counts / [0,1] intensities and z-scores heterogeneous columns; True/False force it. For tensors: dims, coordinate ranges, fill rate per axis, value distribution. Ends with one consistent hint on what to do next.

Declare columns of a table as labels / coordinates (integer class codes, year, hour, lat, lon, ids): they are excluded from the feature matrix used by describe/project/cluster/neighbors and stay available for color_by and annotate. unset=True reverses it.

Preprocess the numeric features of a table into a new table (labels, ids and datetimes carried over unchanged).

ops: log1p, log (params offset), sqrt, abs, clip (lo/hi or z), winsorize (q), rank, zscore (params clip), minmax, demean, detrend (params by), diff / pct_change / log_return / cumsum along rows (params periods, by=, group_by, dropna), row_normalize (params target; library-size normalization for counts), center_rows (subtract each row's mean: removes the equal-weight market / common mode of a panel), deseasonalize (params by=, period=month|dayofweek|hour|dayofyear|quarter: subtracts per-period means -> anomalies), residualize (params on=[cols]: regress every feature on those columns and keep residuals, e.g. remove a market factor), dropna. Typical chains: counts -> row_normalize -> log1p -> project(clip_z=10); prices -> log_return -> describe; monthly series -> deseasonalize -> project(pca, standardize=False).

Filter / select / sample rows and columns of a table; preview a few rows; optionally store the result as new_name.

where: pandas query string, e.g. "iv > 0.3 and maturity_days <= 30" (backtick-quote odd column names). columns: list of names, "re:", or "top_variance:". top_variance keeps the k highest-variance numeric features. This is the only way to read raw rows; preview is capped (200 rows x 60 cols).

k nearest neighbours of one row (index = row label or integer position) or of an explicit point (vector in feature order), in the original high-dimensional space. Returns distances, positions and a few columns of each neighbour, plus how those distances compare with the distance to a random row (relative contrast). metric: any scikit-learn metric (euclidean, cosine, manhattan, ...); cosine is usually the right choice for embeddings and other hub-prone spaces. standardize: False (default), True or 'auto'.

Reduce a table's numeric features to n_components and store the embedding as a new table (name__method) with label, datetime and cluster columns carried over for colouring. Returns quality metrics, not coordinates.

methods: pca (explained variance, top loadings; full vectors via loadings()), svd, ica, random (Gaussian random projection), kpca, tsne, umap (needs umap-learn), isomap, lle, mds, spectral. Non-linear methods run on a random sample (default caps: tsne 5000, umap 50000, isomap/lle/spectral 5000, mds 2000; override with sample) and get an automatic PCA to 50 dims first when there are more than 50 features (pre_pca). params passes method-specific keywords (perplexity, n_neighbors, min_dist, kernel, metric, ...). standardize: 'auto' (default; raw scales for pixel/count/[0,1] data, z-scores otherwise), True or False. clip_z=10 clips z-scores so near-constant or sparse features cannot dominate. Every method reports trustworthiness (1.0 = neighbourhoods perfectly preserved). ascii=True adds a coarse text density map.

Loading vector of one principal component of a PCA embedding made by project(method='pca'): the strongest positive and negative features, or the whole vector in feature order (all_features=True) to read shapes such as level/slope/curvature.

Cluster a table (or an embedding) and add a label column (cluster_) usable as color_by in plot. If run on an embedding, labels are also written back to the parent table. methods: kmeans (k omitted -> silhouette over 2..k_max, with the whole silhouette-by-k table, the Calinski-Harabasz choice and the finest k within 10% of the best, because silhouette favours coarse splits), gmm (k by BIC), dbscan (eps from 5-NN distances), hdbscan, agglomerative. standardize 'auto'/True/False and clip_z as in project. Returns sizes (with a warning when clusters hold <0.5% of rows: outliers, not groups), silhouette, Davies-Bouldin and each cluster's centroid signature (features most displaced from the global mean, in SD units).

Condense a long/tidy table into a labelled N-d tensor: each column in axes becomes a dimension and each value column fills the cells; several value columns add a trailing 'variable' dimension. Coordinates: numeric/datetime axes sort numerically, text axes sort naturally (b2 before b10); sort=False keeps first-appearance order; order={'tenor': ['1m','3m','1y']} pins an explicit order per axis. Duplicate axis combinations are aggregated with agg (mean|sum|first|last|max|min|median|count); missing combinations become NaN. Note that a later reduce 'mean' over a dim averages cells, not underlying rows: to get a row-weighted mean, tensorize agg='sum' and agg='count' and divide. Example: tensorize('opts', axes=['date','strike','maturity_days'], values=['iv']) -> a date x strike x maturity volatility cube.

Turn a wide table whose column names encode several keys (AAPL_close, AAPL_volume, MSFT_close, ...) into a tensor index x dims..., splitting column names on sep. dims default to ['series', 'feature']. The wide table's column order is kept for the name-derived dims (so a tenor axis 1m,3m,6m,1y,2y stays in that order); sort=True sorts naturally instead, order={dim: [...]} pins an explicit order.

Preprocess a tensor into a new tensor. Elementwise: log1p, log (params offset), sqrt, abs, fillna (params value, or dim for a mean fill). Along dim: demean, zscore, diff / pct_change / log_return (params periods; the axis shrinks), cumsum, normalize (params target). anomaly (dim = a datetime axis, params period=month|dayofyear|dayofweek|hour|quarter, baseline=[start,end]) subtracts the per-period climatology -> anomalies. area_weight (dim = latitude axis in degrees, params power: 0.5 for PCA/EOF, 1 for means) multiplies by cos(lat)^power. Typical EOF chain: tensor_transform(op='anomaly', dim='time') -> tensor_transform(op='area_weight', dim='lat') -> tensor_to_table(row_dim='time') -> project(pca, standardize=False).

Flatten a tensor into a table: one row per coordinate of row_dim, one feature column per combination of the other dims (time x lat x lon -> time rows x gridpoint columns). All-NaN columns are dropped; the row coordinate becomes a label column. select/reduce (as in slice_tensor) are applied first. Then describe/project/cluster it like any table.

Stack several same-shaped tables (their numeric blocks) or tensors along a new leading dimension, e.g. one CSV per ticker -> ticker x row x column. Shapes and column sets must match exactly (use subset/tensorize to align first).

Read part of a tensor. select maps dim -> scalar (exact, or nearest for numeric/datetime axes), list (several values), {"range": [lo, hi]}, {"isin": [...]}, {"nearest": v}, {"iloc": [start, stop]}, {"positions": [...]}, {"every": n}. reduce maps dim -> mean|std|var|sum|min|max|median|count|first|last (NaN-aware). Scalar selections drop the dim. When 1 or 2 dims remain the values are returned as a table (evenly thinned if over max_cells, default 4000); with more dims you get summary stats and should select/reduce further. new_name stores the result (2-D or less becomes a table that describe/project/plot accept; otherwise a tensor).

Render a PNG picture (returned as an image you can look at; also saved under HIDIM_OUT) with a one-line caption.

Tables: scatter (x,y[,z]; defaults to the first components of an embedding; color_by a label/cluster/numeric/datetime column, parent columns of an embedding work too: categories get a legend of up to 20 classes, numeric and datetime columns a colour bar; annotate= writes that column's value next to each point, up to 300 points), pairs (scatter matrix of <=6 features), parallel (parallel coordinates, min-max scaled), corr (correlation heatmap with feature labels, reordered by hierarchical clustering so correlated blocks are visible; order='given' keeps table order), heatmap (raw value matrix rows x columns), scree (PCA variance curve, auto scaling), hist (small-multiple histograms), andrews (Andrews curves by class). Tensors: tensor (heatmap of a 2-D slice), surface (3-D surface of a 2-D slice; numeric and datetime axes are drawn at their true values, log_axes=['maturity_days'] draws a dim on a log10 scale), lines (1-D slices over one dim, up to 12 series from series_dim; use select={'dim': {'isin': [...]}} to pick). Use select/reduce exactly as in slice_tensor to get down to 2 dims (or 1-2 for lines). Large tables are randomly sampled to HIDIM_PLOT_MAX_POINTS (20000) points; categorical colours beyond HIDIM_MAX_CLASSES (20) fold into 'Other'.