check_text_columns
Inspect text-heavy columns to detect near-duplicates, encoding artifacts, and content types with length and vocabulary stats.
Instructions
Analyze text-heavy string columns (avg length ≥ 20).
Per column: length distribution (avg/p50/p95/max), vocab size + top
tokens, near-duplicate %, content kind (prose/code/html/url/numeric/
categorical/mixed), encoding-artifact (mojibake) %. Flags high dedup
rate, mojibake, HTML content. Sample size cap on big data. Output size: small.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| sample | No | ||
| columns | No | ||
| source_id | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||