Convert dataset to RAG documents
dataset_to_rag_documentsConvert scraped items into embedding-ready documents: token-bounded chunks with overlap, source attribution, stable content-hash IDs, and selected metadata. Paginate large datasets with offset/limit.
Instructions
Turn scraped items into embedding-ready documents: token-bounded chunks with overlap, source attribution, stable content-hash ids for idempotent vector upserts, and selected metadata. Paginate with offset/limit for large datasets.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| format | No | "json" emits one JSON document per line (jsonl), ready for embedding pipelines | json |
| offset | No | ||
| dataset_id | Yes | ||
| content_fields | No | Item fields to use as document text; auto-detects markdown/text/content when omitted | |
| overlap_tokens | No | ||
| metadata_fields | No | Item fields to carry into each document’s metadata | |
| max_response_tokens | No | ||
| max_tokens_per_chunk | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| documents | Yes | ||
| truncated | Yes | ||
| dataset_id | Yes | ||
| next_offset | Yes | ||
| total_tokens | Yes | ||
| skipped_items | Yes |