Skip to main content
Glama

Convert dataset to RAG documents

dataset_to_rag_documents
Read-onlyIdempotent

Convert scraped items into embedding-ready documents: token-bounded chunks with overlap, source attribution, stable content-hash IDs, and selected metadata. Paginate large datasets with offset/limit.

Instructions

Turn scraped items into embedding-ready documents: token-bounded chunks with overlap, source attribution, stable content-hash ids for idempotent vector upserts, and selected metadata. Paginate with offset/limit for large datasets.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
limitNo
formatNo"json" emits one JSON document per line (jsonl), ready for embedding pipelinesjson
offsetNo
dataset_idYes
content_fieldsNoItem fields to use as document text; auto-detects markdown/text/content when omitted
overlap_tokensNo
metadata_fieldsNoItem fields to carry into each document’s metadata
max_response_tokensNo
max_tokens_per_chunkNo

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
documentsYes
truncatedYes
dataset_idYes
next_offsetYes
total_tokensYes
skipped_itemsYes
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare readOnlyHint and idempotentHint, and the description adds useful behavioral context by explaining the mechanism for idempotency ('stable content-hash ids') and the chunking/overlap behavior. It does not go into edge cases or failure modes, but the combination of annotations and description is solid.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, highly informative, front-loaded with the core transformation, and no filler. Every clause adds meaning: input, output, chunking, idempotency, metadata, pagination.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (9 parameters, transformation, output schema exists), the description covers the main operational aspects: chunking, overlap, idempotency, metadata, and pagination. It leaves some ambiguity about 'selected metadata' and exact chunking behavior, but the output schema and annotations fill in enough gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is only 33%, so the description must compensate. It does explain the purpose of chunking, overlap, metadata selection, and pagination, but it does not explicitly map these to parameter names, and parameters like max_response_tokens and format are left to the reader's inference. This is useful but not complete compensation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb+resource transformation: 'Turn scraped items into embedding-ready documents'. It enumerates distinct output characteristics (token-bounded chunks, overlap, source attribution, content-hash ids, metadata) that clearly differentiate it from sibling tools like fetch_dataset_items or validate_dataset.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use the tool: after scraping, before embedding, and it advises pagination for large datasets. However, it does not explicitly name alternatives or state when not to use it, so it misses the explicit exclusion guidance that would earn a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/fctpe/web-data-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server