read_file_chunked
Process large Excel or CSV files in sections to manage data within token constraints, enabling analysis of extensive datasets without exceeding processing limits.
Instructions
Read large files in manageable chunks to avoid token limits
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| filePath | Yes | Path to the CSV or Excel file | |
| sheet | No | Sheet name for Excel files (optional) | |
| chunkIndex | No | Chunk index to read (0-based, defaults to 0) | |
| chunkSize | No | Number of rows per chunk (optional, auto-calculated if not provided) |
Implementation Reference
- src/handlers/data-operations.ts:556-658 (handler)Main execution logic for the read_file_chunked tool. Handles chunked reading of large files by calculating optimal chunk sizes, validating boundaries, slicing data, managing headers for context, and providing comprehensive chunk metadata including navigation info.async readFileChunked(args: ToolArgs): Promise<ToolResponse> { try { if (!args.filePath) { return { content: [ { type: 'text', text: JSON.stringify({ success: false, error: 'Missing required parameter: filePath', }, null, 2), }, ], }; } const { filePath, sheet, chunkIndex = 0, chunkSize } = args; const result = await readFileContentWithWarnings(filePath, sheet); const totalRows = result.data.length; const totalColumns = result.data[0]?.length || 0; // Calculate optimal chunk size if not provided const optimalChunkSize = chunkSize || calculateOptimalChunkSize(totalRows, totalColumns); // Calculate offset for requested chunk const offset = chunkIndex * optimalChunkSize; if (offset >= totalRows) { return { content: [ { type: 'text', text: JSON.stringify({ success: false, error: `Chunk index ${chunkIndex} is out of range. File has ${Math.ceil(totalRows / optimalChunkSize)} chunks.`, }, null, 2), }, ], }; } // Validate and apply chunking const { validOffset, validLimit } = validateChunkBoundaries(result.data, offset, optimalChunkSize); const endRow = validOffset + validLimit; const chunkedData = result.data.slice(validOffset, endRow); // Include headers if this is the first chunk or if specifically requested let finalData = chunkedData; if (chunkIndex === 0 && totalRows > 0) { // First chunk already includes headers naturally } else if (chunkIndex > 0 && totalRows > 0) { // For subsequent chunks, optionally include headers as context finalData = [result.data[0], ...chunkedData.slice(validOffset === 0 ? 1 : 0)]; } const totalChunks = Math.ceil(totalRows / optimalChunkSize); const response: any = { success: true, data: finalData, chunkInfo: { chunkIndex, chunkSize: optimalChunkSize, totalChunks, totalRows, totalColumns, currentChunkRows: chunkedData.length, hasNext: chunkIndex < totalChunks - 1, hasPrevious: chunkIndex > 0, nextChunkIndex: chunkIndex < totalChunks - 1 ? chunkIndex + 1 : null, previousChunkIndex: chunkIndex > 0 ? chunkIndex - 1 : null, }, headers: totalRows > 0 ? result.data[0] : [], }; // Include warnings if they exist if (result.warnings) { response.warnings = result.warnings; } return { content: [ { type: 'text', text: JSON.stringify(response, null, 2), }, ], }; } catch (error) { return { content: [ { type: 'text', text: JSON.stringify({ success: false, error: error instanceof Error ? error.message : 'Unknown error occurred', }, null, 2), }, ], }; } }
- src/index.ts:250-275 (schema)JSON Schema defining the input parameters for the read_file_chunked tool: filePath (required), sheet, chunkIndex (default 0), chunkSize (optional).name: 'read_file_chunked', description: 'Read large files in manageable chunks to avoid token limits', inputSchema: { type: 'object', properties: { filePath: { type: 'string', description: 'Path to the CSV or Excel file', }, sheet: { type: 'string', description: 'Sheet name for Excel files (optional)', }, chunkIndex: { type: 'number', description: 'Chunk index to read (0-based, defaults to 0)', default: 0, }, chunkSize: { type: 'number', description: 'Number of rows per chunk (optional, auto-calculated if not provided)', }, }, required: ['filePath'], }, },
- src/index.ts:1215-1218 (registration)Registration of the read_file_chunked tool in the MCP CallToolRequestSchema handler switch statement, dispatching calls to DataOperationsHandler.readFileChunked method.case 'read_file_chunked': return await this.dataOpsHandler.readFileChunked(toolArgs); case 'get_file_info': return await this.dataOpsHandler.getFileInfo(toolArgs);
- src/utils/file-utils.ts:198-204 (helper)Utility function to compute optimal chunk size based on file dimensions and target token limit to prevent exceeding LLM context windows.export function calculateOptimalChunkSize(totalRows: number, totalColumns: number, targetTokens: number = 8000): number { const avgCellLength = 10; const tokensPerRow = Math.ceil((totalColumns * avgCellLength) / 4); const optimalRows = Math.floor(targetTokens / tokensPerRow); return Math.max(100, Math.min(optimalRows, 5000)); // Between 100 and 5000 rows }
- src/utils/file-utils.ts:206-217 (helper)Helper to validate and adjust chunk offset and limit to stay within file data bounds.export function validateChunkBoundaries(data: any[][], offset: number, limit: number): { validOffset: number; validLimit: number } { const totalRows = data.length; // Ensure offset is within bounds const validOffset = Math.max(0, Math.min(offset, totalRows - 1)); // Ensure limit doesn't exceed remaining data const remainingRows = totalRows - validOffset; const validLimit = Math.min(limit, remainingRows); return { validOffset, validLimit }; }