Batch Chunker
Splits a DataTable into smaller in-memory batches of N rows for processing through downstream systems with rate limits or batch-size constraints.
Provided as-is, without warranty of any kind. Review and test each pattern in a non-production environment before deploying it to live automations. See our Terms.
Problem this solves
Rate limits and batch-size constraints on downstream systems.
Cross-references: CSV Splitter for writing batches to files. Parallel-Ready Batch Splitter for distributing work across machines. Large Datatable Streamer for reading huge files in chunks.
1Configure Batch Size
Sets the batch size and initializes counters.
Variables: varBatchSize (rows per batch), varCurrentMessage (logging), varBatchCount (counter — batches processed), varRowCounter (counter — rows in current batch), varTotalProcessed (counter — total rows processed)
2Chunk and Process Loop
Loops through the source DataTable, accumulates rows into a batch DataTable, and calls a processing block every time the batch reaches the configured size. After processing, the batch is cleared and the next chunk begins. The last partial batch is processed after the loop.
Variables: varSourceData (source DataTable to chunk), varBatchSize (rows per batch), varCurrentMessage (logging), varBatchTable (current batch DataTable), TransactionItem (loop iterator), varRowCounter (rows in current batch), varBatchCount (batch counter), varTotalProcessed (total rows counter)
3Batch Chunker with Rate Limit Delay
Same as Pattern 2 but with a configurable delay between batches. Use when calling rate-limited APIs.
Variables: Same as Pattern 2 plus varDelaySeconds (seconds to wait between batches)
Replace the WAIT $fx'=1' in Pattern 2 with:
Variable Reference Summary
| Variable | Type | Used In | Purpose |
| `varSourceData` | DataTable | Pattern 2 | Source DataTable to chunk |
| `varBatchSize` | Numeric | Patterns 1, 2 | Rows per batch |
| `varCurrentMessage` | Text | All patterns | Logging message for Subflow_Logging |
| `varBatchCount` | Numeric | Patterns 1, 2 | Batches processed counter |
| `varRowCounter` | Numeric | Patterns 1, 2 | Rows in current batch |
| `varTotalProcessed` | Numeric | Patterns 1, 2 | Total rows processed |
| `varBatchTable` | DataTable | Pattern 2 | Current batch being accumulated |
| `TransactionItem` | DataRow | Pattern 2 | Current row in loop |
| `varDelaySeconds` | Numeric | Pattern 3 | Seconds between batches |
Notes
- In-memory vs file-based. This pattern keeps batches in memory as DataTables — no files written. Use CSV Splitter or Parallel-Ready Batch Splitter when you need file output.
- Processing logic goes in TWO places. Insert your batch processing code both in the main IF block (full batches) and after the loop (final partial batch). Consider extracting the processing into a CALL to a subflow for DRY.
- ClearDataTable after each batch. Keeps memory flat — only one batch-worth of data in memory at a time.
- Rate limit delay (Pattern 3). For APIs that enforce N requests/minute, set
varDelaySecondsto spread batches across the window. A 500-row batch with 5-second delay = 6,000 rows/minute throughput. - No ON ERROR on SET or IF. Simple
ON ERROR REPEATform on any action-level statements you add to the processing block.
Dependencies
- CSV Splitter
- Parallel-Ready Batch Splitter