batch

Batch

The Batch transform component groups incoming messages into batches of a specified size for further processing.

Config Definition

class Batch extends Transform {
  fixed sourceName = "batch"
  size: UInt32
}

Attributes

I/O Types

• Input Type: any
• Output Type: chan any

Pkl Configuration Example

Basic Batch Transform

new Transforms.Batch {
  name = "example-batch"
  size = 100
}

Description

The Batch transform collects a specified number of incoming messages and processes them as a single unit (batch). This is useful for reducing the overhead of individual message processing or aggregating messages before further transformations or outputs.

Use Cases

1. Batch Processing
   • Efficiently handle large volumes of data by processing them in batches.
2. Data Aggregation
   • Aggregate data for downstream components that require grouped input.

Notes

• Ensure the size attribute is configured appropriately for your use case.
• The batch size should be optimized based on the downstream component's processing capabilities.