Documentation
¶
Overview ¶
Package batch processes SQS, Kinesis and DynamoDB Streams batches one record at a time and reports partial failures, so Lambda retries only the records that failed instead of the whole batch.
Turn on ReportBatchItemFailures in the event source mapping (FunctionResponseTypes: ReportBatchItemFailures). Without it, Lambda ignores the response: a batch with failed records counts as a success and the failed records are deleted from the queue or skipped in the stream.
How failures are handled depends on the source:
- SQS standard queues: every record is processed, and each failed record is reported.
- SQS FIFO queues: processing stops at the first failure, and that record and all later ones are reported, so their order is kept.
- Kinesis and DynamoDB Streams: processing stops at the first failure and only that record is reported. Lambda retries the stream from it, so the later records are processed again anyway, in order.
A record whose function panics counts as failed; the panic does not crash the invocation. If no record succeeds, the handler also returns an error matching ErrAllFailed, so Lambda counts the invocation as failed and backs off.
When the invocation gets within the timeout margin of its deadline (one second by default, see WithTimeoutMargin), the remaining records are reported as failed without being processed. That leaves time to return the response, so records that already succeeded are not retried.
To process each record at most once, call idempotency.Do inside the record function.
Index ¶
- Variables
- func DynamoDB(process func(context.Context, events.DynamoDBEventRecord) error, ...) ...
- func Kinesis(process func(context.Context, events.KinesisEventRecord) error, opts ...Option) ...
- func SQS(process func(context.Context, events.SQSMessage) error, opts ...Option) func(context.Context, events.SQSEvent) (events.SQSEventResponse, error)
- type Option
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrAllFailed is returned when every record in a non-empty batch failed // or was not processed. ErrAllFailed = errors.New("batch: no record succeeded") // ErrNotProcessed is passed to [OnError] for records reported as failed // without being processed: after an earlier failure in a FIFO queue, or // when the invocation ran out of time. ErrNotProcessed = errors.New("batch: record not processed") )
Functions ¶
func DynamoDB ¶
func DynamoDB( process func(context.Context, events.DynamoDBEventRecord) error, opts ...Option, ) func(context.Context, events.DynamoDBEvent) (events.DynamoDBEventResponse, error)
DynamoDB returns a Lambda handler for DynamoDB Streams events that calls process once per record, stops at the first failure and reports it.
func Kinesis ¶
func Kinesis( process func(context.Context, events.KinesisEventRecord) error, opts ...Option, ) func(context.Context, events.KinesisEvent) (events.KinesisEventResponse, error)
Kinesis returns a Lambda handler for Kinesis events that calls process once per record, stops at the first failure and reports it.
func SQS ¶
func SQS( process func(context.Context, events.SQSMessage) error, opts ...Option, ) func(context.Context, events.SQSEvent) (events.SQSEventResponse, error)
SQS returns a Lambda handler for SQS events that calls process once per message and reports the failed ones. FIFO queues are detected from the queue ARN and keep their order: see the package documentation.
Example ¶
Each order is charged at most once, even when SQS delivers its message twice, and only messages that fail are retried.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/nixmaldonado/lambda-kit-go/batch"
"github.com/nixmaldonado/lambda-kit-go/idempotency"
)
type Order struct {
ID string `json:"id"`
}
// Each order is charged at most once, even when SQS delivers its message
// twice, and only messages that fail are retried.
func main() {
// In production, use dynamodbstore.New instead.
idem := idempotency.New(idempotency.NewMemoryStore(), idempotency.WithKeyPrefix("charge"))
handler := batch.SQS(func(ctx context.Context, msg events.SQSMessage) error {
var o Order
if err := json.Unmarshal([]byte(msg.Body), &o); err != nil {
return err
}
_, err := idempotency.Do(ctx, idem, o.ID, func(context.Context) (struct{}, error) {
fmt.Println("charging", o.ID)
return struct{}{}, nil
})
if errors.Is(err, idempotency.ErrResponseNotSaved) {
// The charge happened; don't report the message as failed, or
// its redelivery could charge again. Log it instead.
return nil
}
return err
})
// In main: lambda.Start(handler)
// The same order arrives twice, plus a malformed message.
ev := events.SQSEvent{Records: []events.SQSMessage{
{MessageId: "m1", Body: `{"id":"o-1"}`},
{MessageId: "m2", Body: `{"id":"o-1"}`},
{MessageId: "m3", Body: `not json`},
}}
resp, _ := handler(context.Background(), ev)
fmt.Println("failed:", resp.BatchItemFailures)
}
Output: charging o-1 failed: [{m3}]
Types ¶
type Option ¶
type Option func(*config)
Option configures a batch handler.
func OnError ¶
OnError registers fn to be called for every record reported as failed, with the record's identifier (message ID or sequence number) and its error. Records reported without being processed get an error matching ErrNotProcessed. Use it to log failures: the handler response only carries identifiers. A panic in fn is ignored.
func WithTimeoutMargin ¶
WithTimeoutMargin sets how long before the invocation deadline the handler stops starting new records, leaving time to return the response. The default is one second. Zero stops only once the deadline has passed, which is too late to respond. It panics if d is negative.