aws-toolkit

module

v0.3.1 Latest Latest Go to latest Published: Nov 19, 2025 License: MIT

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/citadel2024/aws-toolkit

Links

Open Source Insights

README ¶

AWS Toolkit

A high-performance, production-ready Go library for Amazon Web Services (AWS) that provides utilities for working with AWS services like SQS (Simple Queue Service). This toolkit is designed to simplify the development of AWS applications with features like structured logging, error handling, and concurrency management.

Installation

go get github.com/citadel2024/aws-toolkit

Quick Start

See examples folder.

SQS

A high-performance, production-ready Go library for Amazon SQS (Simple Queue Service) that provides both publishing and consuming capabilities with advanced features like batching, graceful shutdown, error handling, and comprehensive observability.

Inspired by https://github.com/awslabs/amazon-sqs-java-temporary-queues-client.

Features

🚀 Publisher

Batch Processing: Automatic message batching for optimal throughput
Configurable Intervals: Time-based and size-based batch triggers
Graceful Shutdown: Clean shutdown with message draining
Error Handling: Comprehensive error handling with custom callbacks
Thread Safety: Safe for concurrent use across multiple goroutines
Observability: Structured logging with zerolog integration

📨 Consumer

Concurrent Processing: Multi-threaded message polling and processing
Error Classification: Distinguishes between retryable and non-retryable errors
Long Polling: Efficient long polling to reduce API calls
Graceful Shutdown: Coordinated shutdown of all processing goroutines
Custom Error Handling: Pluggable exception handlers

Configuration Options

Publisher Options

Option	Description	Default
`WithLogger(logger)`	Custom zerolog logger	`zerolog.Nop()`
`WithClient(client)`	Custom SQS client	AWS default config
`WithBatchMaxMessages(limit)`	Max messages per batch	10
`WithPublishInterval(interval)`	Batch publish interval	5 seconds
`WithSendingMessageTimeoutSeconds(timeout)`	Send timeout	30 seconds
`WithOnSendMessageBatchComplete(handler)`	Batch completion callback	Default error logging

Consumer Options

Option	Description	Default
`WithLogger(logger)`	Custom zerolog logger	`zerolog.Nop()`
`WithMessageHandler(handler)`	Message processing function	Required
`WithExceptionHandler(handler)`	Exception handling function	Default error logging
`WithPollingGoroutines(count)`	Number of polling goroutines	1
`WithProcessingConcurrency(count)`	Max concurrent message processors	10
`WithMaxMessagesPerBatch(count)`	Messages per receive call (1-10)	10
`WithWaitTimeSeconds(seconds)`	Long polling wait time (0-20)	20
`WithPollIntervalMilliseconds(interval)`	Delay between polling attempts (>=0 ms)	0
`WithShutdownHook(hook)`	Function called on shutdown	None

Advanced Usage

Architecture

Publisher Architecture

┌─────────────────┐    ┌──────────────┐    ┌─────────────┐
│   Application   │───▶│   Publisher  │───▶│  SQS Queue  │
└─────────────────┘    └──────────────┘    └─────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │  Batch Worker    │
                    │  • Time-based    │
                    │  • Size-based    │
                    │  • Error handling│
                    └──────────────────┘

Consumer Architecture

┌─────────────┐    ┌─────────────────┐    ┌──────────────────┐
│  SQS Queue  │◀───│ Polling Workers │───▶│ Processing Pool  │
└─────────────┘    │ (Configurable)  │    │ (Configurable)   │
                   └─────────────────┘    └──────────────────┘
                           │                        │
                           ▼                        ▼
                  ┌─────────────────┐    ┌──────────────────┐
                  │  Long Polling   │    │ Message Handler  │
                  │  • Batch fetch  │    │ • Error handling │
                  │  • Wait timeout │    │ • Retry logic    │
                  └─────────────────┘    └──────────────────┘

Performance Considerations

Publisher Optimization

Batch Size: Increase WithBatchMaxMessages for higher throughput
Publish Interval: Lower intervals for lower latency, higher for better batching
Buffer Size: Channel buffer is automatically sized based on batch limit

Consumer Optimization

Polling Workers: Increase for better queue draining with multiple consumers
Processing Concurrency: Tune based on message processing time and resources
Long Polling: Use maximum WithWaitTimeSeconds(20) to reduce API calls
Batch Size: Use maximum WithMaxMessagesPerBatch(10) for efficiency

Testing

make rapid_test
make fuzz_test
make unit_test

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Built on the official AWS SDK for Go v2
Logging powered by zerolog
Concurrency utilities from golang.org/x/sync

Directories ¶

Path	Synopsis
cloudwatchx Package cloudwatchx is a generated GoMock package.	Package cloudwatchx is a generated GoMock package.
examples
sqsconsumer command
sqspublisher command
secretsmanagerx Package secretsmanagerx is a generated GoMock package.	Package secretsmanagerx is a generated GoMock package.
sqsx Package sqsx is a generated GoMock package.	Package sqsx is a generated GoMock package.
consumer
publisher

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL