Documentation
¶
Index ¶
- Constants
- Variables
- func GetLocalStack(t testing.TB, envVars []string, readyFns ...func(port string) error) (port string)
- func GetSession(ctx context.Context, parsedConf *service.ParsedConfig, ...) (aws.Config, error)
- type S3StreamWriterStats
- type S3StreamingWriter
- type S3StreamingWriterConfig
- type WaitForSpaceResult
Constants ¶
View Source
const ( // ErrCodeKMSThrottlingException is defined in the API Reference // https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/#Kinesis.GetRecords ErrCodeKMSThrottlingException = "KMSThrottlingException" )
Variables ¶
View Source
var (
ErrLeaseNotAcquired = errors.New("the shard could not be leased due to a collision")
)
Common errors that might occur throughout checkpointing.
Functions ¶
func GetLocalStack ¶ added in v1.6.0
func GetLocalStack(t testing.TB, envVars []string, readyFns ...func(port string) error) (port string)
TODO: Add config + options pattern or use an already existing library like https://github.com/elgohr/go-localstack
func GetSession ¶
func GetSession(ctx context.Context, parsedConf *service.ParsedConfig, opts ...func(*config.LoadOptions) error) (aws.Config, error)
Types ¶
type S3StreamWriterStats ¶ added in v1.16.0
type S3StreamWriterStats struct {
TotalMessages int64
TotalBytes int64
PartsUploaded int32
Age time.Duration
LastWriteAge time.Duration
}
WriterStats contains statistics about the writer
type S3StreamingWriter ¶ added in v1.16.0
type S3StreamingWriter struct {
// contains filtered or unexported fields
}
S3StreamingWriter writes content incrementally to S3 using multipart uploads. It buffers bytes and uploads parts as they reach the S3 minimum part size (5MB).
func NewS3StreamingWriter ¶ added in v1.16.0
func NewS3StreamingWriter(config S3StreamingWriterConfig) (*S3StreamingWriter, error)
NewS3StreamingWriter creates a new streaming S3 writer
func (*S3StreamingWriter) Close ¶ added in v1.16.0
func (w *S3StreamingWriter) Close(ctx context.Context) error
Close finalizes the file by flushing remaining data and completing the upload
func (*S3StreamingWriter) Initialize ¶ added in v1.16.0
func (w *S3StreamingWriter) Initialize(ctx context.Context) error
Initialize starts the S3 multipart upload
func (*S3StreamingWriter) Stats ¶ added in v1.16.0
func (w *S3StreamingWriter) Stats() S3StreamWriterStats
Stats returns current writer statistics
func (*S3StreamingWriter) WriteBytes ¶ added in v1.16.0
func (w *S3StreamingWriter) WriteBytes(ctx context.Context, data []byte) error
WriteBytes adds bytes to the buffer and flushes when thresholds are reached
type S3StreamingWriterConfig ¶ added in v1.16.0
type S3StreamingWriterConfig struct {
S3Client s3StreamingAPI
Bucket string
Key string
MaxBufferBytes int64 // Maximum bytes to buffer before flushing (default: 10MB)
MaxBufferCount int // Maximum messages to buffer before flushing (default: 10000)
MaxBufferPeriod time.Duration // Maximum time to buffer before flushing (default: 10s)
ContentType string // Content type for S3 object
ContentEncoding string // Content encoding for S3 object (optional)
BackoffCtor func() backoff.BackOff
}
S3StreamingWriterConfig contains configuration for creating an S3StreamingWriter
type WaitForSpaceResult ¶ added in v1.16.0
type WaitForSpaceResult int
WaitForSpaceResult indicates the outcome of WaitForSpace.
const ( // WaitForSpaceOK indicates space is available. WaitForSpaceOK WaitForSpaceResult = iota // WaitForSpaceCancelled indicates the context was cancelled. WaitForSpaceCancelled // WaitForSpaceTimeout indicates the timeout was reached while waiting. WaitForSpaceTimeout )
Source Files
¶
- cache_dynamodb.go
- cache_s3.go
- client_factory.go
- input_kinesis.go
- input_kinesis_checkpointer.go
- input_kinesis_efo.go
- input_kinesis_pending_pool.go
- input_kinesis_record_batcher.go
- input_s3.go
- input_sqs.go
- integration_test_helpers.go
- metrics_cloudwatch.go
- output_dynamodb.go
- output_kinesis.go
- output_kinesis_firehose.go
- output_s3.go
- output_s3_stream.go
- output_s3_stream_writer.go
- output_sns.go
- output_sqs.go
- processor_dynamodb_partiql.go
- processor_lambda.go
- processor_s3.go
- session.go
Directories
Click to show internal directories.
Click to hide internal directories.