kinesis

package
v1.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 20, 2026 License: MIT Imports: 36 Imported by: 0

README

Kinesis

Parity grade: A · SDK aws-sdk-go-v2/service/kinesis@v1.43.2 · last audited 2026-07-13 (2b2086c9)

Coverage

Metric Value
Operations audited 39 (39 ok)
Feature families 4 (4 ok)
Known gaps 5
Deferred items 2
Resource leaks clean
Known gaps
  • KMSAccessDeniedException / KMS key existence validation for StartStreamEncryption/UpdateMaxRecordSize KeyId: not modeled. Real Kinesis StartStreamEncryption exceptions per the SDK model are InvalidArgumentException/LimitExceededException/ResourceInUseException/ResourceNotFoundException/AccessDeniedException — there is no KMS-specific exception in the Kinesis API itself, and validating a KeyId against the kms service backend would require a cross-service dependency out of scope for this pass (services/kinesis/ only). (bd: gopherstack-ud2)
  • UpdateStreamMode does not reshard when switching PROVISIONED -> ON_DEMAND (or back); AWS auto-adjusts shard count on mode transitions based on throughput history, which this in-memory emulator has no model for. Low priority: consumers of UpdateStreamMode generally re-describe the stream afterward. (bd: gopherstack-ud2)
  • GetShardIterator/SubscribeToShard AT_TIMESTAMP with a zero/omitted Timestamp is not rejected with ValidationException (silently treated as position 0). Minor; no test exercises this AWS edge case. (bd: gopherstack-ud2)
  • ListShards ShardFilter AT_TIMESTAMP/FROM_TIMESTAMP approximated as 'include closed+open' rather than true timestamp-bounded shard-lineage filtering (would need per-shard closed-at timestamps, which are not tracked). (bd: gopherstack-ud2)
  • CORRECTED this pass: the previous gap entry claiming resource policies (PutResourcePolicy/GetResourcePolicy/DeleteResourcePolicy) are lost across a persistence restart was stale/incorrect. persistence.go's backendSnapshot already has a ResourcePolicies field wired into both Snapshot (line ~60) and Restore (line ~119), and TestInMemoryBackend_FullStateSnapshotRestoreRoundTrip already exercises PutResourcePolicy through an actual snapshot/restore cycle and passes. No code change needed; this was a documentation-only correction.
Deferred
  • Enhanced fan-out SubscribeToShard real streaming cadence / HTTP2 push semantics beyond the polling emulation already in place
  • Cross-service Lambda event-source-mapping trigger wiring (lives in cli.go per task constraints; not touched)

More

Documentation

Index

Constants

View Source
const (

	// StreamModeProvisioned is the PROVISIONED stream mode.
	StreamModeProvisioned = "PROVISIONED"
	// StreamModeOnDemand is the ON_DEMAND stream mode.
	StreamModeOnDemand = "ON_DEMAND"
)

Variables

View Source
var (
	ErrStreamNotFound                = awserr.New("ResourceNotFoundException", awserr.ErrNotFound)
	ErrStreamAlreadyExists           = awserr.New("ResourceInUseException", awserr.ErrAlreadyExists)
	ErrInvalidArgument               = awserr.New("InvalidArgumentException", awserr.ErrInvalidParameter)
	ErrUnknownAction                 = errors.New("UnknownOperationException")
	ErrShardIteratorExpired          = errors.New("ExpiredIteratorException")
	ErrConsumerNotFound              = awserr.New("ResourceNotFoundException", awserr.ErrNotFound)
	ErrConsumerAlreadyExists         = awserr.New("ResourceInUseException", awserr.ErrAlreadyExists)
	ErrResourcePolicyNotFound        = awserr.New("ResourceNotFoundException", awserr.ErrNotFound)
	ErrProvisionedThroughputExceeded = awserr.New(
		"ProvisionedThroughputExceededException",
		errRateExceeded,
	)
	ErrTagLimitExceeded = awserr.New("LimitExceededException", awserr.ErrInvalidParameter)
	ErrLimitExceeded    = awserr.New("LimitExceededException", awserr.ErrInvalidParameter)
)

Sentinel errors for Kinesis operations.

View Source
var ErrNilAppContext = errors.New("kinesis: AppContext must not be nil")

ErrNilAppContext is returned by Init when the AppContext is nil.

View Source
var ErrShardCountScaling = errors.New(
	"UpdateShardCount cannot scale by more than double or less than half " +
		"of the current shard count within a single call",
)

ErrShardCountScaling indicates that an UpdateShardCount target fell outside the AWS per-call scaling window: within a single call the target shard count may not be more than double or less than half of the current open shard count. AWS surfaces this as ValidationException. The sentinel's message is the error text returned to the client.

View Source
var ErrValidation = errors.New("kinesis: validation error")

ErrValidation is the sentinel error for Kinesis input validation failures.

Functions

This section is empty.

Types

type ConfigProvider

type ConfigProvider interface {
	GetKinesisSettings() Settings
}

ConfigProvider is a private interface to extract Kinesis configuration from the abstract AppContext Config.

type Consumer

type Consumer struct {
	ConsumerCreationTimestamp time.Time `json:"consumerCreationTimestamp"`
	ConsumerName              string    `json:"consumerName"`
	ConsumerARN               string    `json:"consumerARN"`
	ConsumerStatus            string    `json:"consumerStatus"`
	StreamARN                 string    `json:"streamARN"`
}

Consumer represents a registered Kinesis enhanced fan-out consumer.

type CreateStreamInput

type CreateStreamInput struct {
	StreamName string
	Region     string
	AccountID  string
	StreamMode string
	ShardCount int
}

CreateStreamInput is the input for CreateStream.

type DecreaseStreamRetentionPeriodInput

type DecreaseStreamRetentionPeriodInput struct {
	StreamName           string
	RetentionPeriodHours int
}

DecreaseStreamRetentionPeriodInput is the input for DecreaseStreamRetentionPeriod.

type DeleteResourcePolicyInput

type DeleteResourcePolicyInput struct {
	ResourceARN string
}

DeleteResourcePolicyInput is the input for DeleteResourcePolicy.

type DeleteStreamInput

type DeleteStreamInput struct {
	StreamName string
}

DeleteStreamInput is the input for DeleteStream.

type DeregisterStreamConsumerInput

type DeregisterStreamConsumerInput struct {
	StreamARN    string
	ConsumerARN  string
	ConsumerName string
}

DeregisterStreamConsumerInput is the input for DeregisterStreamConsumer.

type DescribeAccountSettingsOutput

type DescribeAccountSettingsOutput struct {
	ShardLimit               int
	OnDemandStreamCount      int
	OnDemandStreamCountLimit int
}

DescribeAccountSettingsOutput is the output for DescribeAccountSettings.

type DescribeStreamConsumerInput

type DescribeStreamConsumerInput struct {
	StreamARN    string
	ConsumerARN  string
	ConsumerName string
}

DescribeStreamConsumerInput is the input for DescribeStreamConsumer.

type DescribeStreamConsumerOutput

type DescribeStreamConsumerOutput struct {
	ConsumerDescription Consumer
}

DescribeStreamConsumerOutput is the output for DescribeStreamConsumer.

type DescribeStreamInput

type DescribeStreamInput struct {
	StreamName string
	// ExclusiveStartShardID resumes shard pagination after the given shard ID.
	ExclusiveStartShardID string
	// Limit caps the number of ShardDescription entries returned (AWS default
	// 100, max 10000). Zero means "use the AWS default".
	Limit int
}

DescribeStreamInput is the input for DescribeStream.

type DescribeStreamOutput

type DescribeStreamOutput struct {
	StreamCreationTimestamp time.Time
	StreamName              string
	StreamARN               string
	StreamStatus            string
	EncryptionType          string
	StreamMode              string
	KeyID                   string
	Shards                  []ShardDescription
	EnhancedMonitoring      []string
	RetentionPeriodHours    int
	// HasMoreShards indicates the shard list was truncated by Limit and more
	// shards can be fetched with a follow-up call using ExclusiveStartShardID.
	HasMoreShards bool
}

DescribeStreamOutput is the output for DescribeStream.

type DisableEnhancedMonitoringInput

type DisableEnhancedMonitoringInput struct {
	StreamName        string
	ShardLevelMetrics []string
}

DisableEnhancedMonitoringInput is the input for DisableEnhancedMonitoring.

type DisableEnhancedMonitoringOutput

type DisableEnhancedMonitoringOutput struct {
	StreamName               string
	CurrentShardLevelMetrics []string
	DesiredShardLevelMetrics []string
}

DisableEnhancedMonitoringOutput is the output for DisableEnhancedMonitoring.

type EnableEnhancedMonitoringInput

type EnableEnhancedMonitoringInput struct {
	StreamName        string
	ShardLevelMetrics []string
}

EnableEnhancedMonitoringInput is the input for EnableEnhancedMonitoring.

type EnableEnhancedMonitoringOutput

type EnableEnhancedMonitoringOutput struct {
	StreamName               string
	CurrentShardLevelMetrics []string
	DesiredShardLevelMetrics []string
}

EnableEnhancedMonitoringOutput is the output for EnableEnhancedMonitoring.

type GetRecordResult

type GetRecordResult struct {
	ApproximateArrivalTimestamp time.Time
	PartitionKey                string
	SequenceNumber              string
	Data                        []byte
}

GetRecordResult is a single record returned by GetRecords.

type GetRecordsInput

type GetRecordsInput struct {
	ShardIterator string
	Limit         int
}

GetRecordsInput is the input for GetRecords.

type GetRecordsOutput

type GetRecordsOutput struct {
	NextShardIterator  string
	Records            []GetRecordResult
	MillisBehindLatest int64
}

GetRecordsOutput is the output for GetRecords.

type GetResourcePolicyInput

type GetResourcePolicyInput struct {
	ResourceARN string
}

GetResourcePolicyInput is the input for GetResourcePolicy.

type GetResourcePolicyOutput

type GetResourcePolicyOutput struct {
	Policy string
}

GetResourcePolicyOutput is the output for GetResourcePolicy.

type GetShardIteratorInput

type GetShardIteratorInput struct {
	Timestamp              time.Time
	StreamName             string
	ShardID                string
	ShardIteratorType      string
	StartingSequenceNumber string
}

GetShardIteratorInput is the input for GetShardIterator.

type GetShardIteratorOutput

type GetShardIteratorOutput struct {
	ShardIterator string
}

GetShardIteratorOutput is the output for GetShardIterator.

type Handler

type Handler struct {
	Backend StorageBackend

	DefaultRegion string
	AccountID     string
	// contains filtered or unexported fields
}

Handler is the Echo HTTP handler for Kinesis operations.

func NewHandler

func NewHandler(backend StorageBackend) *Handler

NewHandler creates a new Kinesis Handler.

func (*Handler) ChaosOperations

func (h *Handler) ChaosOperations() []string

ChaosOperations returns all operations that can be fault-injected.

func (*Handler) ChaosRegions

func (h *Handler) ChaosRegions() []string

ChaosRegions returns all regions this Kinesis instance handles.

func (*Handler) ChaosServiceName

func (h *Handler) ChaosServiceName() string

ChaosServiceName returns the lowercase AWS service name for fault rule matching.

func (*Handler) ExecuteFISAction

func (h *Handler) ExecuteFISAction(ctx context.Context, action service.FISActionExecution) error

ExecuteFISAction executes a FIS action against resolved Kinesis targets.

func (*Handler) ExtractOperation

func (h *Handler) ExtractOperation(c *echo.Context) string

ExtractOperation extracts the Kinesis action from the X-Amz-Target header.

func (*Handler) ExtractResource

func (h *Handler) ExtractResource(c *echo.Context) string

ExtractResource extracts the stream name from the JSON request body.

func (*Handler) FISActions

func (h *Handler) FISActions() []service.FISActionDefinition

FISActions returns the FIS action definitions that the Kinesis service supports.

func (*Handler) GetSupportedOperations

func (h *Handler) GetSupportedOperations() []string

GetSupportedOperations returns the list of supported Kinesis operations.

func (*Handler) Handler

func (h *Handler) Handler() echo.HandlerFunc

Handler returns the Echo handler function for Kinesis operations.

func (*Handler) MatchPriority

func (h *Handler) MatchPriority() int

MatchPriority returns the routing priority for the Kinesis handler.

func (*Handler) Name

func (h *Handler) Name() string

Name returns the service name.

func (*Handler) Purge

func (h *Handler) Purge(ctx context.Context, cutoff time.Time)

Purge implements service.Purgeable by removing all Kinesis streams older than cutoff.

func (*Handler) Reset

func (h *Handler) Reset()

Reset clears all in-memory state from the backend. It is used by the POST /_gopherstack/reset endpoint for CI pipelines and rapid local development.

func (*Handler) Restore

func (h *Handler) Restore(ctx context.Context, data []byte) error

Restore implements persistence.Persistable by delegating to the backend.

func (*Handler) RouteMatcher

func (h *Handler) RouteMatcher() service.Matcher

RouteMatcher returns a function that matches incoming Kinesis requests.

func (*Handler) Snapshot

func (h *Handler) Snapshot(ctx context.Context) []byte

Snapshot implements persistence.Persistable by delegating to the backend.

func (*Handler) StartWorker

func (h *Handler) StartWorker(ctx context.Context) error

StartWorker starts the background janitor if one is configured.

func (*Handler) StopWorker

func (h *Handler) StopWorker()

StopWorker stops the background worker if one is configured.

func (*Handler) WithJanitor

func (h *Handler) WithJanitor(interval time.Duration, taskTimeout ...time.Duration) *Handler

WithJanitor attaches a background janitor to the handler. If the backend is not an *InMemoryBackend, this is a no-op.

type InMemoryBackend

type InMemoryBackend struct {
	OnStreamPurged func(string)
	// contains filtered or unexported fields
}

InMemoryBackend implements StorageBackend using in-memory maps.

Streams are keyed by a composite "region/name" key (see streamKey) inside a single flat store.Table, with a secondary store.Index grouping them by region — so same-named streams in different regions remain fully isolated, including their shards, records, and consumers, all of which stay inline fields on Stream (see pkgs/store's package doc: shards/records are the per-stream hot path and are not decomposed into their own tables). fisThroughputFaults and resourcePolicies are NOT store.Table candidates: their value types (a pointer with no self-describing identity, and a bare string) carry no key of their own to hand a Table keyFn, so they remain plain nested maps guarded the same way as before.

func NewInMemoryBackend

func NewInMemoryBackend() *InMemoryBackend

NewInMemoryBackend creates a new empty InMemoryBackend with default account/region.

func NewInMemoryBackendWithConfig

func NewInMemoryBackendWithConfig(accountID, region string) *InMemoryBackend

NewInMemoryBackendWithConfig creates a new InMemoryBackend with the given account ID and region.

func (*InMemoryBackend) AddStreamInternal

func (b *InMemoryBackend) AddStreamInternal(stream *Stream)

AddStreamInternal seeds a stream directly into the backend for testing. Caller must provide a non-nil stream with at least Name and ARN set. The stream is placed in the region encoded in its ARN, falling back to the backend's default region when the ARN carries none.

func (*InMemoryBackend) CountOpenShards

func (b *InMemoryBackend) CountOpenShards(ctx context.Context) int

CountOpenShards returns the total number of open (non-closed) shards across every stream in the region carried on ctx. DescribeLimits is region-scoped in AWS, so this counts within a single region.

func (*InMemoryBackend) CreateStream

func (b *InMemoryBackend) CreateStream(ctx context.Context, input *CreateStreamInput) error

CreateStream creates a new Kinesis stream.

func (*InMemoryBackend) DecreaseStreamRetentionPeriod

func (b *InMemoryBackend) DecreaseStreamRetentionPeriod(
	ctx context.Context,
	input *DecreaseStreamRetentionPeriodInput,
) error

DecreaseStreamRetentionPeriod decreases the retention period for a stream. Mirroring IncreaseStreamRetentionPeriod, a target equal to the current retention period is an idempotent no-op returning success (HTTP 200), matching real AWS behaviour. A target above the current period or below minRetentionHours (24h) is rejected with InvalidArgumentException.

func (*InMemoryBackend) DeleteResourcePolicy

func (b *InMemoryBackend) DeleteResourcePolicy(ctx context.Context, input *DeleteResourcePolicyInput) error

DeleteResourcePolicy removes the resource-based policy for the given stream or consumer ARN.

func (*InMemoryBackend) DeleteStream

func (b *InMemoryBackend) DeleteStream(ctx context.Context, input *DeleteStreamInput) error

DeleteStream removes a stream.

func (*InMemoryBackend) DeregisterStreamConsumer

func (b *InMemoryBackend) DeregisterStreamConsumer(ctx context.Context, input *DeregisterStreamConsumerInput) error

DeregisterStreamConsumer removes a registered consumer from a stream.

func (*InMemoryBackend) DescribeAccountSettings

func (b *InMemoryBackend) DescribeAccountSettings(ctx context.Context) (*DescribeAccountSettingsOutput, error)

DescribeAccountSettings returns account-level limits for this Kinesis account. The ON_DEMAND stream count is reported per region (AWS account-level limits are tracked per region), using the region carried on ctx.

func (*InMemoryBackend) DescribeStream

func (b *InMemoryBackend) DescribeStream(
	ctx context.Context,
	input *DescribeStreamInput,
) (*DescribeStreamOutput, error)

DescribeStream returns full stream details including shards.

func (*InMemoryBackend) DescribeStreamConsumer

func (b *InMemoryBackend) DescribeStreamConsumer(
	ctx context.Context,
	input *DescribeStreamConsumerInput,
) (*DescribeStreamConsumerOutput, error)

DescribeStreamConsumer returns details about a registered consumer. Lookup is by ConsumerARN, or by StreamARN + ConsumerName.

func (*InMemoryBackend) DisableEnhancedMonitoring

DisableEnhancedMonitoring removes shard-level metrics from a stream.

func (*InMemoryBackend) EnableEnhancedMonitoring

func (b *InMemoryBackend) EnableEnhancedMonitoring(
	ctx context.Context,
	input *EnableEnhancedMonitoringInput,
) (*EnableEnhancedMonitoringOutput, error)

EnableEnhancedMonitoring adds shard-level metrics to a stream.

func (*InMemoryBackend) GetRecords

func (b *InMemoryBackend) GetRecords(ctx context.Context, input *GetRecordsInput) (*GetRecordsOutput, error)

GetRecords retrieves records starting at the given shard iterator position.

The region is taken from the iterator token (encoded by GetShardIterator), not from ctx, so an iterator issued for one region always reads that region's records even if the GetRecords call carries a different ctx region.

func (*InMemoryBackend) GetResourcePolicy

func (b *InMemoryBackend) GetResourcePolicy(
	ctx context.Context,
	input *GetResourcePolicyInput,
) (*GetResourcePolicyOutput, error)

GetResourcePolicy retrieves the resource-based policy for the given stream or consumer ARN.

func (*InMemoryBackend) GetShardIterator

func (b *InMemoryBackend) GetShardIterator(
	ctx context.Context,
	input *GetShardIteratorInput,
) (*GetShardIteratorOutput, error)

GetShardIterator returns an iterator for reading records from a shard.

func (*InMemoryBackend) IncreaseStreamRetentionPeriod

func (b *InMemoryBackend) IncreaseStreamRetentionPeriod(
	ctx context.Context,
	input *IncreaseStreamRetentionPeriodInput,
) error

IncreaseStreamRetentionPeriod increases the retention period for a stream. A target equal to the current retention period is treated as an idempotent no-op returning success (HTTP 200), matching real AWS behaviour: the Terraform AWS provider calls IncreaseStreamRetentionPeriod on stream create for ANY configured retention_period > 0 (see the provider's resourceStreamCreate, guard `v.(int) > 0`), so a stream created with the default retention_period of 24h receives IncreaseStreamRetentionPeriod(24) against a stream already at 24h. Rejecting that equal value with InvalidArgumentException (as a strict reading of the SDK doc "Must be more than the current retention period" would suggest) breaks every default-retention Terraform stream, so real AWS accepts it. A target below the current period, below minRetentionHours (24h), or above maxRetentionHours (8760h) is rejected with InvalidArgumentException.

func (*InMemoryBackend) ListAll

func (b *InMemoryBackend) ListAll(_ context.Context) []StreamInfo

ListAll returns a snapshot of all streams as StreamInfo values across every region. It is used by the dashboard, which presents a global inventory.

func (*InMemoryBackend) ListShards

func (b *InMemoryBackend) ListShards(ctx context.Context, input *ListShardsInput) (*ListShardsOutput, error)

ListShards returns the shards for a stream.

func (*InMemoryBackend) ListStreamConsumers

func (b *InMemoryBackend) ListStreamConsumers(
	ctx context.Context,
	input *ListStreamConsumersInput,
) (*ListStreamConsumersOutput, error)

ListStreamConsumers lists all registered consumers for a stream.

func (*InMemoryBackend) ListStreams

func (b *InMemoryBackend) ListStreams(ctx context.Context, input *ListStreamsInput) (*ListStreamsOutput, error)

ListStreams returns stream names with optional pagination.

AWS contract: results are returned in alphabetical order. When `Limit` is set the response contains at most that many names. Pagination is keyed on either `ExclusiveStartStreamName` or the opaque `NextToken` (which we treat as the previously returned last stream name) so that callers can iterate over arbitrarily large account inventories.

func (*InMemoryBackend) ListTagsForResource

func (b *InMemoryBackend) ListTagsForResource(
	ctx context.Context,
	input *ListTagsForResourceInput,
) (*ListTagsForResourceOutput, error)

ListTagsForResource returns the tags associated with a stream identified by its ARN. Tags are those stored on the stream's internal Tags store (set via TagResource).

func (*InMemoryBackend) MergeShards

func (b *InMemoryBackend) MergeShards(ctx context.Context, input *MergeShardsInput) error

MergeShards merges two adjacent shards into one. The merged shard spans the combined hash key range of both parent shards.

func (*InMemoryBackend) Purge

func (b *InMemoryBackend) Purge(ctx context.Context, cutoff time.Time)

Purge removes all Kinesis streams and consumers created before the cutoff time.

func (*InMemoryBackend) PutRecord

func (b *InMemoryBackend) PutRecord(ctx context.Context, input *PutRecordInput) (*PutRecordOutput, error)

PutRecord writes a single record to a stream shard.

func (*InMemoryBackend) PutRecords

func (b *InMemoryBackend) PutRecords(ctx context.Context, input *PutRecordsInput) (*PutRecordsOutput, error)

PutRecords writes multiple records to a stream.

Request-level validation errors (empty/oversized batch, unknown stream) fail the whole call with a single top-level exception, matching the AWS contract: only per-record issues (throughput, per-record validation) surface as per-entry ErrorCode/ErrorMessage inside a 200 response.

func (*InMemoryBackend) PutResourcePolicy

func (b *InMemoryBackend) PutResourcePolicy(ctx context.Context, input *PutResourcePolicyInput) error

PutResourcePolicy stores a resource-based policy for the given stream or consumer ARN.

func (*InMemoryBackend) Region

func (b *InMemoryBackend) Region() string

Region returns the AWS region this backend is configured to use as its default.

func (*InMemoryBackend) RegisterStreamConsumer

func (b *InMemoryBackend) RegisterStreamConsumer(
	ctx context.Context,
	input *RegisterStreamConsumerInput,
) (*RegisterStreamConsumerOutput, error)

RegisterStreamConsumer registers a new enhanced fan-out consumer on a stream.

func (*InMemoryBackend) Reset

func (b *InMemoryBackend) Reset()

Reset clears all in-memory state from the backend. It is used by the POST /_gopherstack/reset endpoint for CI pipelines and rapid local development.

func (*InMemoryBackend) Restore

func (b *InMemoryBackend) Restore(ctx context.Context, data []byte) error

Restore loads backend state from a JSON snapshot. It implements persistence.Persistable.

func (*InMemoryBackend) Snapshot

func (b *InMemoryBackend) Snapshot(ctx context.Context) []byte

Snapshot serialises the backend state to JSON. It implements persistence.Persistable. Note: shard sequence number counters are now serialised via the NextSeq field.

func (*InMemoryBackend) SplitShard

func (b *InMemoryBackend) SplitShard(ctx context.Context, input *SplitShardInput) error

SplitShard splits a shard into two at the given new starting hash key.

func (*InMemoryBackend) StartStreamEncryption

func (b *InMemoryBackend) StartStreamEncryption(ctx context.Context, input *StartStreamEncryptionInput) error

StartStreamEncryption enables server-side encryption on a stream.

func (*InMemoryBackend) StopStreamEncryption

func (b *InMemoryBackend) StopStreamEncryption(ctx context.Context, input *StopStreamEncryptionInput) error

StopStreamEncryption disables server-side encryption on a stream.

func (*InMemoryBackend) SubscribeToShard

func (b *InMemoryBackend) SubscribeToShard(
	ctx context.Context,
	input *SubscribeToShardInput,
) (*SubscribeToShardOutput, error)

SubscribeToShard delivers records from a shard to an enhanced fan-out consumer. For mock purposes this is a single-shot delivery of all available records.

func (*InMemoryBackend) TagResource

func (b *InMemoryBackend) TagResource(ctx context.Context, input *TagResourceInput) error

TagResource adds or updates tags on a stream identified by its ARN. This is the ARN-based counterpart to AddTagsToStream.

func (*InMemoryBackend) UntagResource

func (b *InMemoryBackend) UntagResource(ctx context.Context, input *UntagResourceInput) error

UntagResource removes tags from a stream identified by its ARN. This is the ARN-based counterpart to RemoveTagsFromStream.

func (*InMemoryBackend) UpdateAccountSettings

func (b *InMemoryBackend) UpdateAccountSettings(_ context.Context, input *UpdateAccountSettingsInput) error

UpdateAccountSettings updates account-level settings such as the ON_DEMAND stream count limit.

func (*InMemoryBackend) UpdateMaxRecordSize

func (b *InMemoryBackend) UpdateMaxRecordSize(ctx context.Context, input *UpdateMaxRecordSizeInput) error

UpdateMaxRecordSize changes the per-record data payload size limit for a stream. The value must be between defaultMaxRecordSizeBytes (1 MiB) and absoluteMaxRecordSizeBytes (10 MiB).

func (*InMemoryBackend) UpdateShardCount

func (b *InMemoryBackend) UpdateShardCount(
	ctx context.Context,
	input *UpdateShardCountInput,
) (*UpdateShardCountOutput, error)

UpdateShardCount resizes a stream to the given number of shards. Existing records in the stream are not migrated; new shards start empty.

func (*InMemoryBackend) UpdateStreamMode

func (b *InMemoryBackend) UpdateStreamMode(ctx context.Context, input *UpdateStreamModeInput) error

UpdateStreamMode changes the mode of a stream identified by its ARN.

func (*InMemoryBackend) UpdateStreamWarmThroughput

func (b *InMemoryBackend) UpdateStreamWarmThroughput(
	ctx context.Context,
	input *UpdateStreamWarmThroughputInput,
) error

UpdateStreamWarmThroughput configures pre-warmed throughput for a stream. This is a no-op in the in-memory backend (no actual warm-up is needed).

type IncreaseStreamRetentionPeriodInput

type IncreaseStreamRetentionPeriodInput struct {
	StreamName           string
	RetentionPeriodHours int
}

IncreaseStreamRetentionPeriodInput is the input for IncreaseStreamRetentionPeriod.

type Janitor

type Janitor struct {
	Backend *InMemoryBackend

	Interval time.Duration
	// TaskTimeout bounds each individual janitor task. When non-zero, each task
	// runs with a child context that expires after this duration, preventing a
	// stalled operation from blocking the janitor loop indefinitely.
	TaskTimeout time.Duration
	// contains filtered or unexported fields
}

Janitor is the Kinesis background worker that enforces per-stream retention periods by evicting records older than stream.RetentionPeriod hours.

func NewJanitor

func NewJanitor(backend *InMemoryBackend, interval time.Duration) *Janitor

NewJanitor creates a new Kinesis Janitor for the given backend. A zero interval falls back to defaultJanitorInterval.

func (*Janitor) Run

func (j *Janitor) Run(ctx context.Context)

Run runs the janitor loop until ctx is cancelled.

func (*Janitor) Stop

func (j *Janitor) Stop()

Stop explicitly shuts down the janitor.

func (*Janitor) SweepOnce

func (j *Janitor) SweepOnce(ctx context.Context)

SweepOnce executes a single retention sweep. Exposed for testing.

type ListShardsInput

type ListShardsInput struct {
	StreamName            string
	NextToken             string
	ExclusiveStartShardID string
	// ShardFilter controls which shards are returned.
	// Supported values: "FROM_TRIM_HORIZON" (all shards including closed),
	// "AT_LATEST" (open shards only), "AFTER_SHARD_ID", "AT_TIMESTAMP", "FROM_TIMESTAMP".
	// Empty string defaults to open shards only.
	ShardFilter        string
	ShardFilterType    string
	ShardFilterShardID string
	MaxResults         int
}

ListShardsInput is the input for ListShards.

type ListShardsOutput

type ListShardsOutput struct {
	NextToken string
	Shards    []ShardDescription
}

ListShardsOutput is the output for ListShards.

type ListStreamConsumersInput

type ListStreamConsumersInput struct {
	StreamARN  string
	NextToken  string
	MaxResults int
}

ListStreamConsumersInput is the input for ListStreamConsumers.

type ListStreamConsumersOutput

type ListStreamConsumersOutput struct {
	NextToken string
	Consumers []Consumer
}

ListStreamConsumersOutput is the output for ListStreamConsumers.

type ListStreamsInput

type ListStreamsInput struct {
	NextToken                string
	ExclusiveStartStreamName string
	Limit                    int
}

ListStreamsInput is the input for ListStreams.

type ListStreamsOutput

type ListStreamsOutput struct {
	NextToken      string
	StreamNames    []string
	HasMoreStreams bool
}

ListStreamsOutput is the output for ListStreams.

type ListTagsForResourceInput

type ListTagsForResourceInput struct {
	ResourceARN string
}

ListTagsForResourceInput is the input for ListTagsForResource.

type ListTagsForResourceOutput

type ListTagsForResourceOutput struct {
	Tags map[string]string
}

ListTagsForResourceOutput is the output for ListTagsForResource.

type MergeShardsInput

type MergeShardsInput struct {
	StreamName           string
	StreamARN            string
	ShardToMerge         string
	AdjacentShardToMerge string
}

MergeShardsInput is the input for MergeShards.

type Provider

type Provider struct{}

Provider implements service.Provider for the Kinesis service.

func (*Provider) Init

Init initializes the Kinesis service backend and handler.

func (*Provider) Name

func (p *Provider) Name() string

Name returns the service provider name.

type PutRecordInput

type PutRecordInput struct {
	StreamName      string
	PartitionKey    string
	ExplicitHashKey string
	Data            []byte
}

PutRecordInput is the input for PutRecord.

type PutRecordOutput

type PutRecordOutput struct {
	ShardID        string
	SequenceNumber string
	EncryptionType string
}

PutRecordOutput is the output for PutRecord.

type PutRecordsEntry

type PutRecordsEntry struct {
	PartitionKey    string
	ExplicitHashKey string
	Data            []byte
}

PutRecordsEntry is a single entry in a PutRecords request.

type PutRecordsInput

type PutRecordsInput struct {
	StreamName string
	Records    []PutRecordsEntry
}

PutRecordsInput is the input for PutRecords.

type PutRecordsOutput

type PutRecordsOutput struct {
	Records           []PutRecordsResultEntry
	FailedRecordCount int
}

PutRecordsOutput is the output for PutRecords.

type PutRecordsResultEntry

type PutRecordsResultEntry struct {
	ShardID        string
	SequenceNumber string
	ErrorCode      string
	ErrorMessage   string
}

PutRecordsResultEntry is a single result entry in a PutRecords response.

type PutResourcePolicyInput

type PutResourcePolicyInput struct {
	ResourceARN string
	Policy      string
}

PutResourcePolicyInput is the input for PutResourcePolicy.

type Record

type Record struct {
	ApproximateArrivalTimestamp time.Time `json:"approximateArrivalTimestamp"`
	PartitionKey                string    `json:"partitionKey"`
	SequenceNumber              string    `json:"sequenceNumber"`
	Data                        []byte    `json:"data"`
}

Record represents a single Kinesis data record.

type RegisterStreamConsumerInput

type RegisterStreamConsumerInput struct {
	StreamARN    string
	ConsumerName string
}

RegisterStreamConsumerInput is the input for RegisterStreamConsumer.

type RegisterStreamConsumerOutput

type RegisterStreamConsumerOutput struct {
	Consumer Consumer
}

RegisterStreamConsumerOutput is the output for RegisterStreamConsumer.

type Settings

type Settings struct {
	JanitorInterval time.Duration `json:"janitor_interval" env:"KINESIS_JANITOR_INTERVAL" default:"1m" help:"Janitor tick interval."` //nolint:lll // Kong struct tag makes this line long
}

Settings holds service-level configuration for the Kinesis backend. Fields are picked up by the Kong CLI parser when this struct is embedded in the root CLI command.

type Shard

type Shard struct {
	ID                    string       `json:"id"`
	HashKeyRangeStart     string       `json:"hashKeyRangeStart"`
	HashKeyRangeEnd       string       `json:"hashKeyRangeEnd"`
	ParentShardID         string       `json:"parentShardId,omitempty"`
	AdjacentParentShardID string       `json:"adjacentParentShardId,omitempty"`
	Records               shardRecords `json:"records"`
	NextSeq               uint64       `json:"nextSeq"`
	Closed                bool         `json:"closed,omitempty"`
}

Shard represents a single Kinesis shard within a stream.

type ShardDescription

type ShardDescription struct {
	ShardID                  string
	HashKeyRangeStart        string
	HashKeyRangeEnd          string
	SequenceNumberRangeStart string
	SequenceNumberRangeEnd   string
	ParentShardID            string
	AdjacentParentShardID    string
	Closed                   bool
}

ShardDescription describes a shard in a DescribeStream response.

type ShardIterator

type ShardIterator struct {
	CreatedAt      time.Time `json:"CreatedAt"`
	StreamName     string    `json:"StreamName"`
	ShardID        string    `json:"ShardID"`
	SequenceNumber string    `json:"SequenceNumber"`
	Region         string    `json:"Region"`
	Position       int       `json:"Position"`
}

ShardIterator holds the position within a shard for GetRecords. Region is encoded into the iterator token so that GetRecords resolves the record store of the same region the iterator was issued in, keeping same-named streams in different regions isolated on the record hot path.

type SplitShardInput

type SplitShardInput struct {
	StreamName         string
	StreamARN          string
	ShardToSplit       string
	NewStartingHashKey string
}

SplitShardInput is the input for SplitShard.

type StartStreamEncryptionInput

type StartStreamEncryptionInput struct {
	StreamName     string
	StreamARN      string
	EncryptionType string
	KeyID          string
}

StartStreamEncryptionInput is the input for StartStreamEncryption.

type StartingPosition

type StartingPosition struct {
	Timestamp      *time.Time `json:"Timestamp,omitempty"`
	Type           string     `json:"Type"`
	SequenceNumber string     `json:"SequenceNumber,omitempty"`
}

StartingPosition describes where to start reading in SubscribeToShard.

type StopStreamEncryptionInput

type StopStreamEncryptionInput struct {
	StreamName     string
	StreamARN      string
	EncryptionType string
	KeyID          string
}

StopStreamEncryptionInput is the input for StopStreamEncryption.

type StorageBackend

type StorageBackend interface {
	CreateStream(ctx context.Context, input *CreateStreamInput) error
	DeleteStream(ctx context.Context, input *DeleteStreamInput) error
	DescribeStream(ctx context.Context, input *DescribeStreamInput) (*DescribeStreamOutput, error)
	ListStreams(ctx context.Context, input *ListStreamsInput) (*ListStreamsOutput, error)
	PutRecord(ctx context.Context, input *PutRecordInput) (*PutRecordOutput, error)
	PutRecords(ctx context.Context, input *PutRecordsInput) (*PutRecordsOutput, error)
	GetShardIterator(ctx context.Context, input *GetShardIteratorInput) (*GetShardIteratorOutput, error)
	GetRecords(ctx context.Context, input *GetRecordsInput) (*GetRecordsOutput, error)
	ListShards(ctx context.Context, input *ListShardsInput) (*ListShardsOutput, error)
	RegisterStreamConsumer(
		ctx context.Context,
		input *RegisterStreamConsumerInput,
	) (*RegisterStreamConsumerOutput, error)
	DescribeStreamConsumer(
		ctx context.Context,
		input *DescribeStreamConsumerInput,
	) (*DescribeStreamConsumerOutput, error)
	ListStreamConsumers(ctx context.Context, input *ListStreamConsumersInput) (*ListStreamConsumersOutput, error)
	DeregisterStreamConsumer(ctx context.Context, input *DeregisterStreamConsumerInput) error
	SubscribeToShard(ctx context.Context, input *SubscribeToShardInput) (*SubscribeToShardOutput, error)
	UpdateShardCount(ctx context.Context, input *UpdateShardCountInput) (*UpdateShardCountOutput, error)
	EnableEnhancedMonitoring(
		ctx context.Context,
		input *EnableEnhancedMonitoringInput,
	) (*EnableEnhancedMonitoringOutput, error)
	DisableEnhancedMonitoring(
		ctx context.Context,
		input *DisableEnhancedMonitoringInput,
	) (*DisableEnhancedMonitoringOutput, error)
	IncreaseStreamRetentionPeriod(ctx context.Context, input *IncreaseStreamRetentionPeriodInput) error
	DecreaseStreamRetentionPeriod(ctx context.Context, input *DecreaseStreamRetentionPeriodInput) error
	MergeShards(ctx context.Context, input *MergeShardsInput) error
	SplitShard(ctx context.Context, input *SplitShardInput) error
	StartStreamEncryption(ctx context.Context, input *StartStreamEncryptionInput) error
	StopStreamEncryption(ctx context.Context, input *StopStreamEncryptionInput) error
	DeleteResourcePolicy(ctx context.Context, input *DeleteResourcePolicyInput) error
	GetResourcePolicy(ctx context.Context, input *GetResourcePolicyInput) (*GetResourcePolicyOutput, error)
	PutResourcePolicy(ctx context.Context, input *PutResourcePolicyInput) error
	ListTagsForResource(ctx context.Context, input *ListTagsForResourceInput) (*ListTagsForResourceOutput, error)
	TagResource(ctx context.Context, input *TagResourceInput) error
	UntagResource(ctx context.Context, input *UntagResourceInput) error
	UpdateStreamMode(ctx context.Context, input *UpdateStreamModeInput) error
	UpdateAccountSettings(ctx context.Context, input *UpdateAccountSettingsInput) error
	UpdateMaxRecordSize(ctx context.Context, input *UpdateMaxRecordSizeInput) error
	UpdateStreamWarmThroughput(ctx context.Context, input *UpdateStreamWarmThroughputInput) error
	DescribeAccountSettings(ctx context.Context) (*DescribeAccountSettingsOutput, error)
	CountOpenShards(ctx context.Context) int
	ListAll(ctx context.Context) []StreamInfo
}

StorageBackend defines the interface for a Kinesis backend.

Every method takes a context.Context so the per-request AWS region can be threaded through and resources kept isolated per region. The region is read from the context via getRegion, falling back to the backend's default region when the context carries no region.

type Stream

type Stream struct {
	CreatedAt time.Time `json:"createdAt"`

	Tags      *tags.Tags           `json:"tags,omitempty"`
	Consumers map[string]*Consumer `json:"consumers,omitempty"`
	Name      string               `json:"name"`
	ARN       string               `json:"arn"`
	// Region is the AWS region this stream lives in. It is the second half of
	// the composite key (see streamKey in backend.go) that keeps same-named
	// streams in different regions isolated inside the single flat
	// store.Table[Stream] — the region-nested map it replaced used the
	// region as an outer map key instead of a field on Stream itself.
	Region             string   `json:"region,omitempty"`
	Status             string   `json:"status"`
	EncryptionType     string   `json:"encryptionType,omitempty"`
	KeyID              string   `json:"keyId,omitempty"`
	StreamMode         string   `json:"streamMode,omitempty"`
	Shards             []*Shard `json:"shards"`
	EnhancedMonitoring []string `json:"enhancedMonitoring,omitempty"`
	RetentionPeriod    int      `json:"retentionPeriod"`
	// MaxRecordSizeBytes is the per-record data payload size limit for this stream.
	// Defaults to defaultMaxRecordSizeBytes (1 MiB); updatable via UpdateMaxRecordSize.
	MaxRecordSizeBytes int `json:"maxRecordSizeBytes,omitempty"`
	// contains filtered or unexported fields
}

Stream represents an in-memory Kinesis stream.

type StreamInfo

type StreamInfo struct {
	Name       string
	ARN        string
	Status     string
	ShardCount int
}

StreamInfo holds summary information about a stream, safe to return without lock.

type StreamModeDetails

type StreamModeDetails struct {
	StreamMode string
}

StreamModeDetails describes the mode of a Kinesis stream.

type SubscribeToShardEvent

type SubscribeToShardEvent struct {
	ContinuationSequenceNumber string
	Records                    []GetRecordResult
	MillisBehindLatest         int64
}

SubscribeToShardEvent is a single event in the SubscribeToShard response.

type SubscribeToShardInput

type SubscribeToShardInput struct {
	ConsumerARN      string
	ShardID          string
	StartingPosition StartingPosition
}

SubscribeToShardInput is the input for SubscribeToShard.

type SubscribeToShardOutput

type SubscribeToShardOutput struct {
	Event SubscribeToShardEvent
}

SubscribeToShardOutput is the output for SubscribeToShard.

type TagResourceInput

type TagResourceInput struct {
	Tags        map[string]string
	ResourceARN string
}

TagResourceInput is the input for TagResource (ARN-based tagging).

type UntagResourceInput

type UntagResourceInput struct {
	ResourceARN string
	TagKeys     []string
}

UntagResourceInput is the input for UntagResource (ARN-based tag removal).

type UpdateAccountSettingsInput

type UpdateAccountSettingsInput struct {
	// OnDemandStreamCountLimit sets the account-level limit for ON_DEMAND streams.
	OnDemandStreamCountLimit int
}

UpdateAccountSettingsInput is the input for UpdateAccountSettings.

type UpdateMaxRecordSizeInput

type UpdateMaxRecordSizeInput struct {
	StreamName         string
	StreamARN          string
	MaxRecordSizeBytes int
}

UpdateMaxRecordSizeInput is the input for UpdateMaxRecordSize.

type UpdateShardCountInput

type UpdateShardCountInput struct {
	StreamName       string
	ScalingType      string
	TargetShardCount int
}

UpdateShardCountInput is the input for UpdateShardCount.

type UpdateShardCountOutput

type UpdateShardCountOutput struct {
	StreamName        string
	CurrentShardCount int
	TargetShardCount  int
}

UpdateShardCountOutput is the output for UpdateShardCount.

type UpdateStreamModeInput

type UpdateStreamModeInput struct {
	StreamARN         string
	StreamModeDetails StreamModeDetails
}

UpdateStreamModeInput is the input for UpdateStreamMode.

type UpdateStreamWarmThroughputInput

type UpdateStreamWarmThroughputInput struct {
	StreamName         string
	StreamARN          string
	WriteCapacityUnits int64
	ReadCapacityUnits  int64
}

UpdateStreamWarmThroughputInput is the input for UpdateStreamWarmThroughput.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL