messagequeue

package
v0.3.0-20260820034428-... Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 20, 2026 License: Apache-2.0 Imports: 3 Imported by: 0

README

Queue Abstractions

Vendor-agnostic interfaces for pub/sub messaging systems.

Interfaces

Queue

Creates publishers and subscribers.

Publisher

Publishes messages to topics.

type Publisher interface {
    Publish(ctx context.Context, topic string, message entityqueue.Message) error
    Close() error
}

(entityqueue is github.com/uber/submitqueue/platform/base/messagequeue.)

Subscriber

Consumes messages from topics with per-subscription configuration.

type Subscriber interface {
    Subscribe(ctx context.Context, topic string, config SubscriptionConfig) (<-chan Delivery, error)
    Close() error
}
Delivery

Message with acknowledgment operations.

type Delivery interface {
    Message() entityqueue.Message
    Ack(ctx context.Context) error
    Nack(ctx context.Context) error
    Postpone(ctx context.Context, delayMs int64) error
    Reject(ctx context.Context, reason string) error
    ExtendVisibilityTimeout(ctx context.Context, durationMillis int64) error
    DeliveryID() string
    Attempt() int
    ReceivedAt() int64
    Metadata() map[string]string
}
  • Ack — message processed successfully, remove from queue
  • Nack — processing failed, requeue for immediate retry
  • Postpone — processed successfully but must wait: redeliver after delay, without consuming retry budget; the message is a barrier its partition waits behind
  • Reject — poison pill, move to DLQ (or ack if DLQ disabled)
  • ExtendVisibilityTimeout — extend processing window for long-running work

Postpone vs Nack vs ExtendVisibilityTimeout: Nack is a failure — the message is immediately eligible again, the redelivery counts toward Retry.MaxAttempts and eventually trips the DLQ, and later offsets in the partition keep flowing past the nacked message (a failed message must not halt its partition). Postpone is a deliberate wait — the redelivery happens after the chosen delay, resets the failure streak (it restarts at attempt 1), and blocks the partition behind it until it redelivers, in order. ExtendVisibilityTimeout is neither: the delivery is still being processed and stays in flight.

SubscriptionConfig

Per-subscription configuration for polling, batching, leasing, retries, and DLQ:

cfg := extqueue.DefaultSubscriptionConfig("worker-1", "consumer-group")
cfg.PollIntervalMs = 50
cfg.BatchSize = 20
cfg.VisibilityTimeoutMs = 60000
cfg.Retry.MaxAttempts = 3
cfg.DLQ.Enabled = true

See subscription_config.go for all fields and defaults.

Usage

import entityqueue "github.com/uber/submitqueue/platform/base/messagequeue"

q, _ := NewQueue(config)
defer q.Close()

// Publish
pub := q.Publisher()
msg := entityqueue.NewMessage("id", []byte("payload"), "partition-key", nil)
pub.Publish(ctx, "topic", msg)

// Subscribe
sub := q.Subscriber()
cfg := extqueue.DefaultSubscriptionConfig("worker-1", "consumer-group")
deliveries, _ := sub.Subscribe(ctx, "topic", cfg)
for delivery := range deliveries {
    if err := process(delivery.Message().Payload); err != nil {
        delivery.Nack(ctx)  // Retry
        continue
    }
    delivery.Ack(ctx)
}

Message IDs

A message ID is the deduplication key, scoped to its topic and partition key. A backend matches a publish against messages it still holds — including ones already consumed, since reclamation is lazy and may lag delivery by an unbounded interval — and a collision is reported to the publisher as a success that stored nothing. There is no error to retry and no row to deliver.

The ID therefore names the occasion to publish, not the entity published about. An entity's own ID buys exactly one message for that entity for as long as the backend remembers the first, so a stage that announces a batch at creation and another that wakes it after a merge would collide, and the wake-up would vanish.

Producers do not choose IDs by hand. They publish through platform/publish, whose IntentID(entityID, cause...) composes the entity with the cause of this particular message: a retry of the same cause dedups, which is what makes redelivery safe, while a new cause about the same entity can never be swallowed. UniqueID is the fallback for a cause with nothing stable to name it by, and it trades that idempotency for guaranteed delivery.

Backends must treat the ID as opaque and must not derive routing, ordering, or storage layout from its structure.

Implementing a Backend

  1. Create platform/extension/messagequeue/{backend}/ directory
  2. Implement Queue, Publisher, Subscriber, Delivery interfaces
  3. Map entityqueue.Message to backend format
  4. Deduplicate publishes on (topic, partition key, message ID)

See platform/extension/messagequeue/mysql/ for the reference implementation.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type DLQConfig

type DLQConfig struct {
	// Enabled enables dead letter queue.
	Enabled bool

	// TopicSuffix is appended to the original topic name to create the DLQ topic.
	// For example, if original topic is "orders" and suffix is "_dlq", DLQ topic will be "orders_dlq".
	TopicSuffix string
}

DLQConfig configures dead letter queue behavior.

type Delivery

type Delivery interface {
	// Message returns the delivered message.
	Message() entityqueue.Message

	// Ack acknowledges successful processing of the message.
	// The message will be removed from the queue and not redelivered.
	Ack(ctx context.Context) error

	// Nack negatively acknowledges the message, indicating processing failure.
	// The message is requeued for redelivery immediately; the visibility
	// timeout is what spaces retries (a crash or missed ack redelivers on the
	// same schedule). The redelivery counts toward the failure budget.
	//
	// f describes the failure. It is carried so that the redelivery which
	// finally exhausts the budget can dead-letter with the reason that caused
	// it, rather than with a generic one — a nack whose reason is dropped
	// leaves the eventual dead letter unable to say what went wrong.
	Nack(ctx context.Context, f failure.Failure) error

	// Postpone finishes this delivery as "processed successfully, redeliver
	// later": the message becomes invisible for delayMs and acts as a barrier —
	// its partition is not consumed past it until it redelivers, in order.
	// Unlike Nack, the redelivery does not count against the failure budget
	// (retry limit / DLQ); postponing resets the failure streak.
	// Postpone is terminal for this delivery, like Ack/Nack/Reject.
	Postpone(ctx context.Context, delayMs int64) error

	// Reject moves the message to the dead letter entityqueue.
	// Use for poison pill messages that should never be retried.
	// f is recorded with the dead-lettered message for diagnosis and is what
	// Failure returns when it is redelivered from the DLQ.
	// If DLQ is not configured, the message is acked (removed from queue).
	Reject(ctx context.Context, f failure.Failure) error

	// ExtendVisibilityTimeout extends the time before this message becomes
	// visible to other consumers. Use when processing takes longer than expected.
	ExtendVisibilityTimeout(ctx context.Context, durationMillis int64) error

	// DeliveryID returns a backend-specific identifier for this delivery.
	DeliveryID() string

	// Attempt returns how many times this message has been delivered.
	// Starts at 1 for first delivery.
	Attempt() int

	// ReceivedAt returns when this delivery was received (Unix milliseconds).
	ReceivedAt() int64

	// Metadata returns backend-specific delivery metadata.
	Metadata() map[string]string

	// Failure returns why this message was dead-lettered, and whether it was
	// dead-lettered at all. It reports false for a message delivered from its
	// original topic, so a DLQ consumer can distinguish "no failure recorded"
	// from a failure that recorded nothing.
	Failure() (failure.Failure, bool)
}

Delivery represents a message delivered by a Subscriber. Provides access to the message and methods to acknowledge or reject it.

Implementations must be safe for concurrent Message() calls. Ack/Nack/ExtendVisibilityTimeout should not be called concurrently on the same instance.

type Publisher

type Publisher interface {
	// Publish sends a message to the specified topic.
	Publish(ctx context.Context, topic string, message entityqueue.Message) error

	// Close gracefully shuts down the publisher, flushing pending messages.
	Close() error
}

Publisher publishes messages to topics. Implementations must be thread-safe.

type Queue

type Queue interface {
	// Publisher returns a Publisher instance.
	// May return a singleton or new instance depending on implementation.
	Publisher() Publisher

	// Subscriber returns a Subscriber instance.
	// May return a singleton or new instance depending on implementation.
	Subscriber() Subscriber

	// Close shuts down the queue and all associated resources.
	Close() error
}

Queue creates and manages queue publishers and subscribers. Implementations handle connection pooling, consumer group configuration, and resource lifecycle.

type RetryConfig

type RetryConfig struct {
	// MaxAttempts is the maximum number of processing attempts.
	// After this many attempts, the message is moved to DLQ (if enabled).
	MaxAttempts int

	// InitialBackoffMs is the initial backoff duration for retries (in milliseconds).
	InitialBackoffMs int64

	// MaxBackoffMs is the maximum backoff duration (in milliseconds).
	MaxBackoffMs int64

	// BackoffMultiplier is the multiplier for exponential backoff.
	BackoffMultiplier float64
}

RetryConfig configures message retry behavior.

type Subscriber

type Subscriber interface {
	// Subscribe starts consuming messages from the specified topic with the given config.
	// Returns a channel of Delivery instances and an error if subscription fails.
	//
	// Each subscription can have its own configuration for polling, batching,
	// retries, and dead letter queue behavior.
	//
	// The channel is closed when the subscriber is closed or context is cancelled.
	// Implementations should handle infrastructure errors internally (e.g., reconnect).
	//
	// Each Delivery provides the message and methods to acknowledge or reject it.
	// Consumers should call delivery.Ack() or delivery.Nack() for each delivery.
	Subscribe(ctx context.Context, topic string, config SubscriptionConfig) (<-chan Delivery, error)

	// Close gracefully shuts down the subscriber.
	// All delivery channels will be closed.
	// Idempotent - safe to call multiple times.
	Close() error
}

Subscriber consumes messages from topics. Implementations must be thread-safe.

type SubscriptionConfig

type SubscriptionConfig struct {
	// SubscriberName uniquely identifies this subscriber instance for partition leases.
	// Different workers should use different names (e.g., hostname, pod name, UUID).
	// Combined with ConsumerGroup, this determines which worker owns a partition lease.
	SubscriberName string

	// ConsumerGroup identifies this consumer for offset tracking.
	// Different consumer groups maintain independent offsets.
	ConsumerGroup string

	// PollIntervalMs is how often to poll for new messages (in milliseconds).
	PollIntervalMs int64

	// PartitionDiscoveryIntervalMs is how often to discover partitions,
	// attempt lease acquisition, and reconcile partition workers (in
	// milliseconds). Separate from PollIntervalMs: message polling needs low
	// latency, while discovery drives topic-wide queries whose volume
	// multiplies with subscribers and topics and whose outcome only changes
	// on membership or partition changes.
	PartitionDiscoveryIntervalMs int64

	// BatchSize is the maximum number of messages to fetch per poll.
	BatchSize int

	// VisibilityTimeoutMs is how long a message is invisible after being fetched (in milliseconds).
	// If the worker crashes or doesn't ack/nack in time, the message becomes
	// visible again after this duration.
	VisibilityTimeoutMs int64

	// LeaseRenewalIntervalMs is how often to renew partition leases (in milliseconds).
	LeaseRenewalIntervalMs int64

	// LeaseDurationMs is how long a lease is valid without renewal (in milliseconds).
	// Stale leases (not renewed within this duration) can be stolen by other workers.
	LeaseDurationMs int64

	// Retry configures message retry behavior.
	Retry RetryConfig

	// DLQ configures dead letter queue behavior.
	DLQ DLQConfig
}

SubscriptionConfig holds per-subscription configuration. Each subscription (topic) can have its own settings for polling, batching, retries, and dead letter queue behavior.

func DLQSubscriptionConfig

func DLQSubscriptionConfig(subscriberName, consumerGroup string) SubscriptionConfig

DLQSubscriptionConfig returns a SubscriptionConfig for consuming a dead-letter topic (DLQ reconciliation). It starts from DefaultSubscriptionConfig and applies the two overrides every DLQ consumer needs:

  • DLQ.Enabled is false, so a reconciliation failure retries in place instead of cascading to a second-level "_dlq_dlq" topic that nobody consumes.
  • Retry.MaxAttempts is a very high backstop so the per-message retry budget effectively never runs out. This pairs with errs.AlwaysRetryableProcessor wired into the DLQ consumer: reconciliation converges eventually instead of being silently dropped after the default retry count.

func DefaultSubscriptionConfig

func DefaultSubscriptionConfig(subscriberName, consumerGroup string) SubscriptionConfig

DefaultSubscriptionConfig returns a SubscriptionConfig with sensible defaults.

Directories

Path Synopsis
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.
Package mysql is a generated GoMock package.
Package mysql is a generated GoMock package.
ctl command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL