messagequeue

package
v0.3.0-20260803204450-... Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 3, 2026 License: Apache-2.0 Imports: 2 Imported by: 0

README

Queue Abstractions

Vendor-agnostic interfaces for pub/sub messaging systems.

Interfaces

Queue

Creates publishers and subscribers.

Publisher

Publishes messages to topics.

type Publisher interface {
    Publish(ctx context.Context, topic string, message entityqueue.Message) error
    PublishAfter(ctx context.Context, topic string, message entityqueue.Message, delayMs int64) error
    Close() error
}

(entityqueue is github.com/uber/submitqueue/platform/base/messagequeue.)

PublishAfter inserts a fresh message that becomes visible to subscribers only after delayMs. It is distinct from Nack(requeueAfterMillis) even though both can produce "next delivery happens at T+delay":

  • Nack is "this delivery failed, try again" — it bumps retry_count and eventually trips DLQ.
  • PublishAfter is "postpone this work" — retry_count resets to 0, DLQ stays available for true failures.

Use PublishAfter for self-driven poll loops (e.g. the orchestrator's buildsignal consumer re-publishing itself between Status calls). Use Nack for processing failures.

Subscriber

Consumes messages from topics with per-subscription configuration.

type Subscriber interface {
    Subscribe(ctx context.Context, topic string, config SubscriptionConfig) (<-chan Delivery, error)
    Close() error
}
Delivery

Message with acknowledgment operations.

type Delivery interface {
    Message() entityqueue.Message
    Ack(ctx context.Context) error
    Nack(ctx context.Context, requeueAfterMillis int64) error
    Reject(ctx context.Context, reason string) error
    ExtendVisibilityTimeout(ctx context.Context, durationMillis int64) error
    DeliveryID() string
    Attempt() int
    ReceivedAt() int64
    Metadata() map[string]string
}
  • Ack — message processed successfully, remove from queue
  • Nack — processing failed, requeue for retry after delay
  • Reject — poison pill, move to DLQ (or ack if DLQ disabled)
  • ExtendVisibilityTimeout — extend processing window for long-running work
SubscriptionConfig

Per-subscription configuration for polling, batching, leasing, retries, and DLQ:

cfg := extqueue.DefaultSubscriptionConfig("worker-1", "consumer-group")
cfg.PollIntervalMs = 50
cfg.BatchSize = 20
cfg.VisibilityTimeoutMs = 60000
cfg.Retry.MaxAttempts = 3
cfg.DLQ.Enabled = true

See subscription_config.go for all fields and defaults.

Usage

import entityqueue "github.com/uber/submitqueue/platform/base/messagequeue"

q, _ := NewQueue(config)
defer q.Close()

// Publish
pub := q.Publisher()
msg := entityqueue.NewMessage("id", []byte("payload"), "partition-key", nil)
pub.Publish(ctx, "topic", msg)

// Subscribe
sub := q.Subscriber()
cfg := extqueue.DefaultSubscriptionConfig("worker-1", "consumer-group")
deliveries, _ := sub.Subscribe(ctx, "topic", cfg)
for delivery := range deliveries {
    if err := process(delivery.Message().Payload); err != nil {
        delivery.Nack(ctx, 0)  // Retry
        continue
    }
    delivery.Ack(ctx)
}

Implementing a Backend

  1. Create platform/extension/messagequeue/{backend}/ directory
  2. Implement Queue, Publisher, Subscriber, Delivery interfaces
  3. Map entityqueue.Message to backend format

See platform/extension/messagequeue/mysql/ for the reference implementation.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type DLQConfig

type DLQConfig struct {
	// Enabled enables dead letter queue.
	Enabled bool

	// TopicSuffix is appended to the original topic name to create the DLQ topic.
	// For example, if original topic is "orders" and suffix is "_dlq", DLQ topic will be "orders_dlq".
	TopicSuffix string
}

DLQConfig configures dead letter queue behavior.

type Delivery

type Delivery interface {
	// Message returns the delivered message.
	Message() entityqueue.Message

	// Ack acknowledges successful processing of the message.
	// The message will be removed from the queue and not redelivered.
	Ack(ctx context.Context) error

	// Nack negatively acknowledges the message, indicating processing failure.
	// The message will be requeued for redelivery after requeueAfterMillis.
	// If requeueAfterMillis is 0, the message is requeued immediately.
	Nack(ctx context.Context, requeueAfterMillis int64) error

	// Reject moves the message to the dead letter entityqueue.
	// Use for poison pill messages that should never be retried.
	// reason is stored as last_error in the DLQ for debugging.
	// If DLQ is not configured, the message is acked (removed from queue).
	Reject(ctx context.Context, reason string) error

	// ExtendVisibilityTimeout extends the time before this message becomes
	// visible to other consumers. Use when processing takes longer than expected.
	ExtendVisibilityTimeout(ctx context.Context, durationMillis int64) error

	// DeliveryID returns a backend-specific identifier for this delivery.
	DeliveryID() string

	// Attempt returns how many times this message has been delivered.
	// Starts at 1 for first delivery.
	Attempt() int

	// ReceivedAt returns when this delivery was received (Unix milliseconds).
	ReceivedAt() int64

	// Metadata returns backend-specific delivery metadata.
	Metadata() map[string]string
}

Delivery represents a message delivered by a Subscriber. Provides access to the message and methods to acknowledge or reject it.

Implementations must be safe for concurrent Message() calls. Ack/Nack/ExtendVisibilityTimeout should not be called concurrently on the same instance.

type Publisher

type Publisher interface {
	// Publish sends a message to the specified topic.
	Publish(ctx context.Context, topic string, message entityqueue.Message) error

	// PublishAfter sends a message that becomes visible to subscribers only
	// after delayMs from now. It is a fresh publish — not a redelivery — so
	// it does not consume a delivery_state retry slot. delayMs <= 0 is
	// equivalent to Publish.
	//
	// Use for "postpone this work" semantics (e.g. spacing out repeated
	// poll cycles for a single key). Use Nack with a delay for "this
	// delivery failed, try again" — the two signals stay separate so
	// retry_count and DLQ behaviour remain meaningful.
	PublishAfter(ctx context.Context, topic string, message entityqueue.Message, delayMs int64) error

	// Close gracefully shuts down the publisher, flushing pending messages.
	Close() error
}

Publisher publishes messages to topics. Implementations must be thread-safe.

type Queue

type Queue interface {
	// Publisher returns a Publisher instance.
	// May return a singleton or new instance depending on implementation.
	Publisher() Publisher

	// Subscriber returns a Subscriber instance.
	// May return a singleton or new instance depending on implementation.
	Subscriber() Subscriber

	// Close shuts down the queue and all associated resources.
	Close() error
}

Queue creates and manages queue publishers and subscribers. Implementations handle connection pooling, consumer group configuration, and resource lifecycle.

type RetryConfig

type RetryConfig struct {
	// MaxAttempts is the maximum number of processing attempts.
	// After this many attempts, the message is moved to DLQ (if enabled).
	MaxAttempts int

	// InitialBackoffMs is the initial backoff duration for retries (in milliseconds).
	InitialBackoffMs int64

	// MaxBackoffMs is the maximum backoff duration (in milliseconds).
	MaxBackoffMs int64

	// BackoffMultiplier is the multiplier for exponential backoff.
	BackoffMultiplier float64
}

RetryConfig configures message retry behavior.

type Subscriber

type Subscriber interface {
	// Subscribe starts consuming messages from the specified topic with the given config.
	// Returns a channel of Delivery instances and an error if subscription fails.
	//
	// Each subscription can have its own configuration for polling, batching,
	// retries, and dead letter queue behavior.
	//
	// The channel is closed when the subscriber is closed or context is cancelled.
	// Implementations should handle infrastructure errors internally (e.g., reconnect).
	//
	// Each Delivery provides the message and methods to acknowledge or reject it.
	// Consumers should call delivery.Ack() or delivery.Nack() for each delivery.
	Subscribe(ctx context.Context, topic string, config SubscriptionConfig) (<-chan Delivery, error)

	// Close gracefully shuts down the subscriber.
	// All delivery channels will be closed.
	// Idempotent - safe to call multiple times.
	Close() error
}

Subscriber consumes messages from topics. Implementations must be thread-safe.

type SubscriptionConfig

type SubscriptionConfig struct {
	// SubscriberName uniquely identifies this subscriber instance for partition leases.
	// Different workers should use different names (e.g., hostname, pod name, UUID).
	// Combined with ConsumerGroup, this determines which worker owns a partition lease.
	SubscriberName string

	// ConsumerGroup identifies this consumer for offset tracking.
	// Different consumer groups maintain independent offsets.
	ConsumerGroup string

	// PollIntervalMs is how often to poll for new messages (in milliseconds).
	PollIntervalMs int64

	// BatchSize is the maximum number of messages to fetch per poll.
	BatchSize int

	// VisibilityTimeoutMs is how long a message is invisible after being fetched (in milliseconds).
	// If the worker crashes or doesn't ack/nack in time, the message becomes
	// visible again after this duration.
	VisibilityTimeoutMs int64

	// LeaseRenewalIntervalMs is how often to renew partition leases (in milliseconds).
	LeaseRenewalIntervalMs int64

	// LeaseDurationMs is how long a lease is valid without renewal (in milliseconds).
	// Stale leases (not renewed within this duration) can be stolen by other workers.
	LeaseDurationMs int64

	// Retry configures message retry behavior.
	Retry RetryConfig

	// DLQ configures dead letter queue behavior.
	DLQ DLQConfig
}

SubscriptionConfig holds per-subscription configuration. Each subscription (topic) can have its own settings for polling, batching, retries, and dead letter queue behavior.

func DLQSubscriptionConfig

func DLQSubscriptionConfig(subscriberName, consumerGroup string) SubscriptionConfig

DLQSubscriptionConfig returns a SubscriptionConfig for consuming a dead-letter topic (DLQ reconciliation). It starts from DefaultSubscriptionConfig and applies the two overrides every DLQ consumer needs:

  • DLQ.Enabled is false, so a reconciliation failure retries in place instead of cascading to a second-level "_dlq_dlq" topic that nobody consumes.
  • Retry.MaxAttempts is a very high backstop so the per-message retry budget effectively never runs out. This pairs with errs.AlwaysRetryableProcessor wired into the DLQ consumer: reconciliation converges eventually instead of being silently dropped after the default retry count.

func DefaultSubscriptionConfig

func DefaultSubscriptionConfig(subscriberName, consumerGroup string) SubscriptionConfig

DefaultSubscriptionConfig returns a SubscriptionConfig with sensible defaults.

Directories

Path Synopsis
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.
Package mysql is a generated GoMock package.
Package mysql is a generated GoMock package.
ctl command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL