msgqueue

package
v0.98.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 28, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source
const (
	MsgIDCancelTasks                  = "cancel-tasks"
	MsgIDDurableCallbackCompleted     = "durable-callback-completed"
	MsgIDDurableRestoreTask           = "durable-restore-task"
	MsgIDCELEvaluationFailure         = "cel-evaluation-failure"
	MsgIDCheckTenantQueue             = "check-tenant-queue"
	MsgIDNewWorker                    = "new-worker"
	MsgIDNewQueue                     = "new-queue"
	MsgIDNewConcurrencyStrategy       = "new-concurrency-strategy"
	MsgIDCreateMonitoringEvent        = "create-monitoring-event"
	MsgIDCreatedDAG                   = "created-dag"
	MsgIDCreatedEventTrigger          = "created-event-trigger"
	MsgIDCreatedTask                  = "created-task"
	MsgIDFailedWebhookValidation      = "failed-webhook-validation"
	MsgIDInternalEvent                = "internal-event"
	MsgIDOffloadPayload               = "offload-payload"
	MsgIDReplayTasks                  = "replay-tasks"
	MsgIDTaskAssignedBulk             = "task-assigned-bulk"
	MsgIDTaskCancelled                = "task-cancelled"
	MsgIDTaskCompleted                = "task-completed"
	MsgIDTaskFailed                   = "task-failed"
	MsgIDTaskStreamEvent              = "task-stream-event"
	MsgIDTaskTrigger                  = "task-trigger"
	MsgIDUserEvent                    = "user-event"
	MsgIDWorkflowRunFinished          = "workflow-run-finished"
	MsgIDWorkflowRunFinishedCandidate = "workflow-run-finished-candidate"
	MsgIDCronCreate                   = "cron-create"
	MsgIDCronUpdate                   = "cron-update"
	MsgIDCronDelete                   = "cron-delete"
	MsgIDBatchStart                   = "batch-start"
)

Message ID constants for tenant messages

View Source
const (
	TASK_PROCESSING_QUEUE        staticQueue = "task_processing_queue_v2"
	OLAP_QUEUE                   staticQueue = "olap_queue_v2"
	DISPATCHER_DEAD_LETTER_QUEUE staticQueue = "dispatcher_dlq_v2"
	TICKER_UPDATE_QUEUE          staticQueue = "ticker_update_queue_v2"
)
View Source
const DefaultCompressionThreshold = 5 * 1024 // 5KB

DefaultCompressionThreshold is the payload size above which messages are gzip-compressed when compression is enabled.

Variables

View Source
var (
	PUB_FLUSH_INTERVAL  = 10 * time.Millisecond
	PUB_BUFFER_SIZE     = 10
	PUB_MAX_CONCURRENCY = 1
	PUB_TIMEOUT         = 10 * time.Second
)

nolint: staticcheck

View Source
var (
	SUB_FLUSH_INTERVAL  = 10 * time.Millisecond
	SUB_BUFFER_SIZE     = 10
	SUB_MAX_CONCURRENCY = 10
)

nolint: staticcheck

Functions

func DecodeAndValidateSingleton added in v0.74.3

func DecodeAndValidateSingleton(dv datautils.DataDecoderValidator, payloads [][]byte, target interface{}) error

func DecompressPayloads added in v0.98.9

func DecompressPayloads(payloads [][]byte) ([][]byte, error)

DecompressPayloads decompresses gzip-compressed message payloads. It lives in the msgqueue package (rather than a single implementation) because a compressed message can cross backends: a producer whose durable queue is RabbitMQ with compression enabled hands the same *Message to whichever pub/sub is configured, so every subscriber must be able to decompress.

func JSONConvert added in v0.74.3

func JSONConvert[T any](payloads [][]byte) []*T

func NewRandomStaticQueue added in v0.74.3

func NewRandomStaticQueue() staticQueue

func NoOpHook

func NoOpHook(task *Message) error

func PubTenantMessage added in v0.98.9

func PubTenantMessage(ctx context.Context, l *zerolog.Logger, mq MessageQueue, ps PubSub, queue Queue, msg *Message) error

PubTenantMessage writes a tenant-scoped message to its destinations: a durable send to queue via mq (when queue is non-nil), plus a publish to the tenant stream when the message ID is one the dispatcher's streams consume.

Durable send errors are returned. Stream publish errors are best-effort (logged, never propagated) when the message also had a durable destination; when the stream is the message's only delivery path (queue == nil), the publish error is returned so callers can decide.

func QueueTypeFromDispatcherID

func QueueTypeFromDispatcherID(d uuid.UUID) dispatcherQueue

func TenantStreamMsgIDs added in v0.98.9

func TenantStreamMsgIDs() []string

TenantStreamMsgIDs returns the message IDs PubTenantMessage publishes to tenant-stream topics, sorted. It exists so the dispatcher can assert its stream consumers stay in sync with this allowlist.

func WithBufferSize added in v0.74.3

func WithBufferSize(bufferSize int) mqSubBufferOptFunc

func WithDisableImmediateFlush added in v0.74.3

func WithDisableImmediateFlush(disableImmediateFlush bool) mqSubBufferOptFunc

func WithFlushInterval added in v0.74.3

func WithFlushInterval(flushInterval time.Duration) mqSubBufferOptFunc

func WithKind added in v0.74.3

func WithKind(kind SubBufferKind) mqSubBufferOptFunc

func WithMaxConcurrency added in v0.74.3

func WithMaxConcurrency(maxConcurrency int) mqSubBufferOptFunc

Types

type CompressionResult added in v0.98.9

type CompressionResult struct {
	Payloads       [][]byte
	WasCompressed  bool
	OriginalSize   int
	CompressedSize int

	// CompressionRatio is the ratio of compressed size to original size (compressed / original)
	CompressionRatio float64
}

type Compressor added in v0.98.9

type Compressor struct {
	Enabled   bool
	Threshold int
}

Compressor holds gzip compression settings for a message queue implementation. Compression settings must agree between the durable queue and the pub/sub, since both publish to the same tenant topics.

func (Compressor) CompressPayloads added in v0.98.9

func (t Compressor) CompressPayloads(payloads [][]byte) (*CompressionResult, error)

CompressPayloads compresses message payloads using gzip if they exceed the minimum size threshold. Returns compression results including the compressed payloads and compression statistics.

type DstFunc added in v0.74.3

type DstFunc func(tenantId uuid.UUID, msgId string, payloads [][]byte) error

type MQPubBuffer added in v0.74.3

type MQPubBuffer struct {
	// contains filtered or unexported fields
}

MQPubBuffer buffers messages coming out of the task queue, groups them by tenantId and msgId, and then flushes them to the task handler as necessary.

func NewMQPubBuffer added in v0.74.3

func NewMQPubBuffer(mq MessageQueue) *MQPubBuffer

func (*MQPubBuffer) Pub added in v0.74.3

func (m *MQPubBuffer) Pub(ctx context.Context, queue Queue, msg *Message, wait bool) error

func (*MQPubBuffer) Stop added in v0.74.3

func (m *MQPubBuffer) Stop()

type MQSubBuffer added in v0.74.3

type MQSubBuffer struct {
	// contains filtered or unexported fields
}

MQSubBuffer buffers messages coming out of the task queue, groups them by tenantId and msgId, and then flushes them to the task handler as necessary.

func NewMQSubBuffer added in v0.74.3

func NewMQSubBuffer(queue Queue, mq MessageQueue, dst DstFunc, fs ...mqSubBufferOptFunc) *MQSubBuffer

func NewSubBufferFromSubscribe added in v0.98.9

func NewSubBufferFromSubscribe(sub SubscribeFunc, dst DstFunc, fs ...mqSubBufferOptFunc) *MQSubBuffer

NewSubBufferFromSubscribe creates a sub buffer over an arbitrary subscribe function, so the buffer can sit on top of either the durable MessageQueue or a best-effort PubSub subscription.

func (*MQSubBuffer) Start added in v0.74.3

func (m *MQSubBuffer) Start() (func() error, error)

type Message

type Message struct {
	// ID is the ID of the task.
	ID string `json:"id"`

	// Payloads is the list of payloads.
	Payloads [][]byte `json:"messages"`

	// TenantID is the tenant ID.
	TenantID uuid.UUID `json:"tenant_id"`

	// Whether the message should immediately expire if it reaches the queue without an active consumer.
	ImmediatelyExpire bool `json:"immediately_expire"`

	// Whether the message should be persisted to disk
	Persistent bool `json:"persistent"`

	// OtelCarrier is the OpenTelemetry carrier for the task.
	OtelCarrier map[string]string `json:"otel_carrier"`

	// Retries is the number of retries for the task.
	//
	// Deprecated: retries are set globally at the moment.
	Retries int `json:"retries"`

	// Compressed indicates whether the payloads are gzip compressed
	Compressed bool `json:"compressed,omitempty"`
}

func NewTenantMessage added in v0.74.3

func NewTenantMessage[T any](tenantId uuid.UUID, id string, immediatelyExpire, persistent bool, payloads ...T) (*Message, error)

func (*Message) Clone added in v0.98.9

func (t *Message) Clone() *Message

Clone returns a shallow copy of the message. Payloads and OtelCarrier are shared with the original: callers must replace them, never mutate in place.

func (*Message) Serialize added in v0.74.3

func (t *Message) Serialize() ([]byte, error)

func (*Message) SetOtelCarrier added in v0.74.3

func (t *Message) SetOtelCarrier(otelCarrier map[string]string)

type MessageQueue

type MessageQueue interface {
	// Clone copies the message queue with a new instance.
	Clone() (func() error, MessageQueue, error)

	// SetQOS sets the quality of service for the message queue.
	SetQOS(prefetchCount int)

	// SendMessage sends a message to the message queue.
	SendMessage(ctx context.Context, queue Queue, msg *Message) error

	// Subscribe subscribes to the task queue. It returns a cleanup function that should be called when the
	// subscription is no longer needed.
	Subscribe(queue Queue, preAck MsgHandler, postAck MsgHandler) (func() error, error)

	// IsReady returns true if the task queue is ready to accept tasks.
	IsReady() bool
}

type MsgHandler added in v0.98.9

type MsgHandler func(task *Message) error

MsgHandler processes a received message. On the durable MessageQueue it is invoked as a pre-ack or post-ack hook; on the best-effort PubSub it is the sole handler with no ack semantics.

type PubFunc added in v0.74.3

type PubFunc func(m *Message) error

type PubSub added in v0.98.9

type PubSub interface {
	// Pub publishes a message to the topic. Delivery is best-effort: if no
	// subscriber is listening, the message is dropped.
	Pub(ctx context.Context, topic Topic, msg *Message) error

	// Sub subscribes to a topic. Delivery is at-most-once with no ack
	// semantics: handler errors are logged, never redelivered. It returns a
	// cleanup function that should be called when the subscription is no
	// longer needed.
	//
	// Fanout to multiple subscribers of the same topic is backend-dependent:
	// rabbitmq delivers every message to every subscriber, while the postgres
	// implementation delivers messages over 8KB to only one subscriber (see
	// postgres.PubSub.Sub).
	Sub(topic Topic, handler MsgHandler) (func() error, error)

	// IsReady returns true if the pub/sub mechanism is ready to accept messages.
	IsReady() bool
}

PubSub is a best-effort, non-durable, at-most-once pub/sub mechanism.

INVARIANT: implementations own their pooled resources (AMQP connections and channel pools, pgx pools) and never share them with the durable MessageQueue or the repository layer, since Pub can be called from within durable-write paths and sharing pools is a deadlock risk.

func NewGatedPubSub added in v0.98.9

func NewGatedPubSub(inner PubSub, disableTenantStreamPubs bool) PubSub

NewGatedPubSub wraps a PubSub so that tenant-stream publishes are dropped when disableTenantStreamPubs is true, except task-stream-event messages. Scheduler partition wake-ups always pass through.

type Queue

type Queue interface {
	// Name returns the name of the queue.
	Name() string

	// Durable returns true if this queue should survive task queue restarts.
	Durable() bool

	// AutoDeleted returns true if this queue should be deleted when the last consumer unsubscribes.
	AutoDeleted() bool

	// Exclusive returns true if this queue should only be accessed by the current connection.
	Exclusive() bool

	// DLQ returns the queue's dead letter queue, if it exists.
	DLQ() Queue

	// IsDLQ returns true if the queue is a dead letter queue.
	IsDLQ() bool

	// We distinguish between a static DLQ or an automatic DLQ. An automatic DLQ will automatically retry messages
	// in a loop with a 5-second backoff, and each subscription to the regular Queue will include a subscription
	// to the DLQ.
	IsAutoDLQ() bool

	// IsExpirable refers to whether the queue itself is expirable
	IsExpirable() bool
}

type SharedBufferedTenantReader added in v0.74.3

type SharedBufferedTenantReader struct {
	// contains filtered or unexported fields
}

func NewSharedBufferedTenantReader added in v0.74.3

func NewSharedBufferedTenantReader(ps PubSub) *SharedBufferedTenantReader

func (*SharedBufferedTenantReader) Subscribe added in v0.74.3

func (s *SharedBufferedTenantReader) Subscribe(tenantId uuid.UUID, f DstFunc) (func() error, error)

type SharedTenantReader

type SharedTenantReader struct {
	// contains filtered or unexported fields
}

func NewSharedTenantReader

func NewSharedTenantReader(ps PubSub) *SharedTenantReader

func (*SharedTenantReader) Subscribe

func (s *SharedTenantReader) Subscribe(tenantId uuid.UUID, postAck MsgHandler) (func() error, error)

type SubBufferKind added in v0.74.3

type SubBufferKind string
const (
	PostAck SubBufferKind = "postAck"
	PreAck  SubBufferKind = "preAck"
)

type SubscribeFunc added in v0.98.9

type SubscribeFunc func(preAck MsgHandler, postAck MsgHandler) (func() error, error)

SubscribeFunc subscribes to a message source with pre-ack and post-ack hooks, returning a cleanup function.

type Topic added in v0.98.9

type Topic struct {
	// contains filtered or unexported fields
}

Topic identifies a best-effort pub/sub destination.

func SchedulerPartitionTopic added in v0.98.9

func SchedulerPartitionTopic(partitionId string) Topic

SchedulerPartitionTopic wakes the scheduler owning a partition. The wire name matches the legacy consumer queue ("<partitionId>_scheduler_v1") so mixed-version fleets interoperate.

func TenantTopic added in v0.98.9

func TenantTopic(tenantId uuid.UUID) Topic

TenantTopic feeds the dispatcher's gRPC streams. The wire name matches the legacy tenant fanout exchange / NOTIFY name ("<uuid>_v1") so mixed-version fleets interoperate.

func (Topic) Kind added in v0.98.9

func (t Topic) Kind() TopicKind

func (Topic) Name added in v0.98.9

func (t Topic) Name() string

type TopicKind added in v0.98.9

type TopicKind string

TopicKind identifies the kind of pub/sub destination a Topic refers to.

const (
	// TopicKindTenantStream feeds the dispatcher's per-tenant gRPC streams.
	TopicKindTenantStream TopicKind = "tenant-stream"

	// TopicKindSchedulerPartition wakes the scheduler owning a partition.
	TopicKindSchedulerPartition TopicKind = "scheduler-partition"
)

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL