Documentation
¶
Index ¶
- Constants
- Variables
- func DecodeAndValidateSingleton(dv datautils.DataDecoderValidator, payloads [][]byte, target interface{}) error
- func DecompressPayloads(payloads [][]byte) ([][]byte, error)
- func JSONConvert[T any](payloads [][]byte) []*T
- func NewRandomStaticQueue() staticQueue
- func NoOpHook(task *Message) error
- func PubTenantMessage(ctx context.Context, l *zerolog.Logger, mq MessageQueue, ps PubSub, ...) error
- func QueueTypeFromDispatcherID(d uuid.UUID) dispatcherQueue
- func TenantStreamMsgIDs() []string
- func WithBufferSize(bufferSize int) mqSubBufferOptFunc
- func WithDisableImmediateFlush(disableImmediateFlush bool) mqSubBufferOptFunc
- func WithFlushInterval(flushInterval time.Duration) mqSubBufferOptFunc
- func WithKind(kind SubBufferKind) mqSubBufferOptFunc
- func WithMaxConcurrency(maxConcurrency int) mqSubBufferOptFunc
- type CompressionResult
- type Compressor
- type DstFunc
- type MQPubBuffer
- type MQSubBuffer
- type Message
- type MessageQueue
- type MsgHandler
- type PubFunc
- type PubSub
- type Queue
- type SharedBufferedTenantReader
- type SharedTenantReader
- type SubBufferKind
- type SubscribeFunc
- type Topic
- type TopicKind
Constants ¶
const ( MsgIDCancelTasks = "cancel-tasks" MsgIDDurableCallbackCompleted = "durable-callback-completed" MsgIDDurableRestoreTask = "durable-restore-task" MsgIDCELEvaluationFailure = "cel-evaluation-failure" MsgIDCheckTenantQueue = "check-tenant-queue" MsgIDNewWorker = "new-worker" MsgIDNewQueue = "new-queue" MsgIDNewConcurrencyStrategy = "new-concurrency-strategy" MsgIDCreateMonitoringEvent = "create-monitoring-event" MsgIDCreatedDAG = "created-dag" MsgIDCreatedEventTrigger = "created-event-trigger" MsgIDCreatedTask = "created-task" MsgIDFailedWebhookValidation = "failed-webhook-validation" MsgIDInternalEvent = "internal-event" MsgIDOffloadPayload = "offload-payload" MsgIDReplayTasks = "replay-tasks" MsgIDTaskAssignedBulk = "task-assigned-bulk" MsgIDTaskCancelled = "task-cancelled" MsgIDTaskCompleted = "task-completed" MsgIDTaskFailed = "task-failed" MsgIDTaskStreamEvent = "task-stream-event" MsgIDTaskTrigger = "task-trigger" MsgIDUserEvent = "user-event" MsgIDWorkflowRunFinished = "workflow-run-finished" MsgIDWorkflowRunFinishedCandidate = "workflow-run-finished-candidate" MsgIDCronCreate = "cron-create" MsgIDCronUpdate = "cron-update" MsgIDCronDelete = "cron-delete" MsgIDBatchStart = "batch-start" )
Message ID constants for tenant messages
const ( TASK_PROCESSING_QUEUE staticQueue = "task_processing_queue_v2" OLAP_QUEUE staticQueue = "olap_queue_v2" DISPATCHER_DEAD_LETTER_QUEUE staticQueue = "dispatcher_dlq_v2" TICKER_UPDATE_QUEUE staticQueue = "ticker_update_queue_v2" )
const DefaultCompressionThreshold = 5 * 1024 // 5KB
DefaultCompressionThreshold is the payload size above which messages are gzip-compressed when compression is enabled.
Variables ¶
var ( PUB_FLUSH_INTERVAL = 10 * time.Millisecond PUB_BUFFER_SIZE = 10 PUB_MAX_CONCURRENCY = 1 PUB_TIMEOUT = 10 * time.Second )
nolint: staticcheck
var ( SUB_FLUSH_INTERVAL = 10 * time.Millisecond SUB_BUFFER_SIZE = 10 SUB_MAX_CONCURRENCY = 10 )
nolint: staticcheck
Functions ¶
func DecodeAndValidateSingleton ¶ added in v0.74.3
func DecodeAndValidateSingleton(dv datautils.DataDecoderValidator, payloads [][]byte, target interface{}) error
func DecompressPayloads ¶ added in v0.98.9
DecompressPayloads decompresses gzip-compressed message payloads. It lives in the msgqueue package (rather than a single implementation) because a compressed message can cross backends: a producer whose durable queue is RabbitMQ with compression enabled hands the same *Message to whichever pub/sub is configured, so every subscriber must be able to decompress.
func JSONConvert ¶ added in v0.74.3
func NewRandomStaticQueue ¶ added in v0.74.3
func NewRandomStaticQueue() staticQueue
func PubTenantMessage ¶ added in v0.98.9
func PubTenantMessage(ctx context.Context, l *zerolog.Logger, mq MessageQueue, ps PubSub, queue Queue, msg *Message) error
PubTenantMessage writes a tenant-scoped message to its destinations: a durable send to queue via mq (when queue is non-nil), plus a publish to the tenant stream when the message ID is one the dispatcher's streams consume.
Durable send errors are returned. Stream publish errors are best-effort (logged, never propagated) when the message also had a durable destination; when the stream is the message's only delivery path (queue == nil), the publish error is returned so callers can decide.
func TenantStreamMsgIDs ¶ added in v0.98.9
func TenantStreamMsgIDs() []string
TenantStreamMsgIDs returns the message IDs PubTenantMessage publishes to tenant-stream topics, sorted. It exists so the dispatcher can assert its stream consumers stay in sync with this allowlist.
func WithBufferSize ¶ added in v0.74.3
func WithBufferSize(bufferSize int) mqSubBufferOptFunc
func WithDisableImmediateFlush ¶ added in v0.74.3
func WithDisableImmediateFlush(disableImmediateFlush bool) mqSubBufferOptFunc
func WithFlushInterval ¶ added in v0.74.3
func WithKind ¶ added in v0.74.3
func WithKind(kind SubBufferKind) mqSubBufferOptFunc
func WithMaxConcurrency ¶ added in v0.74.3
func WithMaxConcurrency(maxConcurrency int) mqSubBufferOptFunc
Types ¶
type CompressionResult ¶ added in v0.98.9
type Compressor ¶ added in v0.98.9
Compressor holds gzip compression settings for a message queue implementation. Compression settings must agree between the durable queue and the pub/sub, since both publish to the same tenant topics.
func (Compressor) CompressPayloads ¶ added in v0.98.9
func (t Compressor) CompressPayloads(payloads [][]byte) (*CompressionResult, error)
CompressPayloads compresses message payloads using gzip if they exceed the minimum size threshold. Returns compression results including the compressed payloads and compression statistics.
type MQPubBuffer ¶ added in v0.74.3
type MQPubBuffer struct {
// contains filtered or unexported fields
}
MQPubBuffer buffers messages coming out of the task queue, groups them by tenantId and msgId, and then flushes them to the task handler as necessary.
func NewMQPubBuffer ¶ added in v0.74.3
func NewMQPubBuffer(mq MessageQueue) *MQPubBuffer
func (*MQPubBuffer) Stop ¶ added in v0.74.3
func (m *MQPubBuffer) Stop()
type MQSubBuffer ¶ added in v0.74.3
type MQSubBuffer struct {
// contains filtered or unexported fields
}
MQSubBuffer buffers messages coming out of the task queue, groups them by tenantId and msgId, and then flushes them to the task handler as necessary.
func NewMQSubBuffer ¶ added in v0.74.3
func NewMQSubBuffer(queue Queue, mq MessageQueue, dst DstFunc, fs ...mqSubBufferOptFunc) *MQSubBuffer
func NewSubBufferFromSubscribe ¶ added in v0.98.9
func NewSubBufferFromSubscribe(sub SubscribeFunc, dst DstFunc, fs ...mqSubBufferOptFunc) *MQSubBuffer
NewSubBufferFromSubscribe creates a sub buffer over an arbitrary subscribe function, so the buffer can sit on top of either the durable MessageQueue or a best-effort PubSub subscription.
func (*MQSubBuffer) Start ¶ added in v0.74.3
func (m *MQSubBuffer) Start() (func() error, error)
type Message ¶
type Message struct {
// ID is the ID of the task.
ID string `json:"id"`
// Payloads is the list of payloads.
Payloads [][]byte `json:"messages"`
// TenantID is the tenant ID.
TenantID uuid.UUID `json:"tenant_id"`
// Whether the message should immediately expire if it reaches the queue without an active consumer.
ImmediatelyExpire bool `json:"immediately_expire"`
// Whether the message should be persisted to disk
Persistent bool `json:"persistent"`
// OtelCarrier is the OpenTelemetry carrier for the task.
OtelCarrier map[string]string `json:"otel_carrier"`
// Retries is the number of retries for the task.
//
// Deprecated: retries are set globally at the moment.
Retries int `json:"retries"`
// Compressed indicates whether the payloads are gzip compressed
Compressed bool `json:"compressed,omitempty"`
}
func NewTenantMessage ¶ added in v0.74.3
func (*Message) Clone ¶ added in v0.98.9
Clone returns a shallow copy of the message. Payloads and OtelCarrier are shared with the original: callers must replace them, never mutate in place.
func (*Message) SetOtelCarrier ¶ added in v0.74.3
type MessageQueue ¶
type MessageQueue interface {
// Clone copies the message queue with a new instance.
Clone() (func() error, MessageQueue, error)
// SetQOS sets the quality of service for the message queue.
SetQOS(prefetchCount int)
// SendMessage sends a message to the message queue.
SendMessage(ctx context.Context, queue Queue, msg *Message) error
// Subscribe subscribes to the task queue. It returns a cleanup function that should be called when the
// subscription is no longer needed.
Subscribe(queue Queue, preAck MsgHandler, postAck MsgHandler) (func() error, error)
// IsReady returns true if the task queue is ready to accept tasks.
IsReady() bool
}
type MsgHandler ¶ added in v0.98.9
MsgHandler processes a received message. On the durable MessageQueue it is invoked as a pre-ack or post-ack hook; on the best-effort PubSub it is the sole handler with no ack semantics.
type PubSub ¶ added in v0.98.9
type PubSub interface {
// Pub publishes a message to the topic. Delivery is best-effort: if no
// subscriber is listening, the message is dropped.
Pub(ctx context.Context, topic Topic, msg *Message) error
// Sub subscribes to a topic. Delivery is at-most-once with no ack
// semantics: handler errors are logged, never redelivered. It returns a
// cleanup function that should be called when the subscription is no
// longer needed.
//
// Fanout to multiple subscribers of the same topic is backend-dependent:
// rabbitmq delivers every message to every subscriber, while the postgres
// implementation delivers messages over 8KB to only one subscriber (see
// postgres.PubSub.Sub).
Sub(topic Topic, handler MsgHandler) (func() error, error)
// IsReady returns true if the pub/sub mechanism is ready to accept messages.
IsReady() bool
}
PubSub is a best-effort, non-durable, at-most-once pub/sub mechanism.
INVARIANT: implementations own their pooled resources (AMQP connections and channel pools, pgx pools) and never share them with the durable MessageQueue or the repository layer, since Pub can be called from within durable-write paths and sharing pools is a deadlock risk.
func NewGatedPubSub ¶ added in v0.98.9
NewGatedPubSub wraps a PubSub so that tenant-stream publishes are dropped when disableTenantStreamPubs is true, except task-stream-event messages. Scheduler partition wake-ups always pass through.
type Queue ¶
type Queue interface {
// Name returns the name of the queue.
Name() string
// Durable returns true if this queue should survive task queue restarts.
Durable() bool
// AutoDeleted returns true if this queue should be deleted when the last consumer unsubscribes.
AutoDeleted() bool
// Exclusive returns true if this queue should only be accessed by the current connection.
Exclusive() bool
// DLQ returns the queue's dead letter queue, if it exists.
DLQ() Queue
// IsDLQ returns true if the queue is a dead letter queue.
IsDLQ() bool
// We distinguish between a static DLQ or an automatic DLQ. An automatic DLQ will automatically retry messages
// in a loop with a 5-second backoff, and each subscription to the regular Queue will include a subscription
// to the DLQ.
IsAutoDLQ() bool
// IsExpirable refers to whether the queue itself is expirable
IsExpirable() bool
}
type SharedBufferedTenantReader ¶ added in v0.74.3
type SharedBufferedTenantReader struct {
// contains filtered or unexported fields
}
func NewSharedBufferedTenantReader ¶ added in v0.74.3
func NewSharedBufferedTenantReader(ps PubSub) *SharedBufferedTenantReader
type SharedTenantReader ¶
type SharedTenantReader struct {
// contains filtered or unexported fields
}
func NewSharedTenantReader ¶
func NewSharedTenantReader(ps PubSub) *SharedTenantReader
func (*SharedTenantReader) Subscribe ¶
func (s *SharedTenantReader) Subscribe(tenantId uuid.UUID, postAck MsgHandler) (func() error, error)
type SubBufferKind ¶ added in v0.74.3
type SubBufferKind string
const ( PostAck SubBufferKind = "postAck" PreAck SubBufferKind = "preAck" )
type SubscribeFunc ¶ added in v0.98.9
type SubscribeFunc func(preAck MsgHandler, postAck MsgHandler) (func() error, error)
SubscribeFunc subscribes to a message source with pre-ack and post-ack hooks, returning a cleanup function.
type Topic ¶ added in v0.98.9
type Topic struct {
// contains filtered or unexported fields
}
Topic identifies a best-effort pub/sub destination.
func SchedulerPartitionTopic ¶ added in v0.98.9
SchedulerPartitionTopic wakes the scheduler owning a partition. The wire name matches the legacy consumer queue ("<partitionId>_scheduler_v1") so mixed-version fleets interoperate.
func TenantTopic ¶ added in v0.98.9
TenantTopic feeds the dispatcher's gRPC streams. The wire name matches the legacy tenant fanout exchange / NOTIFY name ("<uuid>_v1") so mixed-version fleets interoperate.