Documentation
¶
Overview ¶
Package messagequeue provides message queue publisher and consumer interfaces with implementations for Google Pub/Sub, Redis, and Amazon SQS.
Index ¶
Constants ¶
const OrderingKeyAttribute = "ordering_key"
OrderingKeyAttribute is the span attribute a publisher records the ordering key under, when one was given. It lives here rather than in each broker because three copies of an attribute name is three chances for one of them to drift, and a trace that names the same thing two ways cannot be queried for it.
It is not recorded as a metric attribute: an ordering key is usually an entity ID, and one time series per entity is a cardinality explosion.
Variables ¶
var ( // ErrEmptyTopicName is returned when a topic name is empty. ErrEmptyTopicName = platformerrors.New("empty topic name") // ErrConsumerAlreadyRegistered is returned when a second consumer is // requested for a topic a provider already has one for. // // Providers cache consumers by topic, and the cache used to win silently: // the second caller got the first caller's consumer, wired to the first // caller's handler, and their own handler was never invoked for any message. // Nothing failed and nothing logged — the messages simply went somewhere else. // // One consumer per topic per provider is the rule; a caller that wants two // behaviors for one topic multiplexes inside its own handler. ErrConsumerAlreadyRegistered = platformerrors.New("a consumer is already registered for this topic") )
Functions ¶
This section is empty.
Types ¶
type Consumer ¶
Consumer reads messages off a queue and hands each to its handler.
Stopping ¶
Consume runs until ctx is done. There is no separate stop channel: every implementation turned one into a context cancellation immediately, so it was a second way to say the same thing — and a `chan bool` at that, which is bidirectional, so nothing stopped a caller from receiving on it and stealing the stop signal from the consumer.
Delivery semantics ¶
These differ by backend, and the difference is load-bearing rather than an implementation detail a caller can ignore:
- redis is at-most-once. It is pub/sub: a message delivered while no consumer is running is gone, and a handler that fails does not get the message again. Do not use it for work that must not be lost.
- sqs, pubsub and kafka are at-least-once. A handler must therefore be idempotent — see the idempotency package — because redelivery is normal operation, not an error case.
Handler errors are reported on errs, and what happens next also differs: kafka stops the consumer, because its commits are cumulative by offset and committing past a failed message would lose it; the others log the failure and continue with the next message.
errs is send-only and must be drained. A consumer whose error channel is not being read does not block forever — it also selects on ctx — but it does discard errors while nobody is listening.
type ConsumerFunc ¶
ConsumerFunc is a function type that handles consumed messages.
type ConsumerProvider ¶
type ConsumerProvider interface {
Close()
NewConsumer(ctx context.Context, topic string, handlerFunc ConsumerFunc) (Consumer, error)
}
ConsumerProvider is a function that provides a Consumer for a given topic.
One consumer per topic: a second NewConsumer for a topic that already has one returns ErrConsumerAlreadyRegistered rather than silently handing back the first caller's consumer, wired to the first caller's handler.
type PublishOption ¶
type PublishOption func(*PublishOptions)
PublishOption adjusts a single Publish or PublishAsync call.
func WithDeduplicationKey ¶
func WithDeduplicationKey(key string) PublishOption
WithDeduplicationKey sets a message's identity for provider-side deduplication. See PublishOptions.DeduplicationKey; only sqs honors it.
func WithOrderingKey ¶
func WithOrderingKey(key string) PublishOption
WithOrderingKey sets the sequence a message belongs to. See PublishOptions.OrderingKey for what each backend does with it.
type PublishOptions ¶
type PublishOptions struct {
// OrderingKey names the sequence this message belongs to. Messages
// published with the same key are delivered in the order they were
// published; messages with different keys have no order relative to
// each other, which is what lets a broker spread the topic over
// partitions, groups or shards and still keep each entity's history
// intact. The usual key is the ID of whatever the message is about — an
// account, an order, a device.
//
// The empty string means "no ordering requirement" and is the default.
// It is not a key: every backend that honors ordering treats it as
// "spread this one freely", not as a group that all unkeyed messages
// share, because one shared group would serialize the whole topic.
//
// What each backend does with it:
//
// - kafka sets it as the message key and partitions on a murmur2 hash
// of it, so one key is one partition and Kafka's per-partition order
// is the guarantee.
// - sqs sets it as MessageGroupId, which a FIFO queue requires on
// every message. See DeduplicationKey, which a FIFO queue also
// needs unless the queue deduplicates on content.
// - pubsub sets it as the message's OrderingKey.
// - redis ignores it. Redis pub/sub has no ordering concept to map it
// onto: there are no partitions, no groups, and no ordering
// guarantee beyond what one connection happens to deliver. A key
// given to a redis publisher changes nothing and is not an error.
//
// Ordering is a joint property of publisher and subscriber, and only the
// publishing half is this package's to set. Kafka delivers a partition
// in order to whichever consumer owns it; SQS FIFO and Pub/Sub both need
// the queue or subscription provisioned for ordered delivery, which
// happens outside this package.
OrderingKey string
// DeduplicationKey names this message's identity for provider-side
// deduplication: a broker that sees the key twice within its
// deduplication window delivers the message once.
//
// Only sqs honors it, as MessageDeduplicationId. A FIFO queue requires
// either this or ContentBasedDeduplication enabled on the queue itself —
// with neither, SendMessage fails rather than publishing — so a caller
// publishing to a FIFO queue that hashes its own bodies can leave this
// empty, and one publishing to a FIFO queue that does not must set it.
//
// kafka, pubsub and redis ignore it. None of them deduplicate on a
// caller-supplied key: Kafka's idempotent producer deduplicates on its
// own sequence numbers, and Pub/Sub and Redis do not deduplicate at all.
// Handlers on those backends must be idempotent regardless; see the
// idempotency package.
DeduplicationKey string
}
PublishOptions is the resolved form of the options given to a single Publish or PublishAsync call. Callers set it through the With functions; Publisher implementations read it, after resolving their variadic through NewPublishOptions.
It is per-message rather than per-publisher on purpose: the ordering key is a property of the entity a message is about, and one publisher serves every entity on its topic.
func NewPublishOptions ¶
func NewPublishOptions(opts ...PublishOption) *PublishOptions
NewPublishOptions resolves a Publish call's variadic into the options every Publisher implementation reads. Options apply in order, so the last one to set a field wins, and a nil option is skipped rather than panicking.
It is exported because implementations of Publisher live outside this package — the four in messagequeue, and any a consumer writes — and resolving the variadic in each of them is the kind of thing that drifts.
type Publisher ¶
type Publisher interface {
// Stop halts all publishing.
Stop()
// Publish writes a message onto a message queue.
Publish(ctx context.Context, data any, opts ...PublishOption) error
// PublishAsync writes a message onto a message queue, logging any error
// instead of returning it.
//
// "Async" names the error handling, not the delivery: it publishes on the
// calling goroutine and returns when the publish has finished, exactly as
// Publish does. A caller that wants the publish off its own goroutine has
// to arrange that itself.
PublishAsync(ctx context.Context, data any, opts ...PublishOption)
}
Publisher writes messages onto a queue.
Per-message options ¶
Both methods take PublishOptions, which carry what the message is rather than how the publisher is built — today the ordering key and the deduplication key. They are per call because one publisher serves every entity on its topic, and the key belongs to the entity.
An option a backend has no concept for is ignored rather than rejected, so that a caller can pass the same options to whichever publisher it was wired with. PublishOptions documents, field by field, which backends honor what.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package messagequeuecfg selects and builds messagequeue publisher and consumer providers from configuration, over Redis, SQS, GCP Pub/Sub, Kafka, or noop.
|
Package messagequeuecfg selects and builds messagequeue publisher and consumer providers from configuration, over Redis, SQS, GCP Pub/Sub, Kafka, or noop. |
|
internal
|
|
|
consumererr
Package consumererr holds the send every messagequeue Consumer uses to report a handler or broker failure on the caller's error channel.
|
Package consumererr holds the send every messagequeue Consumer uses to report a handler or broker failure on the caller's error channel. |
|
mqmetrics
Package mqmetrics holds the instruments every messagequeue broker records, so that the four brokers agree on what each number means.
|
Package mqmetrics holds the instruments every messagequeue broker records, so that the four brokers agree on what each number means. |
|
receivewait
Package receivewait paces a consumer's receive loop after a failed receive.
|
Package receivewait paces a consumer's receive loop after a failed receive. |
|
Package kafka is a messagequeue publisher and consumer over Apache Kafka.
|
Package kafka is a messagequeue publisher and consumer over Apache Kafka. |
|
Package messagequeuemock provides moq-generated mocks for the messagequeue package's Publisher, PublisherProvider, Consumer, and ConsumerProvider interfaces.
|
Package messagequeuemock provides moq-generated mocks for the messagequeue package's Publisher, PublisherProvider, Consumer, and ConsumerProvider interfaces. |
|
Package noop is the messagequeue publisher and consumer pair for a service with no broker.
|
Package noop is the messagequeue publisher and consumer pair for a service with no broker. |
|
Package pubsub is a messagequeue publisher and consumer over Google Cloud Pub/Sub.
|
Package pubsub is a messagequeue publisher and consumer over Google Cloud Pub/Sub. |
|
Package redis is a messagequeue publisher and consumer over Redis pub/sub.
|
Package redis is a messagequeue publisher and consumer over Redis pub/sub. |
|
Package sqs is a messagequeue publisher and consumer over Amazon SQS.
|
Package sqs is a messagequeue publisher and consumer over Amazon SQS. |