queue

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 13, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package queue — range 0.3.53.* (ADR 0054 service/queue block).

Package queue — the consumer engine: the pull loop that runs a Handler on a CONSUMER'S goroutine, which is the third axis of ADR 0053's frontier.

Package queue — the consumer engine's configuration, its one mandatory assertion, and the clamps that make every other field safe to omit.

Package queue — declares the sentinel *errs.Error implementation outcomes. Each var's name equals its errs.Define Reason in SCREAMING_SNAKE form.

Package queue — the durable broker: one directory, three subdirectories, and rename(2) as the only thing that changes a fact.

Package queue — the durable broker's configuration, and the directory checks it runs before anything is opened.

Package queue — the durable broker's failure path: handing a message back, renewing a lease, and the record left behind for whoever investigates.

Package queue — the durable queue's state machine is a NAME. This file is its grammar: what a queued, an in-flight and an abandoned message are called, and how a name is read back into the facts it carries.

Package queue — the durable broker's read path: reclaiming the leases of consumers that died, and leasing what is visible.

Package queue — one entry in the in-memory broker's lease-deadline heap.

Package queue — the record the in-memory broker keeps about one message.

Package queue — the in-memory broker: the double a consumer's own tests run against, and the control the durability measurements are read against.

Package queue — the in-memory broker's configuration.

Package queue implements the asynchronous, durable message queue declared in internal/core/queue (ADR 0054): two brokers — one in the heap, one on the filesystem — the consumer engine that drives a corequeue.Handler against either, and the dead-letter record they both write.

Two brokers, one set of refusals

NewMemory exists so a consumer can test its own code without a directory, and that is only worth something if the two answer identically. They therefore share the core policy guard (corequeue.PolicyValue.Validate), the same typed sentinels, and a table-driven conformance suite that runs the SAME cases against both. Where they cannot be identical — surviving a restart, crossing a process boundary, what a flush costs — the difference is named here rather than discovered by a reader.

The one thing the file broker does that the memory broker cannot

It survives the consumer. Every piece of state the durable queue keeps — which messages are queued, which are leased, when each lease lapses, how many times each has been delivered — lives in a DIRECTORY ENTRY, not in this process's heap. A consumer that is SIGKILLed mid-handler leaves an entry whose deadline is in the past, and the next consumer to look, in any process on that machine, renames it back and delivers it again. TestAKilledConsumerLosesItsLeaseAndTheMessageComesBack does exactly that, with a real subprocess and a real SIGKILL, because a test that calls Nack proves only that Nack works.

Why the durable broker holds no long-lived handle

There is no Close in this package and no descriptor kept between calls. Every operation opens what it needs, moves what it must, and returns. That costs a syscall or two per call and buys the property the domain is for: a process that dies leaves nothing half-open, nothing to recover, and no state that only it could have interpreted. It is also what makes the broker genuinely inter-process — two brokers in two processes over one directory are the same queue, with no coordination beyond the filesystem.

Why there is no lock file

Exclusion between consumers is os.Rename, not a lock. Renaming a queued message into the in-flight directory is atomic and it FAILS for the loser, so two consumers reaching for the same message resolve it in the kernel with no lock, no lease and no polling — and it resolves the same way for two goroutines in one process as for two processes, which ADR 0052 measured that flock(2) emphatically does not.

Index

Constants

View Source
const CodeConsumerMisconfigured errs.Code = 0x00_03_35_03 // 0.3.53.3

CodeConsumerMisconfigured identifies a ConsumerConfig the engine refuses: no Handler, or a Handler whose author has not asserted that it is idempotent under at-least-once delivery.

View Source
const CodeHandlerPanicked errs.Code = 0x00_03_35_04 // 0.3.53.4

CodeHandlerPanicked identifies a handler whose call panicked; the engine recovered it, nacked the message, and kept consuming.

There is deliberately no companion CodeHandlerFailed. A handler that merely returns an error hands that error to Nack UNMODIFIED — see Consume's guard — because the destination is a dead-letter record whose structured fields already say "a handler failed", and an engine verdict wrapped around the cause would displace the reason an investigator needs.

View Source
const CodeQueueBackendFailed errs.Code = 0x00_03_35_01 // 0.3.53.1

CodeQueueBackendFailed identifies a filesystem operation the durable broker depends on that the operating system refused.

View Source
const CodeQueueDirectoryUnusable errs.Code = 0x00_03_35_02 // 0.3.53.2

CodeQueueDirectoryUnusable identifies a queue directory that cannot be prepared, is not a directory, or is writable by accounts that must not be able to inject or remove messages.

View Source
const DefaultPollInterval time.Duration = 100 * time.Millisecond

DefaultPollInterval is how long an idle worker waits before asking again, when ConsumerConfig.PollInterval is left at zero.

It is a CLAMP and not a refusal (ADR 0031): its zero has one reading — "you did not think about polling" — and 100 ms is defensible for every queue, because it bounds idle latency at a tenth of a second while costing one directory read per worker per tenth of a second. A caller who cares picks their own.

Variables

View Source
var (
	// QueueBackendFailed wraps a refusal from the operating system: an open,
	// a rename, an unlink, a directory read or a flush.
	//
	// The cause stays in the chain, so a caller may still ask
	// errors.Is(err, fs.ErrNotExist) or reach the *fs.PathError. Its fields
	// name the operation and, where it is not the payload, the path — never
	// the message bytes.
	QueueBackendFailed = errs.Define(CodeQueueBackendFailed, "QUEUE_BACKEND_FAILED",
		"The queue's storage refused an operation",
		"service/queue: a filesystem call failed; the fields name the operation and the path, never the payload",
		errs.WithExitCode(exitIOErr))

	// QueueDirectoryUnusable refuses a queue directory at CONSTRUCTION.
	//
	// A durable queue whose directory any account may write is a queue any
	// account may inject work into or silently drain, and the failure is
	// invisible: messages simply stop arriving. It is the same refusal
	// service/lock makes about its lock directory, and it is made at wiring
	// time for the same reason — at first use it would be a mystery, at
	// construction it names the directory.
	QueueDirectoryUnusable = errs.Define(CodeQueueDirectoryUnusable, "QUEUE_DIRECTORY_UNUSABLE",
		"The queue directory cannot be used safely",
		"service/queue: the queue directory is missing, is not a directory, or is writable by other accounts",
		errs.WithExitCode(exitConfig))

	// ConsumerMisconfigured refuses a ConsumerConfig the engine cannot run.
	//
	// It is the sentinel behind the domain's one mandatory assertion. This
	// queue delivers AT LEAST ONCE, so a handler will be called twice for
	// one message — after a crash, after a lease expiry, after a slow
	// downstream. The SDK cannot detect whether that is safe, so it makes
	// the caller say so in code: ConsumerConfig.HandlerIsIdempotent, whose
	// zero value is false and is refused. That is exactly
	// resilience.HedgeConfig.Idempotent's instrument, for exactly its reason
	// — a precondition the SDK cannot check becomes a required, greppable
	// assertion instead of a comment nobody reads.
	ConsumerMisconfigured = errs.Define(CodeConsumerMisconfigured, "CONSUMER_MISCONFIGURED",
		"The consumer configuration is not runnable and was refused",
		"service/queue: Handler is nil, or HandlerIsIdempotent is false under an at-least-once queue; the field names which",
		errs.WithExitCode(exitConfig))

	// HandlerPanicked is the failure of a handler whose call panicked.
	//
	// The engine recovers it rather than letting it reach the runtime: a
	// panic escaping one handler would kill the consumer PROCESS, abandoning
	// every other in-flight lease — which then has to time out, one
	// visibility timeout at a time, before anything moves again. Recovered,
	// it becomes an ordinary nack: this message is retried and eventually
	// dead-lettered, and its siblings are untouched.
	//
	// The recovered value and the panicking goroutine's stack travel as
	// FIELDS, never as the wrap origin, so a panic carrying an *errs.Error
	// cannot hijack this code.
	// A handler that merely RETURNS an error gets no sentinel of its own:
	// Consume hands that error to Broker.Nack unmodified. service/events
	// joins its ListenerFailed verdict beside the listener's cause because
	// Publish returns the aggregate to a caller who must be able to ask "did
	// anything fail?"; here the aggregate's destination is a dead-letter
	// record, which already says that by existing and has structured fields
	// for the rest. A verdict around the cause would displace the reason an
	// investigator needs, and TestAFailingHandlerIsRetriedAndThenDeadLettered
	// pins that it does not.
	HandlerPanicked = errs.Define(CodeHandlerPanicked, "HANDLER_PANICKED",
		"The queue handler panicked and was recovered",
		"service/queue: handler panicked; the fields carry the message id, the panic value and the stack")
)

Functions

func Consume

func Consume(ctx context.Context, broker corequeue.Broker, cfg ConsumerConfig) error

Consume runs cfg.Handler against broker until ctx is done.

It is the third column of ADR 0053's frontier made concrete: the handler runs on a goroutine this function owns, not on the publisher's, and it runs in a different process from the publisher whenever the broker is a durable one. Publish returned long ago and knows nothing about any of this.

It returns nil when ctx ends — a cancelled consumer is a stopped consumer, not a failed one — and returns the broker's error when the BROKER fails, because a queue whose storage has stopped answering is not something a consumer can poll its way out of. A HANDLER's error is never returned: it is nacked, retried, and eventually dead-lettered, which is the entire point of the domain.

Every worker returns before Consume does, so a caller that cancels and waits knows no handler is still running.

func NewFile

func NewFile(cfg FileConfig) (broker corequeue.Broker, err error)

NewFile returns a broker whose messages survive the process that published them.

Why internal/service/vfs, and where it stops

The durable publication of a message is exactly what corevfs.AtomicWriter promises: a temporary in the same directory, the payload, an fsync of the FILE, a rename, an fsync of the DIRECTORY — five steps whose failure paths ADR 0056 tested by injecting a failure at each one and comparing the destination's hash. Rewriting that here would be eighty lines of subtle code duplicating something the SDK merged, measured and proved, and the first thing to rot would be the failure paths, which are the only part that matters.

It stops at corevfs.WritableFS, which has WriteFile, MkdirAll, Remove and RemoveAll and NO RENAME. That is not an oversight in vfs and it is not worked around here: the port is FROZEN (ADR 0039), so adding a fifth method to carry this domain's state machine would break every downstream implementation at compile time. The state transitions therefore go through os.Root directly, which confines them to the queue directory by the same mechanism vfs itself uses, and the boundary between the two is exactly the boundary between "publish a whole file" and "move one".

It refuses, at construction rather than at first use: a policy it cannot honour (corequeue.QueueMisconfigured), a directory it cannot use safely (QueueDirectoryUnusable), and a platform with no atomic replace or no flushable directory handle — core/proc.UnsupportedPlatform, raised by internal/service/vfs.NewOS and passed through unmodified (ADR 0018).

func NewMemory

func NewMemory(cfg MemoryConfig) (broker corequeue.Broker, err error)

NewMemory returns an in-process broker holding its messages in the heap.

It refuses a policy it cannot honour at CONSTRUCTION rather than at first use, through the same core/queue.PolicyValue.Validate the file broker runs — which is what makes this an honest double for it.

Types

type ConsumerConfig

type ConsumerConfig struct {
	// Handler processes one delivery. It must be non-nil.
	Handler corequeue.Handler
	// Clock drives the idle wait. Nil means clock.System.
	//
	// Nothing in this package calls time.Sleep; an idle worker waits through
	// Waiter.After and a cancelled context wakes it immediately.
	Clock clock.Timed
	// PollInterval is how long a worker waits after finding the queue empty.
	// Zero means [DefaultPollInterval]; negative is read as zero.
	PollInterval time.Duration
	// Parallelism is how many independent workers run. Each one polls, leases
	// and processes on its OWN goroutine, which is what makes this the
	// consumer's goroutine rather than the publisher's.
	//
	// Non-positive is clamped to 1: one worker is what a caller who did not
	// think about concurrency wants, it is never wrong, and there is nothing
	// here for the SDK to invent.
	//
	// Above 1 there is NO ordering left, in any sense.
	Parallelism int
	// BatchSize is how many messages one worker leases per poll. Non-positive
	// is clamped to 1.
	//
	// Above 1 a worker processes its batch SEQUENTIALLY, so the extra
	// messages are held under a lease they are not yet being worked on: a
	// batch of 50 with a 30-second visibility timeout gives the last message
	// 30 seconds minus the other 49's processing time. Batch for throughput
	// against a durable broker, where one poll is a directory read; do not
	// batch to get parallelism, which is what Parallelism is.
	BatchSize int
	// HandlerIsIdempotent asserts, IN CODE, that Handler is safe to run more
	// than once for the same message.
	//
	// It must be true. Its zero value is false and is REFUSED
	// ([ConsumerMisconfigured]), and that refusal is the domain's one piece
	// of deliberate ceremony.
	//
	// The reason is that this queue delivers AT LEAST ONCE and cannot do
	// otherwise. A consumer that dies between finishing the work and
	// acknowledging it has done the work and not said so, and the message
	// comes back — that is not a bug to be fixed, it is the only honest
	// behaviour available to a queue whose consumers can die. Exactly-once
	// delivery does not exist over a transport; what exists is at-least-once
	// plus an idempotent consumer, and the second half is the caller's.
	//
	// The SDK cannot check it, so it makes the caller assert it where a
	// reviewer will see it. It is precisely
	// resilience.HedgeConfig.Idempotent's instrument, for precisely its
	// reason, and it has the same useful side effect:
	// `grep -rn HandlerIsIdempotent` enumerates every place in a codebase
	// where somebody promised this.
	HandlerIsIdempotent bool
}

ConsumerConfig configures Consume: what to run, how many of it, how often to look, and the one thing about the handler the SDK cannot check for itself.

Every field except Handler and HandlerIsIdempotent is CLAMPED, so a caller with no opinion writes two fields and gets a correct consumer.

type FileConfig

type FileConfig struct {
	// Clock reads time. It is the narrow half of kernel/clock because this
	// broker never waits: a lapsed lease is noticed by the next Receive, not
	// by a timer.
	//
	// Injecting a ManualClock makes every deadline in a SINGLE-process test
	// instant. It cannot make a cross-process test instant, and that
	// asymmetry is not an oversight: a lease deadline is a wall-clock instant
	// written into a filename precisely so that another process can read it,
	// and another process does not share this one's fake clock. The
	// consumer-death test therefore waits on the real clock, and says so.
	//
	// Nil means clock.System.
	Clock clock.Clock
	// Dir is the queue directory. It is created owner-only when absent, and
	// refused when it exists and any account outside the owner and group
	// could replace its entries. Its three state directories are held to the
	// same rule — created owner-only, checked when somebody else made them —
	// and refused besides when one is a symlink or not a directory at all.
	//
	// Two brokers over one Dir — in one process or in two — are ONE queue.
	// That is the domain's inter-process claim, and it is the whole reason
	// the directory is the configuration rather than an implementation
	// detail.
	Dir string
	// Policy is the delivery discipline. Its zero value is refused; see
	// core/queue.PolicyValue.
	Policy corequeue.PolicyValue
}

FileConfig configures NewFile: which directory holds the queue, which clock reads its deadlines, and what delivery discipline it enforces.

Its Policy's zero value is refused at construction — see core/queue.PolicyValue — and so is a directory whose entries any account could replace.

type MemoryConfig

type MemoryConfig struct {
	// Clock reads time. A memory broker never WAITS — expiry is noticed on
	// the next Receive — so it asks for the narrow half of kernel/clock and
	// a test can drive every deadline in this package with a ManualClock and
	// no sleeping at all.
	//
	// Nil means clock.System.
	Clock clock.Clock
	// Policy is the delivery discipline. Its zero value is refused; see
	// core/queue.PolicyValue.
	Policy corequeue.PolicyValue
}

MemoryConfig configures NewMemory: which clock reads its deadlines, and what delivery discipline it enforces.

Its Policy's zero value is refused at construction through the SAME guard NewFile runs, which is what makes this broker an honest double for that one.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL