Documentation
¶
Overview ¶
Package queue — the consumer engine: the pull loop that runs a Handler on a CONSUMER'S goroutine, which is the third axis of ADR 0053's frontier.
Package queue — the consumer engine's configuration, its one mandatory assertion, and the clamps that make every other field safe to omit.
Package queue — who else can write a queue directory, asked of the mode bits: the whole answer on Unix, where a directory's other-write bit and its sticky bit are exactly the two facts the question needs.
Package queue — the durable broker: one directory, three subdirectories, and rename(2) as the only thing that changes a fact.
Package queue — the durable broker's configuration, and the directory checks it runs before anything is opened.
Package queue — the durable broker's failure path: handing a message back, renewing a lease, and the record left behind for whoever investigates.
Package queue — the durable queue's state machine is a NAME. This file is its grammar: what a queued, an in-flight and an abandoned message are called, and how a name is read back into the facts it carries.
Package queue — the durable broker's read path: reclaiming the leases of consumers that died, and leasing what is visible.
Package queue — the durable broker's dead-letter decisions: putting a dead letter back into the queue, or deleting it.
Package queue — one entry in the in-memory broker's lease-deadline heap.
Package queue — the record the in-memory broker keeps about one message.
Package queue — the in-memory broker: the double a consumer's own tests run against, and the control the durability measurements are read against.
Package queue — the in-memory broker's configuration.
Package queue implements the asynchronous, durable message queue declared in internal/core/data/queue (ADR 0054): three brokers — one in the heap, one on the filesystem, one in a table of the caller's own SQL database (ADR 0151) — the consumer engine that drives a corequeue.Handler against any of them, and the dead-letter record they all write.
Three brokers, one set of refusals ¶
NewMemory exists so a consumer can test its own code without a directory or a database, and that is only worth something if the brokers answer identically. They therefore share the core policy guard (corequeue.PolicyValue.Validate), the retry delay (retryDelay), the same typed sentinels, and a table-driven conformance suite that runs the SAME cases against all three. Where they cannot be identical — surviving a restart, crossing a process boundary, joining the caller's transaction, what a flush costs — the difference is named rather than discovered by a reader.
The one thing the SQL broker does that neither other can ¶
NewSQL keeps the queue in one table of the database the application already writes, and every call runs on the transaction its context carries. So a message published inside the caller's transaction exists if and only if that transaction commits: the transactional outbox, which no broker whose publication is durable on its own can offer.
The one thing the file broker does that the memory broker cannot ¶
It survives the consumer. Every piece of state the durable queue keeps — which messages are queued, which are leased, when each lease lapses, how many times each has been delivered — lives in a DIRECTORY ENTRY, not in this process's heap. A consumer that is SIGKILLed mid-handler leaves an entry whose deadline is in the past, and the next consumer to look, in any process on that machine, renames it back and delivers it again. TestAKilledConsumerLosesItsLeaseAndTheMessageComesBack does exactly that, with a real subprocess and a real SIGKILL, because a test that calls Nack proves only that Nack works.
Why the durable broker holds no long-lived handle ¶
There is no Close in this package and no descriptor kept between calls. Every operation opens what it needs, moves what it must, and returns. That costs a syscall or two per call and buys the property the domain is for: a process that dies leaves nothing half-open, nothing to recover, and no state that only it could have interpreted. It is also what makes the broker genuinely inter-process — two brokers in two processes over one directory are the same queue, with no coordination beyond the filesystem.
Why there is no lock file ¶
Exclusion between consumers is os.Rename, not a lock. Renaming a queued message into the in-flight directory is atomic and it FAILS for the loser, so two consumers reaching for the same message resolve it in the kernel with no lock, no lease and no polling — and it resolves the same way for two goroutines in one process as for two processes, which ADR 0052 measured that flock(2) emphatically does not.
Package queue — how long a nacked message waits, the same way in all three brokers.
Package queue — the SQL broker: the queue in one table of the caller's own database, every call on the transaction its context carries (ADR 0151).
Package queue — the SQL broker's construction parameters, and the refusals a configuration no SQL broker could run gets.
Package queue — the SQL broker's dead letters: burying one a handler rejected, reading them back, and the two decisions an operator takes.
Package queue — the only place the SQL broker renders SQL. Every statement it sends is built here, once, at NewSQL, for its dialect and its table; the two whose length depends on a batch are rendered per call from the same parts. Every one is spelled with the vocabulary core/data/sql's Dialect owns: the bind markers, the quoting and the row lock.
Package queue — the SQL broker's table, as a migration the caller runs under its own version table.
Package queue — the SQL broker's read path: leasing what is due, and burying the leases whose consumers died with no attempt left.
Package queue — waking an idle consumer: the broadcast every broker closes on a publication, and the table that lets two durable brokers over one queue — one directory, or one database and table — in one process wake each other's consumers.
Index ¶
- Constants
- func Consume(ctx context.Context, broker corequeue.Broker, cfg ConsumerConfig) error
- func NewFile(cfg FileConfig) (broker corequeue.Broker, err error)
- func NewMemory(cfg MemoryConfig) (broker corequeue.Broker, err error)
- func NewSQL(cfg SQLConfig) (broker corequeue.Broker, err error)
- func SQLMigration(dialect coresql.Dialect, table string, version uint64) (coresql.MigrationValue, error)
- type ConsumerConfig
- type FileConfig
- type MemoryConfig
- type SQLConfig
Constants ¶
const DefaultPollInterval time.Duration = 100 * time.Millisecond
DefaultPollInterval is how long an idle worker waits before asking again, when ConsumerConfig.PollInterval is left at zero.
It is a CLAMP and not a refusal (ADR 0031): its zero has one reading — "you did not think about polling" — and 100 ms is defensible for every queue, because it bounds idle latency at a tenth of a second while costing one directory read per worker per tenth of a second. A caller who cares picks their own.
Every broker in this package is a corequeue.Waker, so for them it bounds only what a wake cannot see — a publication made by ANOTHER process into a durable queue — and a caller that has no such producer can raise it a hundredfold. It stays at 100 ms because a zero must keep meaning what it always meant for a broker that cannot wake anyone.
const MaxSQLTableLen int = 63
MaxSQLTableLen is the longest table name NewSQL and SQLMigration accept, in bytes: PostgreSQL's identifier limit. The broker derives no table from it, so the whole limit is the caller's.
Variables ¶
This section is empty.
Functions ¶
func Consume ¶
Consume runs cfg.Handler against broker until ctx is done.
It is the third column of ADR 0053's frontier made concrete: the handler runs on a goroutine this function owns, not on the publisher's, and it runs in a different process from the publisher whenever the broker is a durable one. Publish returned long ago and knows nothing about any of this.
It returns nil when ctx ends — a cancelled consumer is a stopped consumer, not a failed one — and returns the broker's error when the BROKER fails, because a queue whose storage has stopped answering is not something a consumer can poll its way out of. A HANDLER's error is never returned: it is nacked, retried, and eventually dead-lettered, which is the entire point of the domain.
Every worker returns before Consume does, so a caller that cancels and waits knows no handler is still running.
func NewFile ¶
func NewFile(cfg FileConfig) (broker corequeue.Broker, err error)
NewFile returns a broker whose messages survive the process that published them.
Why internal/service/data/vfs, and where it stops ¶
The durable publication of a message is exactly what corevfs.AtomicWriter promises: a temporary in the same directory, the payload, an fsync of the FILE, a rename, an fsync of the DIRECTORY — five steps whose failure paths ADR 0056 tested by injecting a failure at each one and comparing the destination's hash. Rewriting that here would be eighty lines of subtle code duplicating something the SDK merged, measured and proved, and the first thing to rot would be the failure paths, which are the only part that matters.
It stops at corevfs.WritableFS, which has WriteFile, MkdirAll, Remove and RemoveAll and NO RENAME. That is not an oversight in vfs and it is not worked around here: the port is FROZEN (ADR 0039), so adding a fifth method to carry this domain's state machine would break every downstream implementation at compile time. The state transitions therefore go through os.Root directly, which confines them to the queue directory by the same mechanism vfs itself uses, and the boundary between the two is exactly the boundary between "publish a whole file" and "move one".
It refuses, at construction rather than at first use: a policy it cannot honour (corequeue.QueueMisconfigured), a directory it cannot use safely (corequeue.QueueDirectoryUnusable), and a platform with no atomic replace or no flushable directory handle — core/proc.UnsupportedPlatform, raised by internal/service/data/vfs.NewOS and passed through unmodified (ADR 0018).
func NewMemory ¶
func NewMemory(cfg MemoryConfig) (broker corequeue.Broker, err error)
NewMemory returns an in-process broker holding its messages in the heap.
It refuses a policy it cannot honour at CONSTRUCTION rather than at first use, through the same core/data/queue.PolicyValue.Validate the file broker runs — which is what makes this an honest double for it.
func NewSQL ¶
NewSQL returns a broker whose queue is one table of the caller's database, created by SQLMigration. It sends no statement: everything decidable without a database is decided here.
It refuses, at construction rather than at first use, a policy it cannot honour (core/data/queue.QueueMisconfigured) and a transactor, dialect or table it cannot use (corequeue.SQLQueueMisconfigured).
func SQLMigration ¶
func SQLMigration(dialect coresql.Dialect, table string, version uint64) (coresql.MigrationValue, error)
SQLMigration returns the migration that creates the one table an SQL queue named table keeps on dialect, as a core/data/sql MigrationValue numbered version, for the caller's Migrator to run under the caller's version table. Its Down drops the table, and every message in it — queued, leased and dead.
The SDK numbers nothing: a migration is a value its consumer constructs (ADR 0055 §D12), and only the consumer knows where the queue's table sits in its own history. Its one statement does nothing when the table already exists — or is already gone — so applying it over a table a previous deployment made changes nothing (ADR 0151).
Types ¶
type ConsumerConfig ¶
type ConsumerConfig struct {
// Handler processes one delivery. It must be non-nil.
Handler corequeue.Handler
// Clock drives the idle wait. Nil means clock.System.
//
// Nothing in this package calls time.Sleep; an idle worker waits through
// Waiter.After and a cancelled context wakes it immediately.
Clock clock.Timed
// PollInterval is how long a worker waits after finding the queue empty.
// Zero means [DefaultPollInterval]; negative is read as zero.
//
// When the broker is a [corequeue.Waker] — every broker in this package
// is — it is an upper bound and not a cadence: a Publish or a Nack in
// this process wakes the worker at once, and a retry delay ending or a
// lease lapsing wakes it at that instant. What is left for the poll to
// find is a publication another process made into a durable queue, so set
// it to how late such a message may be noticed — seconds, not
// milliseconds, is the usual answer — and an idle consumer then costs
// nothing between polls.
PollInterval time.Duration
// Parallelism is how many independent workers run. Each one polls, leases
// and processes on its OWN goroutine, which is what makes this the
// consumer's goroutine rather than the publisher's.
//
// Non-positive is clamped to 1: one worker is what a caller who did not
// think about concurrency wants, it is never wrong, and there is nothing
// here for the SDK to invent.
//
// Above 1 there is NO ordering left, in any sense.
Parallelism int
// BatchSize is how many messages one worker leases per poll. Non-positive
// is clamped to 1.
//
// Above 1 a worker processes its batch SEQUENTIALLY, so the extra
// messages are held under a lease they are not yet being worked on: a
// batch of 50 with a 30-second visibility timeout gives the last message
// 30 seconds minus the other 49's processing time. Batch for throughput
// against a durable broker, where one poll is a directory read; do not
// batch to get parallelism, which is what Parallelism is.
BatchSize int
// HandlerIsIdempotent asserts, IN CODE, that Handler is safe to run more
// than once for the same message.
//
// It must be true. Its zero value is false and is REFUSED
// ([corequeue.ConsumerMisconfigured]), and that refusal is the domain's
// one piece of deliberate ceremony.
//
// The reason is that this queue delivers AT LEAST ONCE and cannot do
// otherwise. A consumer that dies between finishing the work and
// acknowledging it has done the work and not said so, and the message
// comes back — that is not a bug to be fixed, it is the only honest
// behaviour available to a queue whose consumers can die. Exactly-once
// delivery does not exist over a transport; what exists is at-least-once
// plus an idempotent consumer, and the second half is the caller's.
//
// The SDK cannot check it, so it makes the caller assert it where a
// reviewer will see it. It is precisely
// resilience.HedgeConfig.Idempotent's instrument, for precisely its
// reason, and it has the same useful side effect:
// `grep -rn HandlerIsIdempotent` enumerates every place in a codebase
// where somebody promised this.
HandlerIsIdempotent bool
}
ConsumerConfig configures Consume: what to run, how many of it, how often to look, and the one thing about the handler the SDK cannot check for itself.
Every field except Handler and HandlerIsIdempotent is CLAMPED, so a caller with no opinion writes two fields and gets a correct consumer.
type FileConfig ¶
type FileConfig struct {
// Clock reads time. It is the narrow half of kernel/clock because this
// broker never waits: a lapsed lease is noticed by the next Receive, not
// by a timer.
//
// Injecting a ManualClock makes every deadline in a SINGLE-process test
// instant. It cannot make a cross-process test instant, and that
// asymmetry is not an oversight: a lease deadline is a wall-clock instant
// written into a filename precisely so that another process can read it,
// and another process does not share this one's fake clock. The
// consumer-death test therefore waits on the real clock, and says so.
//
// Nil means clock.System.
Clock clock.Clock
// Dir is the queue directory. It is created owner-only when absent, and
// refused when it exists and any account outside the owner and group
// could replace its entries. Its three state directories are held to the
// same rule — created owner-only, checked when somebody else made them —
// and refused besides when one is a symlink or not a directory at all.
// On Windows "owner-only" is not something a mode can say: a directory
// inherits its parent's access control list, and the rule reads that list
// instead (dirtrust_windows.go).
//
// Two brokers over one Dir — in one process or in two — are ONE queue.
// That is the domain's inter-process claim, and it is the whole reason
// the directory is the configuration rather than an implementation
// detail.
Dir string
// Policy is the delivery discipline. Its zero value is refused; see
// core/data/queue.PolicyValue.
Policy corequeue.PolicyValue
}
FileConfig configures NewFile: which directory holds the queue, which clock reads its deadlines, and what delivery discipline it enforces.
Its Policy's zero value is refused at construction — see core/data/queue.PolicyValue — and so is a directory whose entries any account could replace.
type MemoryConfig ¶
type MemoryConfig struct {
// Clock reads time. A memory broker never WAITS — expiry is noticed on
// the next Receive — so it asks for the narrow half of kernel/clock and
// a test can drive every deadline in this package with a ManualClock and
// no sleeping at all.
//
// Nil means clock.System.
Clock clock.Clock
// Policy is the delivery discipline. Its zero value is refused; see
// core/data/queue.PolicyValue.
Policy corequeue.PolicyValue
}
MemoryConfig configures NewMemory: which clock reads its deadlines, and what delivery discipline it enforces.
Its Policy's zero value is refused at construction through the SAME guard NewFile runs, which is what makes this broker an honest double for that one.
type SQLConfig ¶
type SQLConfig struct {
// Transactor is the transaction manager of the database the queue lives
// in — the one the caller's own transactions are opened with, so a call
// made under one of them joins it. It must also be a core/data/sql Joiner and
// Deferrer, which the SDK's own transactor is: the first is how a
// publication joins the caller's transaction, the second how it wakes a
// consumer only once that transaction has committed.
Transactor coresql.Transactor
// Clock stamps every instant the broker writes — a message's visibility,
// a lease's deadline, a dead letter's failure — and reads the instant a
// lease is judged against. Nil means clock.System.
//
// The instants are the BROKER's, never the database's NOW(): a test
// drives every deadline with a ManualClock, as it does the other two
// brokers'. Brokers in several processes over one table compare
// instants their own clocks wrote, so those clocks must agree to within
// the precision a lease needs — as they must for the file broker.
Clock clock.Clock
// Table names the queue's one table, which [SQLMigration] creates. It is
// a lower-case SQL identifier of at most [MaxSQLTableLen] bytes, holds no
// three underscores in a row — docstore's derived tables do — and does
// not start with "sqlite_".
//
// Two brokers over one database and one Table — in one process or in
// twenty — are ONE queue, as two file brokers over one directory are.
Table string
// Policy is the delivery discipline. Its zero value is refused; see
// core/data/queue.PolicyValue.
Policy corequeue.PolicyValue
// Dialect is the engine's, and must be the Transactor's own: the SQL the
// broker speaks to it.
Dialect coresql.Dialect
}
SQLConfig configures NewSQL: the database the queue lives in, its table, which clock stamps its instants, and what delivery discipline it enforces. Transactor, Dialect and Table are required, and Policy's zero value is refused as it is for every broker.