stream

package
v0.30.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: MIT Imports: 36 Imported by: 0

Documentation

Overview

Package stream holds the engines of core/data/stream's corestream.Stream port — memory, a directory of segments over kernel/fs/seglog, and a SQL table joined to the caller's transaction — and the subscriber that reads a stream on behalf of a consumer group (ADR 0231). The engines arrive one per milestone of kitsunium/sdk#890; their error codes, 0.3.94.*, are already declared in the core at this path (ADR 0160).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewFile

func NewFile(cfg FileConfig) (s corestream.Stream, err error)

NewFile opens the stream kept in cfg.Dir, creating it when absent, and recovers it: every partition's segmented log, the batches and instants its records carry, and the retention they were under.

It refuses a policy it cannot honour with the core's Validate, as every engine does, and a MaxRecordBytes above what one segment record holds; a directory it cannot trust, or one another engine holds open, with StreamDirectoryUnusable; a directory holding more partitions than the policy counts with StreamMisconfigured; and a segment it cannot trust with the segmented log's own refusal under StreamBackendFailed. The stream it returns also answers corestream.Waker, corestream.Seeker, corestream.Trimmer, corestream.Compactor, corestream.Committer and io.Closer; Close releases the directory. A compaction rewrites the segments holding a record it drops, each published by a rename then a directory sync, a dropped record leaving its offset and instant behind.

The groups' cursors are published through vfs's disk filesystem, so where vfs has none — Windows, illumos and Solaris, which lack a flushable directory handle or a mode that is an access rule (ADR 0018) — NewFile refuses with proc.UnsupportedPlatform, under StreamBackendFailed, before it reads the policy or the directory, and returns no stream.

func NewMemory

func NewMemory(cfg MemoryConfig) (s corestream.Stream, err error)

NewMemory returns a stream whose partitions live in the heap of this process: the test double of every durable engine.

It refuses a policy it cannot honour at CONSTRUCTION, through the same core/data/stream.PolicyValue.Validate every engine runs, which is what makes it an honest double. The stream it returns also answers corestream.Waker, corestream.Seeker, corestream.Trimmer, corestream.Compactor and corestream.Committer; it keeps nothing past the process, so a restart is a total loss.

func NewSQL

func NewSQL(cfg SQLConfig) (s corestream.Stream, err error)

NewSQL returns a stream kept in three tables of the caller's database, created by SQLMigration: the records, one head row per partition and the groups' cursors. It sends no statement: everything decidable without a database is decided here.

An Append inside a transaction its context carries is a savepoint of that transaction (ADR 0151): its records exist if and only if the transaction commits, and the stream's Waker fires only after the commit. The head row of the partition is locked for the rest of that transaction, so the appenders of one partition are serialised and its offsets commit in order, with no gap a reader could skip. A Commit made on a handler's own transaction commits the group's offset with the handler's effect.

It refuses a policy it cannot honour with the core's Validate, as every engine does, and a RetainBytes or MaxBytes, which it does not keep, with corestream.SQLStreamMisconfigured; a transactor, dialect or table it cannot use the same way. The stream it returns also answers corestream.Waker, corestream.Seeker, corestream.Trimmer, corestream.Compactor and corestream.Committer; a compaction deletes the rows it drops in bounded statements.

func SQLMigration

func SQLMigration(dialect coresql.Dialect, table string, version uint64) (coresql.MigrationValue, error)

SQLMigration returns the migration that creates the three tables an SQL stream named table keeps on dialect — table, table__heads and table__cursors —, as a core/data/sql MigrationValue numbered version, for the caller's Migrator to run under the caller's version table. Its Down drops the three, and every record and cursor in them.

The SDK numbers nothing (ADR 0055 §D12). Each statement does nothing when its table already exists, or is already gone, so applying it over tables a previous deployment made changes nothing. Its Checksum fingerprints the statements Up runs, which are therefore never edited: a later column comes as a migration of its own.

func Subscribe

func Subscribe(ctx context.Context, s corestream.Stream, cfg SubscriberConfig) error

Subscribe reads s on behalf of the consumer group cfg.Group until ctx ends: one worker per partition it is assigned, each handing cfg.Handler the partition's records in offset order, batch after batch, and moving the group's offset past each batch the handler accepted. It is internal/service/data/queue.Consume for a stream: the handler runs on a goroutine Subscribe owns, a handler's error is never returned, and every worker has returned before Subscribe does.

A worker starts at the offset the group committed on its partition, or at the partition's first retained offset when the group never committed there; an offset retention has since removed resumes at the first one still kept, the records in between never delivered. It takes the stream's corestream.Waker signal before each Read, so an Append in this process wakes it at once, and waits at most cfg.PollInterval on cfg.Clock for what another process appended.

A failing handler blocks its partition, by design

A batch whose handler fails is handed to it again after the cfg.Retry wait, and the partition goes no further until it is accepted: the records of one key are delivered in the order they were appended, which is the promise a stream makes and a queue does not (ADR 0054's deferred per-key ordering, kept here as head-of-line blocking stated rather than hidden). The other partitions carry on. With cfg.Park, a batch failing After times is narrowed to its first record, and a record failing After times alone is published to the park's queue as a dead letter and the group's offset committed past it.

What a stop or a crash leaves

A stop cancels the handler's context. A batch accepted as the stop lands is still committed, on ctx without its cancellation, bounded by cfg.SettleTimeout on cfg.Clock, as a dead letter is published and a fence released. Under CommitAfterHandler a crash between the handler's return and the commit delivers the batch again; under CommitByHandler on the SQL engine, the commit being part of the handler's own transaction, there is no such moment.

Fencing

With cfg.Fence each worker holds the lock "stream:<Name>:<Group>:<partition>" while it reads the partition, so two processes running the same group over the same partitions never read one partition at once: a worker whose lock is held elsewhere waits a PollInterval and asks again. A lease that can expire is renewed a third of its life after it was taken or last renewed; a renewal refused gives the partition up — its batch is not committed — and the worker goes back to asking for the lock, resuming at whatever the group committed.

It returns nil when ctx ends, corestream.SubscriberMisconfigured for a configuration it cannot run, and the storage's error — the stream's, the park's queue's, the fence's — when one of them refuses, which stops every worker: a medium that has stopped answering is not one a subscriber can poll its way out of.

Types

type CommitMode

type CommitMode uint8

CommitMode is who moves a group's offset past a batch the handler accepted: CommitAfterHandler (the zero) or CommitByHandler.

const (
	// CommitAfterHandler is the zero CommitMode: once the handler returns nil,
	// the subscriber commits the offset after the batch's last record on the
	// stream's [corestream.Committer]. A stop or a crash between that return
	// and the commit delivers the batch again — at least once, on every engine.
	CommitAfterHandler CommitMode = 0
	// CommitByHandler leaves the commit to the handler, which calls the
	// stream's [corestream.Committer] Commit itself — the group, the batch's
	// partition, the offset after its last record — inside the transaction that
	// holds its effect. On the SQL engine the effect and the offset then commit
	// or roll back together: exactly-once in effect. On an engine whose cursors
	// live apart from the handler's effect — memory, file — it stays at least
	// once, the cursor durable after the handler. The subscriber commits
	// nothing for a batch the handler accepted, only past a record it parked.
	CommitByHandler CommitMode = 1
)

The two ways Subscribe moves a group's offset past a batch its handler accepted. The zero is the subscriber's own commit, which asks nothing of the handler.

type FileConfig

type FileConfig struct {
	// Clock stamps [corestream.EntryValue].AppendedAt, ages records for
	// [corestream.PolicyValue].RetainFor, and times the wait
	// [corestream.PolicyValue].BlockTimeout allows. The instants are kept
	// in the segments, so a stream reopened on another clock ages its
	// records by that one. Nil means clock.System.
	Clock clock.Timed
	// Dir is the stream directory: one segmented log per partition under
	// p<partition>/, the groups' cursors under cursors/. It is created
	// owner-only when absent, refused when it exists and an account
	// outside the owner and group could replace its entries, and its
	// subdirectories are refused besides when one is a symbolic link, not a
	// directory, or writable by such an account at all — the queue's rule
	// (on Windows, read from the access-control list). One engine at a time
	// opens a directory: a second, in this process or another, is refused
	// while the first is open.
	Dir string
	// Policy is what the stream enforces; its zero value works. Its
	// Partitions cannot shrink across a reopen: a directory holding a
	// partition the policy no longer counts is refused.
	Policy corestream.PolicyValue
}

FileConfig configures NewFile: the directory that holds the stream, the clock that stamps and ages its records, and the policy it enforces.

type Handler

type Handler func(ctx context.Context, entries []corestream.EntryValue) error

Handler processes one batch of a partition's records, in offset order: entries is never empty and holds one partition only. A nil return accepts the whole batch; an error, or a panic, refuses it, and the batch is handed back after the retry wait. The entries' bytes are the handler's for the call; a handler that keeps them past it copies them.

type MemoryConfig

type MemoryConfig struct {
	// Clock stamps [corestream.EntryValue].AppendedAt, ages records for
	// [corestream.PolicyValue].RetainFor, and times the wait
	// [corestream.PolicyValue].BlockTimeout allows, so a test drives
	// retention and backpressure with a ManualClock and no sleeping. Nil
	// means clock.System.
	Clock clock.Timed
	// Policy is what the stream enforces. Its zero value works: one
	// partition, records of up to corestream.DefaultMaxRecordBytes, kept
	// forever, never full.
	Policy corestream.PolicyValue
}

MemoryConfig configures NewMemory: the policy it enforces and the clock that stamps every batch and times every wait.

type ParkConfig

type ParkConfig struct {
	// After is how many consecutive refusals park a record: a batch
	// refused After times is narrowed to its first record, and a record
	// refused After times alone is parked. Below 1 it is read as 1.
	After int
	// Broker is the queue the dead letters are published to. It must be
	// non-nil; its MaxMessageBytes must hold the JSON of the largest
	// record, or the park is refused and Subscribe returns that refusal.
	Broker corequeue.Broker
}

ParkConfig sends the records a handler keeps refusing to a queue as dead letters, so their partition moves on: the dead letter's payload is the record's corestream.EntryValue encoded by encoding/json, which json.Unmarshal decodes back into one.

type SQLConfig

type SQLConfig struct {
	// Transactor is the transaction manager of the database the stream
	// lives in — the one the caller's own transactions are opened with, so
	// a call made under one of them joins it. It must also be a
	// core/data/sql Joiner and Deferrer, which the SDK's own transactor
	// is: the first is how an Append or a Commit joins the caller's
	// transaction, the second how the Waker fires only once that
	// transaction has committed.
	Transactor coresql.Transactor
	// Dialect is the engine's, and must be the Transactor's own: the SQL
	// the stream speaks to it.
	Dialect coresql.Dialect
	// Table names the records' table; the heads and the cursors are kept
	// in Table__heads and Table__cursors, which [SQLMigration] creates
	// with it. It is a lower-case SQL identifier of at most 54 bytes —
	// PostgreSQL's 63 less the longest suffix —, words of letters and
	// digits joined by single underscores, starting with a letter and not
	// with "sqlite_". Two streams over one database and one Table, in one
	// process or in twenty, are ONE stream.
	Table string
	// Policy is what the stream enforces; its zero value works. Its
	// RetainBytes and MaxBytes must be zero: the engine keeps no byte
	// count. Its Partitions must stay the same for the life of the
	// tables: a key's partition is its hash modulo that number.
	Policy corestream.PolicyValue
	// Clock stamps [corestream.EntryValue].AppendedAt and ages records for
	// [corestream.PolicyValue].RetainFor. The instants are the ENGINE's,
	// never the database's NOW(), so a test drives retention with a
	// ManualClock; engines in several processes over one table compare
	// instants their own clocks wrote, and a batch is never stamped before
	// the partition's previous one. Nil means clock.System.
	Clock clock.Clock
}

SQLConfig configures NewSQL: the database the stream lives in, its tables, the clock that stamps its records, and the policy it enforces. Transactor, Dialect and Table are required; Policy's zero value works.

type SubscriberConfig

type SubscriberConfig struct {
	// BatchSize is the most records one Read hands the handler. Below 1
	// it is read as 1.
	BatchSize int
	// Clock times the idle wait, the retry wait, the bound on what is sent
	// past a stop and the fence's renewals. Nil means clock.System;
	// nothing in the subscriber calls time.Sleep.
	Clock clock.Timed
	// Commit is who moves the offset past an accepted batch:
	// [CommitAfterHandler], the zero, or [CommitByHandler]. Any other
	// value is refused.
	Commit CommitMode
	// Fence, when set, makes each worker hold the lock
	// "stream:<Name>:<Group>:<partition>" while it reads its partition,
	// so the same group in two processes never reads one partition
	// twice at once. Nil fences nothing: one process runs the group.
	Fence corelock.Locker
	// Group names the consumer group whose offsets are read and
	// committed. It must not be empty; two groups over one stream read
	// every record each, at their own pace.
	Group string
	// Handler processes each batch. It must be non-nil.
	Handler Handler
	// HandlerIsIdempotent asserts, in code, that Handler is safe to run
	// more than once for the same records. It must be true; its zero is
	// refused, as queue.ConsumerConfig's is: under [CommitAfterHandler]
	// a batch is delivered at least once, and under [CommitByHandler]
	// the assertion is that the handler's effect and its commit are one
	// transaction, which the SDK cannot see either.
	HandlerIsIdempotent bool
	// Name names the stream in the fence's lock names. It is required
	// when Fence is set, and read for nothing else.
	Name string
	// Park, when set, sends a record its handler keeps refusing to a
	// queue and moves past it. Nil retries forever: the partition stays
	// blocked until the handler accepts.
	Park *ParkConfig
	// Partitions are the partitions this subscriber reads, one worker
	// each: static assignment, two processes splitting a group naming
	// disjoint sets (or both every partition, under Fence). Empty means
	// every partition of the stream; a partition outside it, or named
	// twice, is refused.
	Partitions []int
	// PollInterval bounds how long an idle worker waits before reading
	// again: the stream's Waker ends the wait at once on an Append in this
	// process, so it bounds only what another process appended. Zero, or
	// negative, means 100 ms.
	PollInterval time.Duration
	// Retry is the wait between two deliveries of a refused batch, its
	// attempt counting the consecutive refusals. A zero BaseDelay means
	// 100 ms doubling to 30 s with a fifth of jitter: a curve with no wait
	// would spin on a failing handler.
	Retry backoff.Value
	// SettleTimeout bounds each commit, dead letter and fence release
	// sent once ctx has ended, on Clock. Zero, or negative, means 5 s.
	SettleTimeout time.Duration
}

SubscriberConfig configures Subscribe: the group, its handler and the one promise about it the SDK cannot check, which partitions, how the offset moves, and what a failure, a stop and a second process do. Group, Handler and HandlerIsIdempotent are required; every other field's zero works.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL