Help us shape the future of Go! Take the 2026 Go Developer Survey

streamkit

package
v0.32.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package streamkit provides optional public helpers for native stream connectors. It supplies encoding, boundary, and protocol-heartbeat mechanics, never runtime admission or delivery guarantees.

Index

Constants

View Source
const HeaderEncodingVersion byte = 1

HeaderEncodingVersion identifies the ordered binary header encoding.

View Source
const IdentityEncodingVersion = 1

IdentityEncodingVersion identifies the canonical event identity encoding.

Variables

View Source
var ErrBoundaryClosed = errors.New("streamkit: boundary closed")

ErrBoundaryClosed reports use after a boundary completed or its tracker closed.

View Source
var ErrHeartbeatRunning = errors.New("streamkit: heartbeat still running")

ErrHeartbeatRunning means Stop timed out before the protocol callback joined. Client cleanup or a canceled context alone is not proof of quiescence.

View Source
var ErrInvalidEnvelope = errors.New("streamkit: invalid envelope")

ErrInvalidEnvelope reports malformed or contradictory envelope data.

View Source
var ErrUnknownCodec = errors.New("streamkit: unknown codec")

ErrUnknownCodec reports an unregistered codec identifier or version.

Functions

func CanonicalIdentity

func CanonicalIdentity(e EventIdentity, r rowmodel.CodecResolver) ([]byte, error)

CanonicalIdentity encodes an identity with its codec-canonical position.

func EncodeHeaders

func EncodeHeaders(headers []Header) ([]byte, error)

EncodeHeaders v1: version byte, presence byte, u32 count; each entry is a u32-length key, null byte, u32-length value. Integers are big endian. Nil and empty lists differ; null values differ from non-null empty bytes.

func EventID

EventID returns the SHA-256 hex digest of the canonical source event identity.

func Retry

func Retry(ctx context.Context, policy RetryPolicy, operation func(context.Context) error) error

Retry runs an operation with bounded attempts and cancellation-aware waits. No attempt starts after cancellation is observed, and cancellation is never converted into success, even when the operation returns nil.

func WithEnvelopeFields

func WithEnvelopeFields(s rowmodel.Schema) (rowmodel.Schema, error)

WithEnvelopeFields appends the SDK schema suffix after rejecting reserved-name collisions.

Types

type BoundaryTracker

type BoundaryTracker struct {
	// contains filtered or unexported fields
}

BoundaryTracker requests one Read's soft boundary. It belongs to the producer owner: Add, Requested, Control and Close must be serialized with row writing. Wake only wakes that owner; no timer goroutine touches a builder. The connector still supplies authoritative source controls and complete candidate coverage.

func NewBoundaryTracker

func NewBoundaryTracker(limits filament.Boundary) (*BoundaryTracker, error)

NewBoundaryTracker starts idle and age clocks for one Read. Zero disables a limit; negative limits are invalid. Create a new tracker for the next Read.

func (*BoundaryTracker) Add

func (b *BoundaryTracker) Add(records int, nbytes int64) error

Add accounts for completed source records and their connector-measured bytes. Call after successful row projection. A request is sticky; subsequent records may exceed a soft target to finish a transaction. Zero records do not reset idle.

func (*BoundaryTracker) Close

func (b *BoundaryTracker) Close()

Close stops the timer. It neither emits a control nor completes coverage.

func (*BoundaryTracker) Control

Control forwards source evidence synchronously on the producer owner. The inlet must flush affected builders and wait for preceding Apply completion. True means a requested boundary completed at TxnEnd or ProgressBoundary with no tracked transaction open. It proves neither source-prefix completeness nor durability. Cancellation or any control error permanently fails this tracker.

func (*BoundaryTracker) Request

func (b *BoundaryTracker) Request() error

Request asks the owner to finish at the next safe source boundary, for example after receiving a native seal command or reaching end-of-input. It does not flush or cancel anything and must run on the same owner as Add and Control.

func (*BoundaryTracker) Requested

func (b *BoundaryTracker) Requested() bool

Requested reports whether any soft limit has been reached. This is a request to finish at a safe source boundary, never permission to truncate a transaction.

func (*BoundaryTracker) Wake

func (b *BoundaryTracker) Wake() <-chan time.Time

Wake returns the timer channel to select alongside native input. After a wake, inspect Requested and seek a safe source boundary, even if no new row arrives. It is nil when time limits are disabled. Count limits are checked after Add.

type Envelope

type Envelope struct {
	Identity    EventIdentity
	Timestamp   *time.Time
	Headers     []Header
	Key         []byte
	KeyNull     bool
	Payload     []byte
	PayloadNull bool
}

Envelope combines source identity with message content and lossless delivery metadata.

type EventIdentity

type EventIdentity = rowmodel.EventIdentity

EventIdentity shares the neutral value type; encoding belongs to this SDK.

func DecodeIdentity

func DecodeIdentity(data []byte, r rowmodel.CodecResolver) (EventIdentity, error)

DecodeIdentity accepts only the canonical versioned SDK representation. It rejects unknown versions, duplicate/unknown fields and ambiguous spellings.

type Header struct {
	Key   string
	Value []byte
	Null  bool
}

Header preserves provider order, duplicate names and arbitrary binary values. Null is authoritative; a null header with nonempty bytes is invalid.

func DecodeHeaders

func DecodeHeaders(data []byte) ([]Header, error)

DecodeHeaders validates and decodes headers into independently owned values.

type Heartbeat

type Heartbeat struct {
	// contains filtered or unexported fields
}

Heartbeat runs one serialized protocol task independently of row production. Its callback must support cancellation and be safe alongside native reads. It must not touch builders, certify coverage, or acknowledge tentative progress. A callback failure cancels Context, which the owner must use for blocked reads and writes. This helper joins only its callback, not the session or destination.

func StartHeartbeat

func StartHeartbeat(ctx context.Context, interval time.Duration, task func(context.Context) error) (*Heartbeat, error)

StartHeartbeat starts periodic work after the first interval. Use Retry explicitly inside the callback if the protocol permits retries. Stop and join heartbeat before releasing any client resources its callback uses.

func (*Heartbeat) Context

func (h *Heartbeat) Context() context.Context

Context is canceled by parent cancellation, Stop, or a protocol task failure. context.Cause preserves the terminal protocol failure for the owner.

func (*Heartbeat) Done

func (h *Heartbeat) Done() <-chan struct{}

Done closes only after the heartbeat goroutine and callback have exited.

func (*Heartbeat) Err

func (h *Heartbeat) Err() error

Err returns a terminal callback error, if one has been observed.

func (*Heartbeat) Stop

func (h *Heartbeat) Stop(ctx context.Context) error

Stop cancels heartbeat and waits for its callback. It is safe to call again after a timeout to join a callback that later exits. A timeout returns ErrHeartbeatRunning and the wait context's error, never a clean outcome.

type MessageBatches added in v0.23.0

type MessageBatches struct {
	// contains filtered or unexported fields
}

MessageBatches owns accepted layouts for one source session. Share it across physical readers of the same logical resources. Only the serial Read owner may call Prepare. A failed epoch ends the session and discards tentative state.

func NewMessageBatches added in v0.23.0

func NewMessageBatches(opts filament.StreamOpenOpts, base func(string) rowmodel.Schema) (*MessageBatches, error)

NewMessageBatches validates and restores layouts. A nil result means disabled. base supplies transport columns, without payload or Filament metadata fields.

func (*MessageBatches) Prepare added in v0.23.0

Prepare validates every payload, restores the decoder and negotiates a writer before any row is emitted. Transport fields and EndRow remain source-owned. Successful admission is tentative until the runner certifies the epoch.

type MessageColumns

type MessageColumns struct {
	// contains filtered or unexported fields
}

MessageColumns projects JSON object members into source-owned columns. Other messages retain their bytes in a source-owned payload column. A resource keeps the first message's schema; drift fails before any row values are appended.

func NewMessageColumns

func NewMessageColumns(base rowmodel.Schema, payload []byte) (*MessageColumns, rowmodel.Schema, error)

NewMessageColumns infers a deterministic schema and rejects collisions with transport columns and Filament's reserved namespace.

func ProbeMessageBatch added in v0.23.0

func ProbeMessageBatch(base rowmodel.Schema, accepted *schema.MessageLayout, payloads [][]byte, policy schema.Policy) (*MessageColumns, rowmodel.Schema, error)

ProbeMessageBatch inspects every buffered message before any row is encoded. It keeps absent fields and adds observed fields in deterministic order. An initial batch establishes a layout; later batches obey the configured policy.

func RestoreMessageColumns added in v0.23.0

func RestoreMessageColumns(base rowmodel.Schema, layout schema.MessageLayout) (*MessageColumns, rowmodel.Schema, error)

RestoreMessageColumns restores the accepted payload layout without sampling.

func (*MessageColumns) Layout added in v0.23.0

func (c *MessageColumns) Layout() schema.MessageLayout

Layout returns the decoder state persisted with an epoch's progress.

func (*MessageColumns) Prepare

func (c *MessageColumns) Prepare(payload []byte) (func(arrowbatch.RowWriter), error)

Prepare validates the entire message before the caller appends transport fields. The returned closure writes only source-owned message columns.

type OpaqueCodec

type OpaqueCodec struct{}

OpaqueCodec permits equality only; differing opaque cursors are incomparable. Nil and empty payloads remain distinct. It does not interpret native cursors.

func (OpaqueCodec) Canonicalize

func (c OpaqueCodec) Canonicalize(p rowmodel.Position) (rowmodel.Position, error)

Canonicalize validates and copies the value, preserving nil versus empty bytes.

func (OpaqueCodec) Compare

Compare permits exact equality only; different opaque values remain incomparable.

func (OpaqueCodec) Validate

func (OpaqueCodec) Validate(p rowmodel.Position) error

Validate checks the position format identity without interpreting opaque bytes.

type Projector

type Projector struct {
	// contains filtered or unexported fields
}

Projector wraps a schema-ordered writer opened with WithEnvelopeFields. Source fields are appended first; EndEvent projects every envelope before batching and before any outer audit writer appends legacy lineage. No last-row Meta is used to reconstruct earlier envelopes. A projector is owned by one producer loop.

func NewEventMetadataProjector

func NewEventMetadataProjector(w arrowbatch.RowWriter, r rowmodel.CodecResolver) *Projector

NewEventMetadataProjector writes the suffix declared by WithEventMetadataFields.

func NewProjector

NewProjector binds an envelope writer to its position codecs. The writer must have been opened with WithEnvelopeFields.

func (*Projector) EndEvent

func (w *Projector) EndEvent(e Envelope, meta rowmodel.Meta) error

EndEvent projects envelope columns and derives stream metadata before completing the row.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry is safe for concurrent lookup/registration. Codecs must be pure, deterministic and concurrency-safe. Registrations cannot replace live codecs.

func (*Registry) Lookup

func (r *Registry) Lookup(name string, version int) (rowmodel.PositionCodec, error)

Lookup resolves a registered codec or returns ErrUnknownCodec.

func (*Registry) Register

func (r *Registry) Register(name string, version int, c rowmodel.PositionCodec) error

Register adds a codec without allowing replacement of an existing registration.

type RetryPolicy

type RetryPolicy struct {
	MaxAttempts int
	Delay       time.Duration
	Retryable   func(error) bool
}

RetryPolicy bounds explicitly retryable protocol operations. MaxAttempts includes the first call; zero means one call. Delay is fixed between attempts. Retryable must be supplied to retry an error. The caller must establish that replaying the operation is safe; do not wrap a partially emitted Read or Apply.

type Uint64Codec

type Uint64Codec struct{}

Uint64Codec encodes a connector-declared unsigned counter as decimal ASCII. Leading zeros normalize away. It must not be used for LSNs, GTID sets or opaque cursors merely because a sample happens to contain digits.

func (Uint64Codec) Canonicalize

func (c Uint64Codec) Canonicalize(p rowmodel.Position) (rowmodel.Position, error)

Canonicalize returns independently owned decimal bytes without leading zeros.

func (Uint64Codec) Compare

Compare orders validated unsigned counters with matching codec identities.

func (Uint64Codec) Validate

func (Uint64Codec) Validate(p rowmodel.Position) error

Validate requires decimal digits representing a value within the uint64 range.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL