Documentation
¶
Overview ¶
Package streamkit provides optional public helpers for native stream connectors. It supplies encoding, boundary, and protocol-heartbeat mechanics, never runtime admission or delivery guarantees.
Index ¶
- Constants
- Variables
- func CanonicalIdentity(e EventIdentity, r rowmodel.CodecResolver) ([]byte, error)
- func EncodeHeaders(headers []Header) ([]byte, error)
- func EventID(e EventIdentity, r rowmodel.CodecResolver) (string, error)
- func Retry(ctx context.Context, policy RetryPolicy, operation func(context.Context) error) error
- func WithEnvelopeFields(s rowmodel.Schema) (rowmodel.Schema, error)
- type BoundaryTracker
- func (b *BoundaryTracker) Add(records int, nbytes int64) error
- func (b *BoundaryTracker) Close()
- func (b *BoundaryTracker) Control(ctx context.Context, sink filament.StreamRecordSink, c rowmodel.Control) (bool, error)
- func (b *BoundaryTracker) Request() error
- func (b *BoundaryTracker) Requested() bool
- func (b *BoundaryTracker) Wake() <-chan time.Time
- type Envelope
- type EventIdentity
- type Header
- type Heartbeat
- type MessageBatches
- type MessageColumns
- func NewMessageColumns(base rowmodel.Schema, payload []byte) (*MessageColumns, rowmodel.Schema, error)
- func ProbeMessageBatch(base rowmodel.Schema, accepted *schema.MessageLayout, payloads [][]byte, ...) (*MessageColumns, rowmodel.Schema, error)
- func RestoreMessageColumns(base rowmodel.Schema, layout schema.MessageLayout) (*MessageColumns, rowmodel.Schema, error)
- type OpaqueCodec
- type Projector
- type Registry
- type RetryPolicy
- type Uint64Codec
Constants ¶
const HeaderEncodingVersion byte = 1
HeaderEncodingVersion identifies the ordered binary header encoding.
const IdentityEncodingVersion = 1
IdentityEncodingVersion identifies the canonical event identity encoding.
Variables ¶
var ErrBoundaryClosed = errors.New("streamkit: boundary closed")
ErrBoundaryClosed reports use after a boundary completed or its tracker closed.
var ErrHeartbeatRunning = errors.New("streamkit: heartbeat still running")
ErrHeartbeatRunning means Stop timed out before the protocol callback joined. Client cleanup or a canceled context alone is not proof of quiescence.
var ErrInvalidEnvelope = errors.New("streamkit: invalid envelope")
ErrInvalidEnvelope reports malformed or contradictory envelope data.
var ErrUnknownCodec = errors.New("streamkit: unknown codec")
ErrUnknownCodec reports an unregistered codec identifier or version.
Functions ¶
func CanonicalIdentity ¶
func CanonicalIdentity(e EventIdentity, r rowmodel.CodecResolver) ([]byte, error)
CanonicalIdentity encodes an identity with its codec-canonical position.
func EncodeHeaders ¶
EncodeHeaders v1: version byte, presence byte, u32 count; each entry is a u32-length key, null byte, u32-length value. Integers are big endian. Nil and empty lists differ; null values differ from non-null empty bytes.
func EventID ¶
func EventID(e EventIdentity, r rowmodel.CodecResolver) (string, error)
EventID returns the SHA-256 hex digest of the canonical source event identity.
Types ¶
type BoundaryTracker ¶
type BoundaryTracker struct {
// contains filtered or unexported fields
}
BoundaryTracker requests one Read's soft boundary. It belongs to the producer owner: Add, Requested, Control and Close must be serialized with row writing. Wake only wakes that owner; no timer goroutine touches a builder. The connector still supplies authoritative source controls and complete candidate coverage.
func NewBoundaryTracker ¶
func NewBoundaryTracker(limits filament.Boundary) (*BoundaryTracker, error)
NewBoundaryTracker starts idle and age clocks for one Read. Zero disables a limit; negative limits are invalid. Create a new tracker for the next Read.
func (*BoundaryTracker) Add ¶
func (b *BoundaryTracker) Add(records int, nbytes int64) error
Add accounts for completed source records and their connector-measured bytes. Call after successful row projection. A request is sticky; subsequent records may exceed a soft target to finish a transaction. Zero records do not reset idle.
func (*BoundaryTracker) Close ¶
func (b *BoundaryTracker) Close()
Close stops the timer. It neither emits a control nor completes coverage.
func (*BoundaryTracker) Control ¶
func (b *BoundaryTracker) Control(ctx context.Context, sink filament.StreamRecordSink, c rowmodel.Control) (bool, error)
Control forwards source evidence synchronously on the producer owner. The inlet must flush affected builders and wait for preceding Apply completion. True means a requested boundary completed at TxnEnd or ProgressBoundary with no tracked transaction open. It proves neither source-prefix completeness nor durability. Cancellation or any control error permanently fails this tracker.
func (*BoundaryTracker) Request ¶
func (b *BoundaryTracker) Request() error
Request asks the owner to finish at the next safe source boundary, for example after receiving a native seal command or reaching end-of-input. It does not flush or cancel anything and must run on the same owner as Add and Control.
func (*BoundaryTracker) Requested ¶
func (b *BoundaryTracker) Requested() bool
Requested reports whether any soft limit has been reached. This is a request to finish at a safe source boundary, never permission to truncate a transaction.
func (*BoundaryTracker) Wake ¶
func (b *BoundaryTracker) Wake() <-chan time.Time
Wake returns the timer channel to select alongside native input. After a wake, inspect Requested and seek a safe source boundary, even if no new row arrives. It is nil when time limits are disabled. Count limits are checked after Add.
type Envelope ¶
type Envelope struct {
Identity EventIdentity
Timestamp *time.Time
Headers []Header
Key []byte
KeyNull bool
Payload []byte
PayloadNull bool
}
Envelope combines source identity with message content and lossless delivery metadata.
type EventIdentity ¶
type EventIdentity = rowmodel.EventIdentity
EventIdentity shares the neutral value type; encoding belongs to this SDK.
func DecodeIdentity ¶
func DecodeIdentity(data []byte, r rowmodel.CodecResolver) (EventIdentity, error)
DecodeIdentity accepts only the canonical versioned SDK representation. It rejects unknown versions, duplicate/unknown fields and ambiguous spellings.
type Header ¶
Header preserves provider order, duplicate names and arbitrary binary values. Null is authoritative; a null header with nonempty bytes is invalid.
func DecodeHeaders ¶
DecodeHeaders validates and decodes headers into independently owned values.
type Heartbeat ¶
type Heartbeat struct {
// contains filtered or unexported fields
}
Heartbeat runs one serialized protocol task independently of row production. Its callback must support cancellation and be safe alongside native reads. It must not touch builders, certify coverage, or acknowledge tentative progress. A callback failure cancels Context, which the owner must use for blocked reads and writes. This helper joins only its callback, not the session or destination.
func StartHeartbeat ¶
func StartHeartbeat(ctx context.Context, interval time.Duration, task func(context.Context) error) (*Heartbeat, error)
StartHeartbeat starts periodic work after the first interval. Use Retry explicitly inside the callback if the protocol permits retries. Stop and join heartbeat before releasing any client resources its callback uses.
func (*Heartbeat) Context ¶
Context is canceled by parent cancellation, Stop, or a protocol task failure. context.Cause preserves the terminal protocol failure for the owner.
func (*Heartbeat) Done ¶
func (h *Heartbeat) Done() <-chan struct{}
Done closes only after the heartbeat goroutine and callback have exited.
type MessageBatches ¶ added in v0.23.0
type MessageBatches struct {
// contains filtered or unexported fields
}
MessageBatches owns accepted layouts for one source session. Share it across physical readers of the same logical resources. Only the serial Read owner may call Prepare. A failed epoch ends the session and discards tentative state.
func NewMessageBatches ¶ added in v0.23.0
func NewMessageBatches(opts filament.StreamOpenOpts, base func(string) rowmodel.Schema) (*MessageBatches, error)
NewMessageBatches validates and restores layouts. A nil result means disabled. base supplies transport columns, without payload or Filament metadata fields.
func (*MessageBatches) Prepare ¶ added in v0.23.0
func (b *MessageBatches) Prepare(ctx context.Context, out filament.StreamRecordSink, base rowmodel.Schema, payloads [][]byte) (*MessageColumns, filament.RowWriter, error)
Prepare validates every payload, restores the decoder and negotiates a writer before any row is emitted. Transport fields and EndRow remain source-owned. Successful admission is tentative until the runner certifies the epoch.
type MessageColumns ¶
type MessageColumns struct {
// contains filtered or unexported fields
}
MessageColumns projects JSON object members into source-owned columns. Other messages retain their bytes in a source-owned payload column. A resource keeps the first message's schema; drift fails before any row values are appended.
func NewMessageColumns ¶
func NewMessageColumns(base rowmodel.Schema, payload []byte) (*MessageColumns, rowmodel.Schema, error)
NewMessageColumns infers a deterministic schema and rejects collisions with transport columns and Filament's reserved namespace.
func ProbeMessageBatch ¶ added in v0.23.0
func ProbeMessageBatch(base rowmodel.Schema, accepted *schema.MessageLayout, payloads [][]byte, policy schema.Policy) (*MessageColumns, rowmodel.Schema, error)
ProbeMessageBatch inspects every buffered message before any row is encoded. It keeps absent fields and adds observed fields in deterministic order. An initial batch establishes a layout; later batches obey the configured policy.
func RestoreMessageColumns ¶ added in v0.23.0
func RestoreMessageColumns(base rowmodel.Schema, layout schema.MessageLayout) (*MessageColumns, rowmodel.Schema, error)
RestoreMessageColumns restores the accepted payload layout without sampling.
func (*MessageColumns) Layout ¶ added in v0.23.0
func (c *MessageColumns) Layout() schema.MessageLayout
Layout returns the decoder state persisted with an epoch's progress.
func (*MessageColumns) Prepare ¶
func (c *MessageColumns) Prepare(payload []byte) (func(arrowbatch.RowWriter), error)
Prepare validates the entire message before the caller appends transport fields. The returned closure writes only source-owned message columns.
type OpaqueCodec ¶
type OpaqueCodec struct{}
OpaqueCodec permits equality only; differing opaque cursors are incomparable. Nil and empty payloads remain distinct. It does not interpret native cursors.
func (OpaqueCodec) Canonicalize ¶
Canonicalize validates and copies the value, preserving nil versus empty bytes.
func (OpaqueCodec) Compare ¶
func (c OpaqueCodec) Compare(a, b rowmodel.Position) (rowmodel.PositionOrder, error)
Compare permits exact equality only; different opaque values remain incomparable.
type Projector ¶
type Projector struct {
// contains filtered or unexported fields
}
Projector wraps a schema-ordered writer opened with WithEnvelopeFields. Source fields are appended first; EndEvent projects every envelope before batching and before any outer audit writer appends legacy lineage. No last-row Meta is used to reconstruct earlier envelopes. A projector is owned by one producer loop.
func NewEventMetadataProjector ¶
func NewEventMetadataProjector(w arrowbatch.RowWriter, r rowmodel.CodecResolver) *Projector
NewEventMetadataProjector writes the suffix declared by WithEventMetadataFields.
func NewProjector ¶
func NewProjector(w arrowbatch.RowWriter, r rowmodel.CodecResolver) *Projector
NewProjector binds an envelope writer to its position codecs. The writer must have been opened with WithEnvelopeFields.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is safe for concurrent lookup/registration. Codecs must be pure, deterministic and concurrency-safe. Registrations cannot replace live codecs.
type RetryPolicy ¶
RetryPolicy bounds explicitly retryable protocol operations. MaxAttempts includes the first call; zero means one call. Delay is fixed between attempts. Retryable must be supplied to retry an error. The caller must establish that replaying the operation is safe; do not wrap a partially emitted Read or Apply.
type Uint64Codec ¶
type Uint64Codec struct{}
Uint64Codec encodes a connector-declared unsigned counter as decimal ASCII. Leading zeros normalize away. It must not be used for LSNs, GTID sets or opaque cursors merely because a sample happens to contain digits.
func (Uint64Codec) Canonicalize ¶
Canonicalize returns independently owned decimal bytes without leading zeros.
func (Uint64Codec) Compare ¶
func (c Uint64Codec) Compare(a, b rowmodel.Position) (rowmodel.PositionOrder, error)
Compare orders validated unsigned counters with matching codec identities.