changefeed

package module
v0.21.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 31, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source
const (
	PathSubscribe = "/changefeed/v1/subscribe"
	PathAck       = "/changefeed/v1/ack"
)

HTTP routes (versioned).

Variables

This section is empty.

Functions

func DecodeEvents

func DecodeEvents(r io.Reader, yield func(Event) bool) error

DecodeEvents streams Events from NDJSON r, calling yield for each. It stops on the first yield returning false, on EOF (nil), or on a read/parse error. Blank lines are skipped.

func Handler

func Handler(cat *db.Catalog, opts ServerOptions) http.Handler

Handler serves the change feed: SSE GET PathSubscribe and POST PathAck. It reads from cat's tables/archives via db.Watch; it never writes. Mount it on any HTTP listener (token-gate it in front). Transport-neutral: no CEDAR.

func Pull

func Pull(ctx context.Context, cfg PullConfig, sink replicate.Sink) error

Pull subscribes to the source feed over SSE and applies each change to sink, resuming from sink.Cursor() on every (re)connect and periodically ACKing the max record timestamp it has durably persisted (so the source can GC). It reconnects with backoff until ctx is cancelled; it returns nil on cancellation.

func ToChange

func ToChange(e Event) (replicate.Change, error)

ToChange parses a wire Event back into an in-memory Change (unmarshaling the ad JSON).

func WriteEvent

func WriteEvent(w io.Writer, e Event) error

WriteEvent encodes one Event as an NDJSON line (a trailing newline, no embedded newlines since json.Marshal escapes them). Suitable as one SSE data payload or one line of a chunked stream.

Types

type Authorizer

type Authorizer func(r *http.Request) (src, subscriber string, ok bool)

Authorizer authenticates a request and returns the source label and subscriber id it is allowed to act as. ok=false rejects (401). A nil Authorizer allows all, taking src/subscriber from query.

type Event

type Event struct {
	Kind   replicate.Kind  `json:"kind"`
	Src    string          `json:"src,omitempty"`
	Ver    uint64          `json:"ver,omitempty"`
	Key    string          `json:"key,omitempty"`
	Ad     json.RawMessage `json:"ad,omitempty"`
	Cursor []byte          `json:"cursor,omitempty"`
	TS     int64           `json:"ts,omitempty"`

	FromMillis int64 `json:"fromMillis,omitempty"`
	ToMillis   int64 `json:"toMillis,omitempty"`
}

Event is the NDJSON wire form of a replicate.Change (one JSON object per line). It is what an external, possibly non-Go, sink parses. Ad is a JSON object present only on an upsert; Cursor marshals as base64 (encoding/json's default for []byte).

func ToEvent

func ToEvent(c replicate.Change) (Event, error)

ToEvent renders a Change as its wire Event (marshaling the ad to JSON).

type MemRegistry

type MemRegistry struct {
	LeaseTTL time.Duration
	// contains filtered or unexported fields
}

MemRegistry is an in-memory Registry: fine for a single-node source. LeaseTTL bounds how long a silent subscriber holds the floor (default defaultLeaseTTL). Safe for concurrent use.

func (*MemRegistry) Ack

func (r *MemRegistry) Ack(table, sub string, ackMillis int64, now time.Time)

func (*MemRegistry) Evict

func (r *MemRegistry) Evict(now time.Time) int

func (*MemRegistry) Floor

func (r *MemRegistry) Floor(table string, now time.Time) (int64, bool)

func (*MemRegistry) Renew

func (r *MemRegistry) Renew(table, sub string, now time.Time)

func (*MemRegistry) Subscribers

func (r *MemRegistry) Subscribers(table string, now time.Time) []SubStatus

type PullConfig

type PullConfig struct {
	BaseURL    string   // source root, e.g. "https://source:9200"
	Table      string   // source table/archive
	Subscriber string   // stable durable-subscription id (identifies this sink to the source)
	Src        string   // label recorded with events (defaults to Subscriber); passed as ?src=
	Constraint string   // optional server-side filter
	Project    []string // optional attribute projection

	Token      string        // bearer token
	AckEvery   time.Duration // ack + commit cadence; default 5s
	Backoff    time.Duration // reconnect backoff cap; default 30s
	HTTPClient *http.Client
}

PullConfig configures a subscription to one source feed.

type Registry

type Registry interface {
	// Ack records that subscriber has durably persisted every record up to ackMillis on table.
	Ack(table, subscriber string, ackMillis int64, now time.Time)
	// Renew marks a subscriber live (an active subscribe stream) without moving its ack.
	Renew(table, subscriber string, now time.Time)
	// Floor returns the min ack over live subscribers of table and their count. With no live
	// subscribers, held is false (no retention hold from the feed).
	Floor(table string, now time.Time) (ackMillis int64, held bool)
	// Evict drops subscribers whose lease expired as of now; returns how many were dropped.
	Evict(now time.Time) int
	// Subscribers lists live subscribers of a table (for observability).
	Subscribers(table string, now time.Time) []SubStatus
}

Registry persists per-subscriber ACK watermarks and leases and computes the GC floor. The ack watermark is a COMPARABLE record timestamp (unix millis, source-stamped as Event.TS), so the floor -- the min ack over live subscribers -- maps directly onto an archive's time-based retention: the source may reclaim records older than the floor. A subscriber whose lease expires (silent past LeaseTTL) is evicted so one dead sink cannot pin retention forever; on return it resumes and, if its cursor was reclaimed, receives a reset+gap.

type ServerOptions

type ServerOptions struct {
	Auth      Authorizer
	Registry  Registry      // records acks + leases, computes the GC floor; MemRegistry if nil
	Heartbeat time.Duration // SSE keep-alive comment cadence; default 15s
	AgeAttr   string        // the record attribute (unix seconds) stamped as Event.TS for GC; "" => no TS
}

ServerOptions configure the feed Handler.

type SubStatus

type SubStatus struct {
	Subscriber string
	AckMillis  int64
	LastSeen   time.Time
}

SubStatus is one subscriber's observable state.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL