Documentation
¶
Overview ¶
Package segment is the scale tier: a durable hot append-log that seals into immutable, time-bounded, compressed columnar segments on a Blob backend (local disk now, S3/R2 next). Memory stays bounded no matter the total volume — a query streams one segment at a time and skips, unread, every segment whose time range can't match. This is the "billions of events, one binary, your data" path (see gtm/SCALE.md), and it satisfies the same store.Store the engine already computes over.
Index ¶
- type Store
- func (s *Store) Clear() error
- func (s *Store) Close() error
- func (s *Store) Count() int
- func (s *Store) DeleteUser(distinctID string) (int, error)
- func (s *Store) Flush() error
- func (s *Store) Ingest(evs ...event.Event) error
- func (s *Store) Names() ([]string, error)
- func (s *Store) Prune(before time.Time) (int, error)
- func (s *Store) Range(from, to time.Time) ([]event.Event, error)
- func (s *Store) Scan(from, to time.Time, fn func(event.Event) error) error
- func (s *Store) Scrub() (VerifyReport, int)
- func (s *Store) Verify() VerifyReport
- type VerifyReport
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
func Open ¶
Open recovers the hot WAL at hotPath and the cold manifest from the blob backend. sealAt is how many events accumulate in the hot tier before being sealed to a segment.
func (*Store) DeleteUser ¶ added in v0.3.0
DeleteUser erases every event for one distinct_id (GDPR erasure) across both tiers. Sealed segments are immutable, so each affected segment is rewritten: decode → filter → seal the kept events under a fresh key → swap the manifest entry → delete the old blob. The manifest is persisted once at the end with rollback, so a failure can orphan an unreferenced blob but never dangle a reference or half-delete a user.
func (*Store) Flush ¶
Flush seals whatever is currently in the hot block (used on graceful shutdown so the newest events are durably columnar, and by tests).
func (*Store) Prune ¶
Prune drops whole cold segments older than the cutoff (segment-granular retention) and prunes the hot block exactly. Returns the number of events removed.
func (*Store) Scan ¶
Scan streams every event in [from, to) through fn, reading only the segments whose time range overlaps and the hot block — bounded memory regardless of total volume.
func (*Store) Scrub ¶ added in v0.3.1
func (s *Store) Scrub() (VerifyReport, int)
Scrub is Verify plus housekeeping: deletes the orphaned blobs Verify found (they're unreferenced by design after a failed delete — safe to remove).
func (*Store) Verify ¶ added in v0.3.1
func (s *Store) Verify() VerifyReport
Verify checks the invariants the design doc promises: every manifest entry Gets from the blob backend, decodes (CRC), and matches its recorded count and time range; the hot WAL is readable; unreferenced blobs are reported as orphans.
type VerifyReport ¶ added in v0.3.1
type VerifyReport struct {
Segments int `json:"segments"`
Events int `json:"events"` // decoded total across segments
HotEvents int `json:"hot_events"` // events in the open hot block
Orphans []string `json:"orphans"` // blobs under seg/ not in the manifest (warning only)
Problems []string `json:"problems"` // corruption / mismatches — any entry means unhealthy
}
VerifyReport is what Verify/scrub found. Problems make the instance unhealthy; orphans are expected debris (failed deletes leave unreferenced blobs by design).
Source Files