Documentation
¶
Overview ¶
Package clickhouse provides a ClickHouse-backed implementation of payload.Sink, payload.Reader, and payload.Closer — the Langfuse-style "text bodies in ClickHouse" backend for the Logs view. Metadata + bodies live in one MergeTree table (bodies as ZSTD String columns); List projects only the metadata columns, Get fetches the body columns by request_id.
Durability mirrors the usage CH sink: a WAL-segment queue. Records append to an active segment; full segments rotate and flush to ClickHouse; a segment is deleted only after CH confirms the insert; leftover segments are drained on boot. Unlike the usage WAL, rotation also triggers on a byte threshold — payload bodies are MB-scale, so a line-count cap alone would let a segment grow to gigabytes.
Out of scope: schema migrations (IF NOT EXISTS), dedup of replayed segments (idempotency is the caller's concern), multi-node coordination (each pod runs its own WAL in its own directory).
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// DSN is the ClickHouse connection string (clickhouse://host:port/db).
DSN string
// RetentionDays controls the MergeTree TTL. Default 30 — payload bodies
// are bulky and short-lived debug/audit artifacts, so a shorter default
// than the usage sink's 90.
RetentionDays int
// WALDir is the directory for WAL segment files.
WALDir string
// MaxLines is the number of records per WAL segment before rotation.
// Default 2000 (lower than usage's 10k — records are large).
MaxLines int
// MaxBytes rotates the active segment once it exceeds this size,
// whichever comes first with MaxLines. Default 64 MiB.
MaxBytes int
// FlushInterval is how often the background goroutine rotates and
// flushes pending segments. Default 10s.
FlushInterval time.Duration
// MaxSegments caps how many pending segment files may accumulate on
// disk. When exceeded, the oldest are dropped and counted in Dropped().
// Default 256.
MaxSegments int
}
Config holds all tunables for the ClickHouse payload sink. It mirrors the usage CH sink's config; the WAL knobs are shared, plus a byte-based rotation threshold since payload bodies are MB-scale (a line-count cap alone would let a segment grow to gigabytes before rotating).
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is the read-only half: a ClickHouse connection serving List/Get with no WAL. The /payloads endpoints use this so the read path carries none of the write-side machinery (segment queue, recovery). Implements payload.Reader + payload.Closer.
type Sink ¶
type Sink struct {
*Reader
// contains filtered or unexported fields
}
Sink is the write half: Reader + a WAL-backed insert path. Implements payload.Sink (and, via the embedded Reader, payload.Reader) + payload.Closer.
func New ¶
New opens a ClickHouse connection, ensures the schema, constructs the WAL segment queue, and drains any segments left from a previous run.
Source Files