Documentation
¶
Overview ¶
Package whtable owns what a webhook source is in the query engine (#1870): two tables and the view readers name.
webhook_{source}_raw is a JSON table over the raw segments, partitioned by dt, hour and minute in the Hive layout the receiver writes: one partition per compaction window, minute being the minute the window starts at. webhook_{source}_compacted is a Parquet table whose partitions are each registered at the directory of the managed resource holding that window. webhook_{source} is the view over both: a window whose compacted partition is registered is read from there, and every other window from the raw segments. Registering a window's partition is the one metastore write that moves it from one side to the other, so no window is missing from the view or served twice.
Every statement runs through the Trino connection's Exec, the platform's one write path into Trino. Registering a partition at an explicit location needs the catalog to set hive.allow-register-partition-procedure=true; see docs/server/scratch-catalog.md.
Index ¶
- Variables
- func ViewStatement(tg Target, src whsource.Source) string
- type Executor
- type Tables
- func (t *Tables) Create(ctx context.Context, tg Target, src whsource.Source) error
- func (t *Tables) Drop(ctx context.Context, tg Target, src whsource.Source) error
- func (t *Tables) Probe(ctx context.Context, tg Target, src whsource.Source) error
- func (t *Tables) RegisterRawWindow(ctx context.Context, tg Target, src whsource.Source, start time.Time) error
- func (t *Tables) RegisterWindow(ctx context.Context, tg Target, src whsource.Source, start time.Time, ...) error
- func (t *Tables) S3Location(prefix string) string
- func (t *Tables) SyncRaw(ctx context.Context, tg Target, src whsource.Source) error
- func (t *Tables) TargetFor(connection string) (Target, error)
- func (t *Tables) UnregisterWindow(ctx context.Context, tg Target, src whsource.Source, start time.Time) error
- type Target
Constants ¶
This section is empty.
Variables ¶
var ( ErrNoScratchTarget = errors.New("this connection has no scratch catalog and schema configured, so a webhook source's table cannot be created on it") ErrReadOnly = errors.New("this connection is read-only, so a webhook source's table cannot be created on it") // ErrRegisterDisabled is the catalog refusing register_partition. Each // compacted window is registered at its own resource's directory, which is // what the procedure is for, so a source cannot run without it. ErrRegisterDisabled = errors.New("the scratch catalog does not allow register_partition; set hive.allow-register-partition-procedure=true on it (see docs/server/scratch-catalog.md)") )
Refusals a source's connection can meet.
var PartitionColumns = []string{"dt", "hour", "minute"}
PartitionColumns are the partition columns of both tables and the view, in order: the UTC date, hour and minute a window starts at.
Functions ¶
Types ¶
type Executor ¶
type Executor interface {
Exec(ctx context.Context, connection, sql string) error
ScratchTarget(connection string) (trino.ScratchConfig, bool)
AcceptsWrites(connection string) bool
}
Executor runs a statement on a Trino connection and reports its scratch target. The Trino toolkit satisfies it.
type Tables ¶
type Tables struct {
// contains filtered or unexported fields
}
Tables creates and maintains sources' tables.
func New ¶
New builds the table manager. bucket is the managed-resources bucket both the raw segments and the compacted windows are written to.
func (*Tables) Create ¶
Create makes the source's schema, both tables and the view. Every statement is idempotent, so creating a source whose tables survived an earlier attempt finishes the job.
func (*Tables) Drop ¶
Drop removes the view and both tables. The objects under them are the compactor's to delete; a dropped external table leaves its files.
func (*Tables) Probe ¶
Probe proves a connection can hold a source: the catalog reads the managed resources bucket, and allows register_partition. It runs against the source's own tables, which Create has made, by registering and removing a window that holds nothing.
func (*Tables) RegisterRawWindow ¶
func (t *Tables) RegisterRawWindow(ctx context.Context, tg Target, src whsource.Source, start time.Time) error
RegisterRawWindow makes a new window's raw segments readable at once, rather than at the next sync. The receiver calls it when it writes the first segment of a window, which is what lets an event be queried as soon as it is acknowledged. A window already registered is left as it is.
func (*Tables) RegisterWindow ¶
func (t *Tables) RegisterWindow(ctx context.Context, tg Target, src whsource.Source, start time.Time, location string) error
RegisterWindow points a window's compacted partition at location, replacing where it pointed before. It is what moves the window from the raw side of the view to the compacted side.
func (*Tables) S3Location ¶
S3Location renders a key prefix in the bucket as the URI a table or partition location names.
func (*Tables) SyncRaw ¶
SyncRaw makes the raw table's partitions match the window directories that exist: a window a segment was just written into becomes readable, and a window whose segments retention deleted stops being listed.
func (*Tables) TargetFor ¶
TargetFor resolves the connection a source names to its scratch catalog and schema, refusing one that has none or will not run write statements.
func (*Tables) UnregisterWindow ¶
func (t *Tables) UnregisterWindow(ctx context.Context, tg Target, src whsource.Source, start time.Time) error
UnregisterWindow removes a window's compacted partition, which puts the window back on the raw side of the view. A window with no partition is already unregistered.
Source Files