config

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source
const (
	RedisStandalone = "standalone"
	RedisCluster    = "cluster"
	RedisSentinel   = "sentinel"
)

Redis deployment modes for cache.redis.mode. RedisSentinel is refused until the backend supports it (#656).

View Source
const (
	EnvConfig   = "WH_CONFIG"
	EnvLogLevel = "WH_LOG_LEVEL"
)

EnvConfig names the boot-config file (default: ./config.yaml) and EnvLogLevel the minimum log level. Neither is a Config field — the first locates the file the struct is read from, the second is read by main directly — so they are declared here as the process-level names unboundEnv must not flag.

View Source
const EnvSettingsDir = "WH_SETTINGS_DIR"

EnvSettingsDir is the environment variable naming the settings directory — the same variable the Settings.Dir struct tag binds. Subcommands (`wavehouse validate`) read it directly without loading config, so the name is exported here as the single authority; a struct tag must be a literal, so a config test pins the tag to this constant to prevent drift.

View Source
const MaxNATSShards = 256

MaxNATSShards is the largest mq.nats.shards boot accepts: internal/mq's own bound, which config cannot import (internal/app pins the two together).

Variables

This section is empty.

Functions

func CheckDataDir added in v0.2.0

func CheckDataDir(dir string) error

CheckDataDir reports whether boot can use dir as data_dir: a directory that exists must be writable, and one that doesn't (boot creates it — a first run, or a missing mount that WarnIfFreshDataDir calls out) must have a writable nearest existing ancestor so that creation can succeed. Run before anything dials out, so a data_dir the process cannot write to refuses boot before ClickHouse discovery rather than after it; a permission denial — on the write probe, or on reaching dir at all through a parent without search permission — carries the UID-65532 hint, since a bind mount owned by root is the typical cause. Writability is probed by creating and removing one temp file: the only portable test that exercises the mount's ownership and mode. A blank dir — reachable through `WH_DATA_DIR=` or `data_dir: ""` — is refused outright: the ancestor walk would otherwise probe the working directory and pass, and NATS and Pebble state would land under it.

func LogStorageInitError

func LogStorageInitError(kind, path string, err error)

LogStorageInitError emits an error log for a storage-init failure, with a UID-65532 hint when the failure looks like a permission denial — the dominant signature of a Docker bind mount whose host directory is owned by root rather than the distroless `nonroot` user. Keeps the actionable remediation right next to the error line so operators don't have to dig.

`kind` is a short label (e.g. "mq", "dedupe"). The caller still decides whether to exit; this only logs.

func WarnIfFreshDataDir

func WarnIfFreshDataDir(kind, dir string)

WarnIfFreshDataDir logs a startup `WARN` if dir doesn't already exist or is empty, signalling that we're about to start with no prior state.

On a first-ever run this is expected; on every subsequent run it should be silent. So when the warning *does* fire on a redeploy, it's the most direct possible signal that the persistent volume isn't actually persisting: either it wasn't mounted, the wrong path was mounted, or the volume was recreated.

`kind` is a short label for the log message (e.g. "nats", "pebble").

Types

type Auth

type Auth struct {
	OperatorKey string `yaml:"operator_key" env:"WH_AUTH_OPERATOR_KEY"`
}

Auth holds the platform's own credential. The verifier wiring — `jwks_url`, `jwt_secret`, `role_claim` — is the settings directory's `auth` block (hot-reloadable: a change rebuilds the tenant's verifier; the HMAC secret is a tenant's own, #529). There is no on/off switch: the middleware always runs. A request with no token, or an invalid/expired one, falls back to the policy default_role; elevated access requires a valid token whose role claim matches a granted role (or the policy admin_role). With neither jwt_secret nor jwks_url set, no token can validate, so every request is the default role — a pure public deployment.

OperatorKey is an optional non-JWT credential for the operator running the deployment: a request presenting it via an "Authorization: Operator <key>" header (or the X-Operator-Key alias) is authorized as a full-access platform operator, independent of the JWT verifier, and keeps working even if the policy is missing/deleted (break-glass recovery, e.g. fixing policies.json). Empty (the default) disables it. Treat it as an admin secret.

type Cache

type Cache struct {
	Backend   CacheBackend     `yaml:"backend" env:"WH_CACHE_BACKEND"`
	L1MaxCost int64            `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST"`
	Redis     CacheRedisConfig `yaml:"redis"`
}

Cache selects and sizes the query-result cache. The time-range bucket structured queries normalize to is a settings-directory key (query.timestamp_bucket_seconds) — query shaping, not process memory.

type CacheBackend added in v0.2.0

type CacheBackend string

CacheBackend names the query-result cache implementation.

const (
	// CacheLocal is the in-process Ristretto cache, sized by
	// cache.l1_max_cost.
	CacheLocal CacheBackend = "local"
	// CacheRedis is one Redis-compatible server shared by every process,
	// configured by cache.redis.
	CacheRedis CacheBackend = "redis"
)

type CacheRedisConfig added in v0.2.0

type CacheRedisConfig struct {
	// Addrs are host:port pairs: the server, or seeds for a cluster.
	Addrs    []string      `yaml:"addrs" env:"WH_CACHE_REDIS_ADDRS"`
	Mode     string        `yaml:"mode" env:"WH_CACHE_REDIS_MODE"`
	Username string        `yaml:"username" env:"WH_CACHE_REDIS_USERNAME"`
	Password string        `yaml:"password" env:"WH_CACHE_REDIS_PASSWORD"`
	DB       int           `yaml:"db" env:"WH_CACHE_REDIS_DB"`
	TLS      CacheRedisTLS `yaml:"tls"`
	// KeyPrefix leads every key, so deployments can share one server.
	KeyPrefix   string        `yaml:"key_prefix" env:"WH_CACHE_REDIS_KEY_PREFIX"`
	Timeout     time.Duration `yaml:"timeout" env:"WH_CACHE_REDIS_TIMEOUT"`
	DialTimeout time.Duration `yaml:"dial_timeout" env:"WH_CACHE_REDIS_DIAL_TIMEOUT"`
	// MaxValueBytes is the largest value stored, after compression.
	MaxValueBytes int `yaml:"max_value_bytes" env:"WH_CACHE_REDIS_MAX_VALUE_BYTES"`
	// CompressMinBytes is the smallest value zstd-compressed; 0 never
	// compresses.
	CompressMinBytes int `yaml:"compress_min_bytes" env:"WH_CACHE_REDIS_COMPRESS_MIN_BYTES"`
	// VersionTTL is how long a version token outlives its last bump.
	VersionTTL time.Duration `yaml:"version_ttl" env:"WH_CACHE_REDIS_VERSION_TTL"`
}

CacheRedisConfig configures cache.backend=redis: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB) shared by every process. Read only when that backend is selected.

type CacheRedisTLS added in v0.2.0

type CacheRedisTLS struct {
	Enabled            bool   `yaml:"enabled" env:"WH_CACHE_REDIS_TLS_ENABLED"`
	CAFile             string `yaml:"ca_file" env:"WH_CACHE_REDIS_TLS_CA_FILE"`
	CertFile           string `yaml:"cert_file" env:"WH_CACHE_REDIS_TLS_CERT_FILE"`
	KeyFile            string `yaml:"key_file" env:"WH_CACHE_REDIS_TLS_KEY_FILE"`
	ServerName         string `yaml:"server_name" env:"WH_CACHE_REDIS_TLS_SERVER_NAME"`
	InsecureSkipVerify bool   `yaml:"insecure_skip_verify" env:"WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY"`
}

CacheRedisTLS is cache.redis.tls. The files are paths, read at boot.

func (CacheRedisTLS) Config added in v0.2.0

func (t CacheRedisTLS) Config() (*tls.Config, error)

Config builds the tls.Config the block describes, reading its files, or nil when TLS is off. A file set while TLS is off is an error rather than a silently plaintext connection.

type ClickHouse

type ClickHouse struct {
	// MaxTotalConns caps the native connections the process may hold open
	// across its pools: the settings directory's clickhouse.max_open_conns
	// must not exceed it. 0, the default, is no ceiling.
	MaxTotalConns int `yaml:"max_total_conns" env:"WH_CH_MAX_TOTAL_CONNS"`
}

ClickHouse holds the connection ceiling. The wiring — address, HTTP port and scheme, database, username, password, query timeout, TLS, headers, pool size — is the settings directory's `clickhouse` block (hot-reloadable: a change swaps the connection; the password is a tenant's own, #529). The ceiling stays here because it is capacity, sized once per process, not wiring.

type Config

type Config struct {
	// DataDir is the root for embedded state. NATS JetStream lives at
	// `<DataDir>/nats`; Pebble, holding every tenant's dedupe store while any
	// tenant has dedupe enabled, at `<DataDir>/pebble`.
	// Subdirectory names are conventions, not config — one knob, one mount.
	// In a container this MUST resolve to a host-backed volume; the relative
	// `./data` default is fine for local binary use only.
	DataDir string `yaml:"data_dir" env:"WH_DATA_DIR"`
	// Roles are the components this process runs (every role by default);
	// a Deployment per role differs only in this. See Role.
	Roles []Role `yaml:"roles" env:"WH_ROLES"`
	// InstanceID names this process: logged at boot, and the holder a
	// coord.backend=nats lease names. Empty resolves to <hostname>-<8 hex> at
	// Load.
	InstanceID string     `yaml:"instance_id" env:"WH_INSTANCE_ID"`
	Server     Server     `yaml:"server"`
	ClickHouse ClickHouse `yaml:"clickhouse"`
	MQ         MQ         `yaml:"mq"`
	Cache      Cache      `yaml:"cache"`
	Dedupe     Dedupe     `yaml:"dedupe"`
	Coord      Coord      `yaml:"coord"`
	Auth       Auth       `yaml:"auth"`
	OTel       OTel       `yaml:"otel"`
	Prometheus Prometheus `yaml:"prometheus"`
	Settings   Settings   `yaml:"settings"`
}

Config is the top-level application configuration.

func Load

func Load(path string) (*Config, error)

Load reads config from a YAML file (if it exists) with env var overrides. Both sources are strict: a YAML key the Config struct doesn't declare, or a WH_* environment variable no field binds, is an error naming every offender, so a tunable that moved to the settings directory (or a typo) can't be silently ignored. The environment always carries unrelated names, so only the WH_ prefix is checked there. Boot is the validator for this half of configuration — there is no dry run; a refused boot is the loud signal.

func (*Config) Distributed added in v0.2.0

func (c *Config) Distributed() bool

Distributed reports whether the message queue is shared with other processes. The embedded one listens on no port, so while it is selected every process is an island: nothing else can reach its queue.

func (*Config) Has added in v0.2.0

func (c *Config) Has(r Role) bool

Has reports whether this process runs role r.

func (*Config) NeedsDataDir added in v0.2.0

func (c *Config) NeedsDataDir() bool

NeedsDataDir reports whether a backend this process opens keeps state under data_dir, and so whether boot must probe it (CheckDataDir). Only the api role opens the dedupe stores.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the loaded configuration for logical consistency.

func (*Config) Warnings added in v0.2.0

func (c *Config) Warnings() []string

Warnings returns what a valid configuration is still likely to get wrong, one line each, for boot to log at WARN. None is an error: an ignored block is harmless, and the shared-queue ones are correct for a single replica, which one process cannot tell from many.

type Coord added in v0.2.0

type Coord struct {
	Backend CoordBackend `yaml:"backend" env:"WH_COORD_BACKEND"`
	// NATS is read only when Backend is nats.
	NATS CoordNATSConfig `yaml:"nats"`
}

Coord selects the coordination layer.

type CoordBackend added in v0.2.0

type CoordBackend string

CoordBackend names where leases are held: the sweeper's on the embedded MQ, and the ingest processes' on a shared one.

const CoordLocal CoordBackend = "local"

CoordLocal holds leases in this process, which is enough while no other process shares its queue.

const CoordNATS CoordBackend = "nats"

CoordNATS holds leases in a KV bucket on mq.nats's connection, so every process on the shared queue contends for the same ones. It needs mq.backend=nats; its settings are the coord.nats block.

type CoordNATSConfig added in v0.2.0

type CoordNATSConfig struct {
	// Bucket is the KV bucket the leases live in; empty is
	// <mq.nats.subject_prefix>_coord, the name the generated manifests give it.
	Bucket string `yaml:"bucket" env:"WH_COORD_NATS_BUCKET"`
}

CoordNATSConfig names the operator's KV bucket. There is no connection block: coord.backend=nats rides mq.nats's connection and credentials.

type Dedupe

type Dedupe struct {
	Backend DedupeBackend `yaml:"backend" env:"WH_DEDUPE_BACKEND"`
	// Lease is how long a claimed id stays pending while its record is
	// published; a claim its request never settles lapses after it.
	Lease time.Duration `yaml:"lease" env:"WH_DEDUPE_LEASE"`
	// ReserveConcurrency bounds the parallel calls one Reserve, Commit or
	// Release makes to a remote backend, and sizes its idle connection pool
	// to match. Pebble ignores it.
	ReserveConcurrency int                  `yaml:"reserve_concurrency" env:"WH_DEDUPE_RESERVE_CONCURRENCY"`
	DynamoDB           DedupeDynamoDBConfig `yaml:"dynamodb"`
}

Dedupe selects the dedupe store. Whether a tenant dedupes, on which field, and for how long are settings-directory keys, not this block's.

type DedupeBackend added in v0.2.0

type DedupeBackend string

DedupeBackend names where ingest dedupe keeps the ids it has seen.

const (
	// DedupePebble is the Pebble instance inside this process, under
	// <data_dir>/pebble, opened while any tenant has dedupe on. Seen ids are
	// per process.
	DedupePebble DedupeBackend = "pebble"
	// DedupeDynamoDB is one DynamoDB table every tenant and every process
	// shares, configured by dedupe.dynamodb.
	DedupeDynamoDB DedupeBackend = "dynamodb"
)

type DedupeDynamoDBConfig added in v0.2.0

type DedupeDynamoDBConfig struct {
	// Table is the shared table; WaveHouse never creates it outside
	// dynamodb-local. Required.
	Table string `yaml:"table" env:"WH_DEDUPE_DYNAMODB_TABLE"`
	// Region overrides the SDK chain's (AWS_REGION).
	Region string `yaml:"region" env:"WH_DEDUPE_DYNAMODB_REGION"`
	// Endpoint points the client at dynamodb-local.
	Endpoint    string        `yaml:"endpoint" env:"WH_DEDUPE_DYNAMODB_ENDPOINT"`
	Timeout     time.Duration `yaml:"timeout" env:"WH_DEDUPE_DYNAMODB_TIMEOUT"`
	MaxAttempts int           `yaml:"max_attempts" env:"WH_DEDUPE_DYNAMODB_MAX_ATTEMPTS"`
	RetryMode   string        `yaml:"retry_mode" env:"WH_DEDUPE_DYNAMODB_RETRY_MODE"`
	// CreateTable creates the table at boot if it is missing. Development
	// only: refused unless Endpoint is set.
	CreateTable bool `yaml:"create_table" env:"WH_DEDUPE_DYNAMODB_CREATE_TABLE"`
}

DedupeDynamoDBConfig is the dynamodb backend's block, read only when it is selected. Credentials are the AWS SDK's default chain (EKS Pod Identity, IRSA, AWS_* variables), never keys here.

type MQ

type MQ struct {
	Backend MQBackend `yaml:"backend" env:"WH_MQ_BACKEND"`
	// NATS is read only when Backend is nats.
	NATS MQNATSConfig `yaml:"nats"`
}

MQ selects the message queue. The per-tenant byte budget, mq.max_bytes_gb, is a settings-directory key, not this block's.

type MQBackend added in v0.2.0

type MQBackend string

MQBackend names the message queue implementation.

const MQEmbedded MQBackend = "embedded"

MQEmbedded is the NATS JetStream server inside this process, under <data_dir>/nats.

const MQNATS MQBackend = "nats"

MQNATS is a NATS JetStream cluster the operator runs, holding the streams and durables deployments/nats describes; every process naming it shares one queue. Its settings are the mq.nats block.

type MQNATSConfig added in v0.2.0

type MQNATSConfig struct {
	URLs []string `yaml:"urls" env:"WH_MQ_NATS_URLS"`
	// Name is the connection name the server reports; empty is
	// wavehouse-<hostname>.
	Name string `yaml:"name" env:"WH_MQ_NATS_NAME"`
	// CredsFile, NKeySeedFile and User are exclusive: one way to
	// authenticate, or none.
	CredsFile    string    `yaml:"creds_file" env:"WH_MQ_NATS_CREDS_FILE"`
	NKeySeedFile string    `yaml:"nkey_seed_file" env:"WH_MQ_NATS_NKEY_SEED_FILE"`
	User         string    `yaml:"user" env:"WH_MQ_NATS_USER"`
	PasswordFile string    `yaml:"password_file" env:"WH_MQ_NATS_PASSWORD_FILE"`
	TLS          MQNATSTLS `yaml:"tls"`
	// JSDomain is the JetStream domain, for a leafnode or hub-and-spoke
	// deployment.
	JSDomain      string `yaml:"js_domain" env:"WH_MQ_NATS_JS_DOMAIN"`
	SubjectPrefix string `yaml:"subject_prefix" env:"WH_MQ_NATS_SUBJECT_PREFIX"`
	Partitions    int    `yaml:"partitions" env:"WH_MQ_NATS_PARTITIONS"`
	// Shards is how many shards each partition has, one durable each.
	Shards int `yaml:"shards" env:"WH_MQ_NATS_SHARDS"`
	// IngestConsumer names the shard durables: <ingest_consumer>-<shard>.
	IngestConsumer string `yaml:"ingest_consumer" env:"WH_MQ_NATS_INGEST_CONSUMER"`
	// HistoryStream is looked up by name; empty is <SUBJECT_PREFIX>_HISTORY,
	// the name the generated manifests give it.
	HistoryStream  string        `yaml:"history_stream" env:"WH_MQ_NATS_HISTORY_STREAM"`
	ConnectTimeout time.Duration `yaml:"connect_timeout" env:"WH_MQ_NATS_CONNECT_TIMEOUT"`
	PublishTimeout time.Duration `yaml:"publish_timeout" env:"WH_MQ_NATS_PUBLISH_TIMEOUT"`
	TopologyWait   time.Duration `yaml:"topology_wait" env:"WH_MQ_NATS_TOPOLOGY_WAIT"`
}

MQNATSConfig is how to reach the operator's NATS and what topology to expect there (mq.NATSConfig, which internal/app builds from it). Secrets are file paths only: nothing inline.

type MQNATSTLS added in v0.2.0

type MQNATSTLS struct {
	CAFile         string `yaml:"ca_file" env:"WH_MQ_NATS_TLS_CA_FILE"`
	CertFile       string `yaml:"cert_file" env:"WH_MQ_NATS_TLS_CERT_FILE"`
	KeyFile        string `yaml:"key_file" env:"WH_MQ_NATS_TLS_KEY_FILE"`
	ServerName     string `yaml:"server_name" env:"WH_MQ_NATS_TLS_SERVER_NAME"`
	HandshakeFirst bool   `yaml:"handshake_first" env:"WH_MQ_NATS_TLS_HANDSHAKE_FIRST"`
}

MQNATSTLS is the client side of TLS to the NATS servers.

type OTel

type OTel struct {
	Enabled bool        `yaml:"enabled" env:"WH_OTEL_ENABLED"`
	Traces  OTelTraces  `yaml:"traces"`
	Metrics OTelMetrics `yaml:"metrics"`
	Logs    OTelLogs    `yaml:"logs"`
}

OTel configures the OpenTelemetry pipeline. `enabled` is the master switch; when false, no signals are initialized regardless of the per-signal toggles. The OTLP destination — endpoint, TLS, custom CA, mutual TLS, and auth headers — is configured through the standard OTEL_EXPORTER_OTLP_* environment variables read by the OpenTelemetry SDK, not WaveHouse config. See docs/src/content/docs/configuration.mdx.

type OTelLogs

type OTelLogs struct {
	Enabled    bool    `yaml:"enabled" env:"WH_OTEL_LOGS_ENABLED"`
	SampleRate float64 `yaml:"sample_rate" env:"WH_OTEL_LOGS_SAMPLE_RATE"`
}

OTelLogs sample rate applies to OTLP export of DEBUG/INFO only. WARN and ERROR always export at 100% — dropping them silently during incidents is too dangerous to expose as a knob. Stdout receives 100% of records regardless of this rate (sampling for scraped-log pipelines like Loki/Promtail belongs at the scraper, not the application).

type OTelMetrics

type OTelMetrics struct {
	Enabled bool `yaml:"enabled" env:"WH_OTEL_METRICS_ENABLED"`
}

type OTelTraces

type OTelTraces struct {
	Enabled    bool    `yaml:"enabled" env:"WH_OTEL_TRACES_ENABLED"`
	SampleRate float64 `yaml:"sample_rate" env:"WH_OTEL_TRACES_SAMPLE_RATE"`
}

type Prometheus

type Prometheus struct {
	Enabled bool   `yaml:"enabled" env:"WH_PROMETHEUS_ENABLED"`
	Path    string `yaml:"path" env:"WH_PROMETHEUS_PATH"`
	Port    int    `yaml:"port" env:"WH_PROMETHEUS_PORT"`
}

Prometheus controls a Prometheus exposition endpoint served alongside (or independently of) the OTLP push exporter. It is its own top-level block so operators using Prometheus scraping (Grafana Alloy / Mimir / etc.) don't have to hunt through `[otel]` for an output they don't otherwise care about. Internally the same OTel MeterProvider drives both — the Prometheus exporter is an additional Reader — but the user-facing config keeps the two outputs distinct.

Works in any of three combinations: OTel only, Prometheus only, or both. If only Prometheus is enabled, the MeterProvider is still created so runtime + custom metrics flow to `/metrics`; no OTLP push is attempted.

Port `0` mounts the endpoint on the existing API server router. A non-zero port spins up a dedicated HTTP listener — useful for firewalling metrics off the public API surface in production.

type Role added in v0.2.0

type Role string

Role is one part of the work a process can run.

const (
	// RoleAPI serves the HTTP API and everything that answers it: schema
	// discovery, the auth verifiers, the dedupe stores, and the SSE hub with
	// its bridge off the queue and its keepalive wheels. Per process: every
	// API process runs its own.
	RoleAPI Role = "api"
	// RoleIngest runs the ingest worker, queue to ClickHouse. Under nats each
	// shard of the queue is delivered to one ingest process at a time.
	RoleIngest Role = "ingest"
	// RoleSweeper runs the sweeper, one per queue, under the sweeper lease, on
	// the embedded MQ only: under nats the streams' own retention replaces it.
	RoleSweeper Role = "sweeper"
)

func AllRoles added in v0.2.0

func AllRoles() []Role

AllRoles is every role, the default: one process runs all the work.

type Server

type Server struct {
	Port            int `yaml:"port" env:"WH_SERVER_PORT"`
	ShutdownTimeout int `yaml:"shutdown_timeout" env:"WH_SERVER_SHUTDOWN_TIMEOUT"`
}

Server holds listener wiring. The CORS allowlist is a tenant tunable and lives in the settings directory's config.json (internal/settings), as do the SSE keepalive and gap-window knobs (stream.*).

type Settings added in v0.2.0

type Settings struct {
	Dir string `yaml:"dir" env:"WH_SETTINGS_DIR"`
}

Settings locates the hot-reloadable settings directory — the four JSON documents (roles.json, policies.json, pipes.json, config.json) validated by internal/settings. Boot-tier by necessity: it's the pointer the reload machinery follows, so it can't live behind itself. REQUIRED, with no default: a baked-in path would turn a missing mount into silent misconfiguration instead of an explicit operator choice, and the binary has no compiled tunable defaults to fall back on — `wavehouse bootstrap [dir]` writes the starter directory (the container images preset WH_SETTINGS_DIR=/app/settings and expect a mount there).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL