api

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: Apache-2.0 Imports: 50 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsMutation added in v0.2.0

func IsMutation(sql string) bool

IsMutation reports whether sql's leading statement is a non-SELECT — i.e. one that returns no result set and must go through Exec, not Query. Leading whitespace and comments are skipped as ClickHouse's lexer skips them, then the first bareword is matched whole, case-insensitively, against mutationVerbs. After a WITH list ClickHouse parses only SELECT, a FROM-first SELECT or INSERT INTO, so a WITH-led statement is a write exactly when it holds INSERT INTO at the top level (hasTopLevelInsertInto). An `EXECUTE AS <user>` prefix is looked through to the statement it runs. A write classified as a read goes through Query, which runs it and then fails the call, so a client that retries the error writes again.

func NewOpsRouter added in v0.2.0

func NewOpsRouter(deps OpsDependencies) http.Handler

NewOpsRouter creates the router of a process without the api role. Every other route — the tenant routes and the rest of /v1/ops — answers 404: the handlers behind them are not wired here. The reload is gated by the operator key alone, as the ops tree is over a nested directory.

func NewRouter

func NewRouter(deps Dependencies) http.Handler

NewRouter creates the chi router with all routes.

func RequireAdmin

func RequireAdmin(store policy.Source) func(http.Handler) http.Handler

RequireAdmin restricts a route to the policy admin role (policy.AdminRole — configurable via admin_role, "admin" by default). The role established by the auth middleware and the live policy are read per request, so an admin_role change applies on reload. A nil policy (policies.json empty) admits nobody via a role — IsAdmin(nil) is false — so no token can re-open a locked-out deployment through this gate. The exception is the operator key: auth.IsOperator passes this gate even under a nil policy, so an operator can still trigger a settings reload after fixing the files (break-glass). A nil store is the same gate with nothing to read — the operator key alone — which is what NewRouter mounts over a nested settings directory.

Authentication is decoupled from this gate: a missing/invalid/expired token resolves to an empty (non-admin) role and is denied here. Denials go through writeAuthzDenied, so a present-but-invalid token fails loud (401 + token reason) rather than as a bare 403.

func StoreFromContext added in v0.2.0

func StoreFromContext(ctx context.Context) (*settings.Store, bool)

StoreFromContext returns the store TenantMW resolved for this request. Handlers call it once and pass the store down as an argument; nothing below a handler reads it from the context.

func TenantMW added in v0.2.0

func TenantMW(tenants *settings.Registry) func(http.Handler) http.Handler

TenantMW resolves the request's tenant before authentication runs and stores it in the request context: the tenant requestTenant names. A malformed id is a 400; an id that cannot be served is resolveStore's 404 or 503. Every answer, the refusals included, carries Vary: X-Tenant-ID so a shared cache cannot replay one tenant's response to another — added, not set, so the CORS Vary: Origin survives.

func WithStore added in v0.2.0

func WithStore(ctx context.Context, store *settings.Store) context.Context

WithStore returns ctx carrying the request's resolved tenant store. These helpers live here rather than in internal/tenant because settings names tenant.ID, so tenant cannot name settings.Store back.

Types

type BootState

type BootState struct {
	// contains filtered or unexported fields
}

BootState tracks a one-shot startup diagnostic surfaced by /livez. While Err() returns non-nil the binary is considered to be in degraded-boot mode: /livez responds 503 with the diagnostic message instead of 200, so an operator can curl the endpoint to learn why the gateway isn't accepting traffic yet. Once boot work (today: the first tenant's ClickHouse schema discovery) succeeds, Set(nil) flips /livez back to 200.

BootState is safe for concurrent use.

func NewBootState

func NewBootState(initialErr error) *BootState

NewBootState returns a BootState seeded with initialErr. Pass nil if the binary is fully ready at construction time; pass a non-nil error to start in degraded mode (the goroutine that performs boot work calls Set(nil) on success).

func (*BootState) Err

func (b *BootState) Err() error

Err returns the current diagnostic, or nil if the binary is ready.

func (*BootState) Set

func (b *BootState) Set(err error)

Set replaces the current diagnostic. Pass nil to mark the binary ready.

type Check added in v0.2.0

type Check struct {
	Name string
	// Run reports why the dependency cannot do its part, nil while it can.
	Run func(context.Context) error
}

Check is one dependency Readiness consults, named for the 503 body: the ClickHouse pools, the external message queue.

type DLQHandler

type DLQHandler struct {
	Counts mq.DeadLetterStats
}

DLQHandler exposes Dead Letter Queue statistics.

func NewDLQHandler

func NewDLQHandler(stats mq.DeadLetterStats) *DLQHandler

func (*DLQHandler) Stats

func (h *DLQHandler) Stats(w http.ResponseWriter, r *http.Request)

Stats returns per-table message counts on one tenant's dead-letter queue: the tenant ?tenant= names (read strictly, as every ops read does — opsTenant), tenant.Default without it. The queue is the MQ's, not the settings', so it is looked up there: a tenant whose folder was rejected or removed is read like one being served, for as long as its queue is kept, and an id with no queue is a 404. Supports optional ?table= query parameter to filter by table name.

type Dependencies

type Dependencies struct {
	Ingest          *IngestHandler
	Query           *QueryHandler
	SSE             *StreamHandler
	Health          *HealthHandler
	Version         *VersionHandler
	Schema          *SchemaHandler
	DLQ             *DLQHandler
	Pipes           *PipesHandler
	StructuredQuery *StructuredQueryHandler
	// Settings, if non-nil, mounts POST /v1/ops/settings/reload — the API
	// trigger for reloading the settings directory. Nil when no settings
	// directory is configured (nothing to reload).
	Settings *SettingsHandler
	AuthMW   func(http.Handler) http.Handler
	// Tenants resolves the tenant.Header of every tenant route to that
	// tenant's settings store (TenantMW), ahead of AuthMW. The probes,
	// /version, the metrics path, and /v1/ops/* are tenant-exempt.
	Tenants *settings.Registry
	// PolicySource backs the RequireAdmin gate: the admin role (policy.AdminRole)
	// is read live from the adopted policy, so admin_role changes apply on reload.
	// NewRouter ignores it when Tenants is nested, whatever was wired here: the
	// ops gate then admits the operator key alone (see NewRouter).
	PolicySource policy.Source
	// CORSOrigins yields one tenant's allowed CORS origins, read per request
	// so a settings reload applies immediately ((*settings.Store).CORSOrigins
	// in production); which tenant's list answers a request is corsOrigins'
	// rule. An empty or nil list — including a nil func, or no Tenants —
	// denies every browser origin; ["*"] is the only allow-all spelling.
	CORSOrigins func(*settings.Store) []string
	// MetricsHandler, if non-nil, is mounted at MetricsPath as an unauthenticated
	// endpoint (Prometheus convention). Wired by internal/app from the OTel Prometheus
	// exporter when observability.metrics.prometheus.enabled is true AND port is 0.
	MetricsHandler http.Handler
	MetricsPath    string
}

Dependencies holds all handler dependencies.

type HealthHandler

type HealthHandler struct {
	// Checks are Readiness's dependency checks, every one run on every call
	// so the 503 names each one that failed, not just the first. Which
	// dependencies a process has is internal/app's to wire: the ClickHouse
	// pools (chconn.Pools.Ping: every open pool at once, ready at the first
	// answer) and the external NATS; a handler with none is ready once
	// booted.
	Checks []Check
	// Boot is consulted by both Liveness and Readiness. When non-nil and
	// its Err() is non-nil, both endpoints report 503 with the diagnostic
	// — used while boot-time schema discovery is still failing in the
	// retry loop. A nil Boot preserves the pre-retry-loop behaviour
	// (Liveness always 200; Readiness 503 only on a failed check).
	Boot *BootState
}

HealthHandler provides liveness and readiness probes.

func NewHealthHandler

func NewHealthHandler(checks ...Check) *HealthHandler

func (*HealthHandler) Liveness

func (h *HealthHandler) Liveness(w http.ResponseWriter, _ *http.Request)

func (*HealthHandler) Online

func (h *HealthHandler) Online(w http.ResponseWriter, _ *http.Request)

Online is a content-free public liveness ping served at /v1/health for the SDK's "is this server reachable / accepting data" check (and for picking among servers in a distributed setup). It mirrors Liveness's status logic — 200 once boot completes, 503 while boot-time schema discovery is still failing — but writes no body: the caller only branches on the status code, so there's nothing to JSON-encode or cache per request.

It deliberately lives under /v1 rather than reusing /livez: /livez (and /readyz, /healthz) are Kubernetes probe paths an operator may filter out at the reverse proxy, whereas /v1/health is documented public API surface the SDK can rely on staying reachable. It does NOT ping ClickHouse — readiness- based load balancing is the proxy/LB's job (via /readyz), not the client's.

func (*HealthHandler) Readiness

func (h *HealthHandler) Readiness(w http.ResponseWriter, r *http.Request)

type IngestFormat added in v0.2.0

type IngestFormat int

IngestFormat is the wire format of an ingest request body. It comes from the request's declared Content-Type and nothing else: the body never overrides what the client said it sent, so a caller can always tell how their bytes will be read without knowing what the first one happens to be.

const (
	// FormatJSON is the application/json family: one flat object, or a
	// top-level array of them. Which of the two is the body's own business —
	// the first non-whitespace byte picks it — because both are the same
	// format, differing only in arity.
	FormatJSON IngestFormat = iota
	// FormatNDJSON is newline-delimited JSON: one flat object per line. A line
	// that is not a JSON object is a per-record error, never a reason to
	// re-read the body as something else.
	FormatNDJSON
)

func (IngestFormat) String added in v0.2.0

func (f IngestFormat) String() string

String renders a format for error messages and logs.

type IngestHandler

type IngestHandler struct {
	// Registry yields the request tenant's schema registry.
	Registry RegistrySource
	// Dedup resolves the request tenant's deduplicator — the tenant's own
	// store, picked off the store the handler already holds (#583 story 7;
	// dedupe.Stores in production). nil when no dedupe store is wired (tests).
	Dedup func(store *settings.Store) dedupe.Deduplicator
	// DedupeSettings resolves the effective dedupe settings for a table of the
	// request's tenant ((*settings.Store).DedupeFor in production). Called
	// once per record so a settings reload lands at a record boundary — one
	// record never mixes two documents' values. Dedup is skipped when nil.
	DedupeSettings func(store *settings.Store, table string) settings.Dedupe
	// DedupeLease is how long a record's claimed id stays pending while it is
	// published; 0 means dedupe.DefaultLease.
	DedupeLease  time.Duration
	Publisher    mq.Publisher
	PolicySource PolicySource

	// Validator and Checker are the per-record seams a native type layer will
	// take over (see ingest_seams.go). Both are optional: nil means the default
	// implementation, which is today's behavior unchanged.
	Validator RecordValidator
	Checker   InsertChecker
	// contains filtered or unexported fields
}

IngestHandler handles POST /v1/ingest?table={table}

func NewIngestHandler

func NewIngestHandler(registry RegistrySource, pub mq.Publisher) *IngestHandler

func (*IngestHandler) Handle

func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request)

type InsertChecker added in v0.2.0

type InsertChecker interface {
	Matches(actual, required any) bool
	InSet(v any, set []any) bool
}

InsertChecker decides whether a record's value satisfies a policy check clause. Matches answers the scalar `_eq` form (the required value), InSet the `_in` form (set membership). It never sees a record as a whole: the auto-injection of a missing check value stays in prepareRecord, where the ordering against validation and canonicalization is load-bearing.

type OpsDependencies added in v0.2.0

type OpsDependencies struct {
	Health  *HealthHandler
	Version *VersionHandler
	// Settings mounts POST /v1/ops/settings/reload.
	Settings *SettingsHandler
	// AuthMW authenticates the reload. It runs no token verifier (those are
	// the api role's), so the operator key is the one credential that passes.
	AuthMW         func(http.Handler) http.Handler
	MetricsHandler http.Handler
	MetricsPath    string
}

OpsDependencies is what the ops-only listener serves: a process that runs no api role still answers its probes, /version, the same-port metrics endpoint, and the settings reload, so the control plane drives every process's tenant tree the same way.

type PipesHandler

type PipesHandler struct {
	// Source yields a tenant's pipes (the store itself in production).
	Source       func(*settings.Store) pipes.Source
	PolicySource PolicySource // resolves empty role to default_role; may be nil
	// Tenants resolves the tenant the admin reads (List, Get) serve: /v1/ops
	// is tenant-exempt, so they carry no request tenant and read the one
	// ?tenant= names, the default one without it (opsStore).
	Tenants *settings.Registry
	// CHConn yields the request tenant's connection (chconn.Pools.For in
	// production); nil is a tenant on no pool, a 503.
	CHConn func(*settings.Store) driver.Conn
	Cache  cache.Cache
	// contains filtered or unexported fields
}

PipesHandler serves named query pipes: execution for callers, read-only listing for admins. Pipes are defined in the settings directory's pipes.json and read per request, so a reload applies immediately.

func NewPipesHandler

func NewPipesHandler(source func(*settings.Store) pipes.Source, policySource PolicySource, conn func(*settings.Store) driver.Conn, c cache.Cache, queryTimeout func(*settings.Store) time.Duration) *PipesHandler

func (*PipesHandler) Execute

func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request)

Execute runs a named query with the provided parameters.

func (*PipesHandler) Get

Get returns a specific named query of the ?tenant= (admin endpoint).

func (*PipesHandler) List

func (h *PipesHandler) List(w http.ResponseWriter, r *http.Request)

List returns all named queries of the ?tenant= (admin endpoint).

type PolicySource added in v0.2.0

type PolicySource func(*settings.Store) *policy.Policy

PolicySource yields the access-control policy of the request's tenant, read per request so a settings reload applies to the next one ((*settings.Store).Policy in production). A nil policy from a wired source is a deliberate lockout; a nil source means policy filtering is not wired at all, which only tests do.

type QueryHandler

type QueryHandler struct {

	// Tenants resolves the tenant the proxy queries: /v1/ops is
	// tenant-exempt, so the request carries no tenant and the query runs
	// against the one ?tenant= names, the default one without it (opsStore).
	Tenants *settings.Registry
	// contains filtered or unexported fields
}

QueryHandler handles POST /v1/ops/query.

Authorization is enforced at the router (the /v1/ops/* RequireAdmin gate in NewRouter). The handler trusts any caller that reaches it. See internal/api/router.go for the role-gate rationale.

Implementation: a thin proxy to ClickHouse's HTTP interface. The SQL string is forwarded verbatim with `default_format=JSON`, ClickHouse returns either a `{"meta":..., "data":[...], ...}` JSON object for read queries or an empty body for mutations, and the handler emits just the `data` array (or `[]` for mutations) back to the caller.

Why a proxy instead of clickhouse-go's native Query/Exec:

  • ClickHouse classifies statements natively, so any single statement (arbitrary DDL/DML verbs, current and future) and inline FORMAT directives all just work without WaveHouse-side parsing. Multi-statement input (`SELECT 1; TRUNCATE t`) also works when the upstream ClickHouse has multi-query enabled, which is the default in recent versions; older or restrictively-configured servers may reject the second statement with a clear error.
  • There is no IsMutation heuristic to maintain — no leading-verb table, no comment stripper, no CTE-aware paren scanner, no class of bug where a future ClickHouse verb routes the wrong way.
  • ClickHouse's own error messages reach the admin verbatim, which is exactly what they want from an escape hatch.
  • The cache (TieredCache + singleflight) was already removed in an earlier commit; this completes the simplification.

func NewQueryHandler

func NewQueryHandler(target func(*settings.Store) chconn.Target, queryTimeout func(*settings.Store) time.Duration) *QueryHandler

NewQueryHandler builds a handler that proxies to ClickHouse over HTTP. target supplies the base URL (`http://host:8123`), the username/password forwarded via ClickHouse's `X-ClickHouse-User` / `X-ClickHouse-Key` headers (matching the ingest worker's convention in internal/ingest), and the database set as the `?database=` query-string parameter when non-empty.

The HTTP client itself has no Timeout — every request gets a queryTimeout deadline from a context derived from the inbound request (see Handle), which bounds the whole exchange including body read. Setting `Timeout` here too would just duplicate that bound (and silently truncate any inbound context longer than queryTimeout).

func (*QueryHandler) Handle

func (h *QueryHandler) Handle(w http.ResponseWriter, r *http.Request)

type RecordValidator added in v0.2.0

type RecordValidator interface {
	Validate(schema *discovery.TableSchema, record map[string]any) error
	CanonicalizeTimestamps(schema *discovery.TableSchema, record map[string]any)
}

RecordValidator covers the two schema-driven steps of the record pipeline. Validate rejects a record the table's schema cannot accept; CanonicalizeTimestamps rewrites DateTime/DateTime64 values to the canonical wire form in place.

CanonicalizeTimestamps returns nothing, matching discovery's function: it is deliberately fail-open (#372) — a value it cannot read is passed through for ClickHouse to judge, and the row filter is what enforces. Giving it an error return would invite a caller to change that.

The two are one interface because they are one contract — "what this schema says about this record" — evaluated at two points in prepareRecord that must stay apart: the insert-check block sits between them deliberately, so checks keep pre-#372 semantics.

type RegistrySource added in v0.2.0

type RegistrySource func(*settings.Store) *discovery.SchemaRegistry

RegistrySource yields a tenant's schema registry, its own since #583 story 6 (the per-tenant discovery map in internal/app). Nil for a tenant whose registry is not built yet — the beat between its adoption and the hook that builds it — which reads as a schema not loaded yet.

type SchemaHandler

type SchemaHandler struct {
	Registry RegistrySource
	// Tenants resolves the tenant the reads and the refresh serve: /v1/ops
	// is tenant-exempt, so they carry no request tenant and read the one
	// ?tenant= names, the default one without it (opsStore).
	Tenants *settings.Registry
}

SchemaHandler exposes the discovered ClickHouse table schemas.

func NewSchemaHandler

func NewSchemaHandler(registry RegistrySource) *SchemaHandler

func (*SchemaHandler) Get

Get returns the schema for a single table of the ?tenant=, every table's without ?table=.

func (*SchemaHandler) List

func (h *SchemaHandler) List(w http.ResponseWriter, r *http.Request)

List returns all discovered table schemas of the ?tenant=.

func (*SchemaHandler) Refresh

func (h *SchemaHandler) Refresh(w http.ResponseWriter, r *http.Request)

Refresh forces an immediate schema refresh of the ?tenant= from its ClickHouse, then returns its schemas. A tenant with no open pool — one could not be opened for it, such as by the connection ceiling — is a 503 with Retry-After, like the reads it would answer.

type SettingsHandler added in v0.2.0

type SettingsHandler struct {
	Tenants *settings.Registry
}

SettingsHandler serves the ops surface for the hot-reloadable settings directory. Constructed only when a settings directory is configured — with none there is nothing to reload, so the route is simply absent (the same pattern as the DLQ and policy handlers).

func NewSettingsHandler added in v0.2.0

func NewSettingsHandler(tenants *settings.Registry) *SettingsHandler

func (*SettingsHandler) Reload added in v0.2.0

func (h *SettingsHandler) Reload(w http.ResponseWriter, r *http.Request)

Reload handles POST /v1/ops/settings/reload — the API trigger for the same serialized reload path SIGHUP and the directory watcher run. 200 when the directory was adopted (warnings included in the body), 422 when validation rejected it and the previous settings remain in effect.

?tenant= narrows the reload to that tenant's folder of a nested directory (400 malformed, 404 unknown): adopted then speaks for that folder alone, and a 422 means the tenant is no longer served. Without it the whole tree is reloaded, and over a nested directory a 422 can mean adopted in part — the error findings name the folders that were not (settings.Registry.Reload).

type StreamHandler

type StreamHandler struct {
	Hub      *stream.Hub
	Replayer mq.Replayer // gap-fill source; nil disables replay
	// Heartbeater yields the request tenant's keepalive wheel — its own, at
	// its own stream.keepalive_* settings (#597) — or nil for a tenant no
	// longer served, whose streams are ending (Hub.Prune, Served). A nil func
	// means no keepalive is wired (tests).
	Heartbeater func(*settings.Store) *stream.Heartbeater
	Metrics     *stream.Metrics
	// Closing, when set, is closed as the server begins shutting down, and
	// every open stream ends at once — mid-replay too: a stream is a
	// connection to close, not in-flight work for the drain to wait on, and
	// the client reconnects and gap-fills via Last-Event-ID. A nil channel
	// never fires (a harness that serves the handler itself).
	Closing <-chan struct{}
	// Served, when set, reports whether a tenant is still being served. A
	// reload that stops serving one evicts its streams (Hub.Prune), but not a
	// stream TenantMW admitted just before the reload and registered with the
	// Hub just after it: this check, made after registering, ends that one.
	Served func(tenant.ID) bool
}

StreamHandler handles GET /v1/stream

func NewStreamHandler

func NewStreamHandler(hub *stream.Hub, replayer mq.Replayer) *StreamHandler

func (*StreamHandler) Handle

func (h *StreamHandler) Handle(w http.ResponseWriter, r *http.Request)

type StructuredQueryHandler

type StructuredQueryHandler struct {
	// CHConn yields the request tenant's connection (chconn.Pools.For in
	// production); nil is a tenant on no pool, a 503.
	CHConn       func(*settings.Store) driver.Conn
	Cache        cache.Cache
	Registry     RegistrySource
	PolicySource PolicySource
	// contains filtered or unexported fields
}

StructuredQueryHandler handles POST /v1/query?table={table}

func NewStructuredQueryHandler

func NewStructuredQueryHandler(
	conn func(*settings.Store) driver.Conn,
	c cache.Cache,
	registry RegistrySource,
	policyStore PolicySource,
	bucketSecs func(*settings.Store) int,
	queryTimeout func(*settings.Store) time.Duration,
	defaultMaxRows func(*settings.Store) int,
) *StructuredQueryHandler

func (*StructuredQueryHandler) Handle

type VersionHandler

type VersionHandler struct {
	// contains filtered or unexported fields
}

VersionHandler serves the binary's build metadata at /version. The payload is fully static for the process lifetime (the ldflags-injected build info plus the Go runtime version), so it's marshaled once in NewVersionHandler and the cached bytes are written on each request — no per-request encoding.

func NewVersionHandler

func NewVersionHandler(version, gitCommit, buildTime string) *VersionHandler

NewVersionHandler pre-marshals the build-info JSON from the ldflags-injected values (version, git commit, build time — see the Makefile / .goreleaser.yaml) plus runtime.Version(). Marshaling a struct of strings cannot fail, so the error is discarded.

func (*VersionHandler) Handle

func (h *VersionHandler) Handle(w http.ResponseWriter, _ *http.Request)

Handle writes the pre-marshaled build metadata. No auth gate (wired among the public routes in NewRouter) — the values are non-sensitive and already in the startup logs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL