Documentation
¶
Overview ¶
Package clickhouse is TinyRaven's ClickHouse adapter. One Client drives both transports ClickHouse exposes (ADR 0013): the HTTP interface (8123) for read-only queries + liveness, and the native protocol (TCP 9000) for the batched insert hot path. It implements model.CHQuerier, model.Pinger and model.CHInserter.
Index ¶
- type CHError
- type Client
- func (c *Client) Backfill(ctx context.Context, dst, src string, cols []string) error
- func (c *Client) Close() error
- func (c *Client) CreateDatabase(ctx context.Context, name string) error
- func (c *Client) CreateMaterializedView(ctx context.Context, m *model.Materialization) error
- func (c *Client) CreateShadowTable(ctx context.Context, ds *model.Datasource, shadowName string) error
- func (c *Client) DropTable(ctx context.Context, name string) error
- func (c *Client) EnsureReadonlyUser(ctx context.Context, name, password string) error
- func (c *Client) EnsureTable(ctx context.Context, ds *model.Datasource) error
- func (c *Client) ExchangeTables(ctx context.Context, a, b string) error
- func (c *Client) Insert(ctx context.Context, ds *model.Datasource, rows []map[string]any) error
- func (c *Client) InsertSelect(ctx context.Context, sql string, params map[string]string) error
- func (c *Client) Ping(ctx context.Context) error
- func (c *Client) Query(ctx context.Context, sql string, params, settings map[string]string) ([]byte, error)
- func (c *Client) WithDatabase(db string) *Client
- type Config
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CHError ¶
CHError carries a ClickHouse exception code (X-ClickHouse-Exception-Code) so the API layer can map it to a Tinybird-compatible status (ADR 0012).
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client talks to one ClickHouse instance/database over both transports.
func New ¶
New builds a Client. The native conn is opened lazily by the driver (no dial here), so New only fails on bad options; readiness is probed via Ping.
func (*Client) Backfill ¶
Backfill copies cols from src into dst (INSERT INTO dst (cols) SELECT cols FROM src) — the second step of a breaking migration, populating the shadow table from the live one over the columns that survive the change.
ponytail: cols must be the type-compatible overlap; the caller (deploy) drops type-changed and removed columns, which then take their CH defaults in the new schema. A genuinely incompatible backfill would error here at the CH layer.
func (*Client) CreateDatabase ¶
CreateDatabase issues CREATE DATABASE IF NOT EXISTS. It runs against the client's currently-scoped database, so the orchestrator calls it on a client pointed at an existing DB, then WithDatabase to target the new one (ADR 0007).
ponytail: name is backticked but not otherwise sanitized — branch names with '-' or '/' (e.g. tr_feature-x) require the quoting. Mapping a git branch to a legal CH database name is the orchestrator's job.
func (*Client) CreateMaterializedView ¶
CreateMaterializedView creates an incremental MV that writes into an existing target table (CREATE MATERIALIZED VIEW ... TO <target> AS <select>). The target table must already exist; deploy ensures it first (ADR 0010). Idempotent via IF NOT EXISTS — re-running deploy never errors on an existing MV.
ponytail: backfill of pre-existing source rows (ADR 0010) is deploy's concern, not this builder's; this only wires the forward MV.
func (*Client) CreateShadowTable ¶
func (c *Client) CreateShadowTable(ctx context.Context, ds *model.Datasource, shadowName string) error
CreateShadowTable creates shadowName with ds's NEW schema/engine — the first step of a breaking migration. Only the main table is created (not the quarantine sibling): the swap rewrites event data, while quarantine schema is fixed and untouched (ADR 0007).
func (*Client) DropTable ¶
DropTable issues DROP TABLE IF EXISTS — the final cleanup after EXCHANGE TABLES drops the now-shadow (old) table.
func (*Client) EnsureReadonlyUser ¶
EnsureReadonlyUser provisions the dedicated read-only ClickHouse identity that the HTTP read path (/v0/sql, pipe queries) authenticates as (ADR 0011). It runs as the read-write/admin user (via exec). Idempotent: re-running converges to the same state and keeps the password in sync with config.
It issues two statements (the HTTP interface runs one statement per request, so they cannot be combined):
CREATE USER OR REPLACE `<name>` IDENTIFIED BY '<password>' SETTINGS readonly = 2 GRANT SELECT ON *.* TO `<name>`
readonly=2 (not the stock `readonly` profile, which is readonly=1) is the deliberate choice from ADR 0011: it refuses writes, DDL and settings *injection* while still letting the query layer forward per-query settings (resource caps, query cache) as URL args — readonly=1 would reject those. A user under readonly=2 also cannot lower its own readonly value, so the guard can't be turned off from a query. GRANT SELECT gives the SELECT privilege the fresh user otherwise lacks under RBAC; readonly=2 is an orthogonal layer that still blocks writes even if broader privileges were ever granted (defence in depth).
ponytail: resource caps (max_result_rows / max_execution_time; ADR 0025/0011) are NOT baked into this user. They stay per-query, forwarded through Query's settings map by the pipe/sql layer, so caps can vary per endpoint without reprovisioning the user. Bake them onto the user here (SETTINGS ... = N) only if a hard, query-independent ceiling is ever wanted.
func (*Client) EnsureTable ¶
EnsureTable creates the datasource's table and (for MergeTree datasources) its quarantine sibling if they don't exist. This is the MVP bootstrap DDL for local dev — full schema diffing and safe migrations are tr deploy's job (Phase 2). The target database is assumed to exist (provisioned by the container/operator).
Connector engines (Kafka/S3/PostgreSQL; ADR 0019) get NO quarantine sibling: quarantine is the landing zone for rows the Gatherer rejects on the HTTP events path (ADR 0018), and connector tables are pulled by ClickHouse, never ingested through the Gatherer, so a quarantine table would misrepresent the data path.
func (*Client) ExchangeTables ¶
ExchangeTables atomically swaps two tables (EXCHANGE TABLES a AND b). It is the final, atomic step of a breaking migration: swap the rebuilt shadow into the live name with no window of missing data (ADR 0007).
func (*Client) Insert ¶
Insert sends rows as one native batch into <db>.<ds.Name> (ADR 0013, the perf path). Values are appended in ds.Schema column order, coerced from their JSON-decoded Go types to what the native driver expects.
func (*Client) InsertSelect ¶
InsertSelect runs an INSERT INTO ... SELECT over HTTP as the read-write user, binding CH query parameters (param_<name>) — the copy-pipe write path (model.CHWriter). It is exec with params: the read-only identity (readonly=2; ADR 0011) would refuse the write, so this always authenticates as user/password. The body is discarded (an INSERT returns none worth keeping).
func (*Client) Query ¶
func (c *Client) Query(ctx context.Context, sql string, params, settings map[string]string) ([]byte, error)
Query runs sql over the HTTP interface and returns the response body verbatim (ADR 0013). It is the read path — /v0/sql and pipe queries — so it authenticates as the read-only identity when one is configured (readonly=2; ADR 0011), guaranteeing ClickHouse refuses writes/DDL no matter what SQL arrives. params (already prefixed param_<name> by the caller) and settings are forwarded as URL query args; the target db comes from cfg.
func (*Client) WithDatabase ¶
WithDatabase returns a shallow copy of c that targets db, sharing the same HTTP client and native connection. deploy uses it to point DDL at a per-branch workspace database (tr_<branch>; ADR 0007) after CreateDatabase, without reopening connections. Only the HTTP Query/DDL path is re-scoped (all deploy needs); the shared native conn keeps its original Auth.Database, so Insert still targets the original database.
ponytail: Close on either the original or a copy closes the shared native pool. Treat the original as the owner and don't Close copies.
type Config ¶
type Config struct {
HTTPURL string // ClickHouse HTTP, e.g. "http://localhost:8123"
NativeAddr string // native TCP, e.g. "localhost:9000"; empty disables Insert
Database string // target database
User string
Password string
// ROUser/ROPassword are the dedicated read-only identity (readonly=2;
// ADR 0011). When set, the HTTP read path (Query) authenticates as this user
// so ClickHouse structurally refuses writes, DDL and settings injection from
// /v0/sql and pipe queries. When empty, the read path falls back to
// User/Password — backward-compatible, no behaviour change. Writes (native
// Insert) and DDL always use User/Password regardless.
ROUser string
ROPassword string
}
Config is the connection info for both transports. Populated by the orchestrator from internal/config.
Source Files