sqlgen

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	ParquetNullRowIDMessage     = "parquet scan produced NULL row_id: a scanned object violates the export schema invariant (#189/#256)"
	ParquetNullChangedAtMessage = "parquet scan produced NULL changed_at: a scanned object violates the export schema invariant (#189/#256)"
)

ParquetNullRowIDMessage and ParquetNullChangedAtMessage are the texts the scan-level system-column guard raises. Each names the offending column and the invariant rather than the object: the scan source is schema-blind and path-blind, and a path in a DuckDB error would be a storage-location leak (#306). The manifest entry and the pre-read validator identify the offending object.

Correlating a fired guard back to a specific object is the read path's job: on a read failure that neither the missing-object classification (#187) nor the corruption confirmation (#251) claims, federated.identifyGuardViolations re-reads each manifest-listed object through this same guarded source one file at a time and names the violator(s) in the returned ParquetGuardViolationError and the engine log (#351). The trigger is deliberately not a guard-specific classification: recognizing a fired guard would mean matching error text, which misses the BIGINT CAST channel entirely — its wording is DuckDB's own — so identification decides by differential drain instead. Manual bisection (design.md §5) remains the fallback for hint-authored path sets, which identification does not cover.

View Source 
const FlushGraceCutoffDisabled int64 = math.MaxInt64

FlushGraceCutoffDisabled renders the pre-#252 dirty barrier: real flushed_at stamps are epoch milliseconds, so `flushed_at >= MaxInt64` is never true and only flushed_at = 0 rows count as dirty. It is also the defensive default when a render path fails to supply the cutoff — rendering 0 would pull every flushed row back to hot (bypassing the parquet tiers), and an absent key would render invalid SQL.

Variables

View Source 
var AdvancedQueryTemplateDuckDB = template.Must(template.New("optimizedQueryDuckDB").Funcs(template.FuncMap{
	"add": func(a, b int) int { return a + b },
}).Parse(`
WITH
dirty_ids AS (
  SELECT row_id
  FROM postgres_scan('{{.PG_CONN}}', '{{.ChangeLogSchema}}', '{{.ChangeLogScanTable}}')
  WHERE schema_id = {{.SCHEMA_ID}}
    AND (flushed_at = 0{{if .HasHot}} OR flushed_at >= {{.FlushGraceCutoffMs}}{{end}})
),

s3_source AS (
  SELECT
    {{.S3SourceSelect}},
    1 AS source_tier_priority
  -- union_by_name resolves the schema UNION across parquet generations
  -- (#189): a file written before an attribute existed contributes NULL for
  -- it, and same-named columns with different physical types widen to the
  -- common supertype instead of being coerced to the first file's schema
  -- (which silently corrupted values on integer→double evolution). The
  -- corruption loudness this relaxes is restored by the pre-read
  -- system-column invariant validator (parquet_schema_validation.go).
  -- A column absent from EVERY file is projected as a typed NULL by the
  -- scan source instead (#255, never-flushed attributes).
  FROM {{.S3_SCAN_SOURCE}}
  WHERE
    CAST(row_id AS UUID) NOT IN (SELECT row_id FROM dirty_ids)
    -- Predicate pushdown as a row_id semijoin: a row qualifies when ANY of
    -- its parquet versions matches, and ALL of its versions then enter the
    -- ranked dedup so the latest version wins before the final filter in
    -- visible. Filtering versions directly here dropped newer non-matching
    -- versions pre-dedup and resurrected stale base rows whose old values
    -- still matched (#173).
    AND row_id IN (
      SELECT row_id FROM {{.S3_SCAN_SOURCE}}
      WHERE ({{.LOGICAL_WHERE_CLAUSE}})
    )
),
{{if .HasHot}}
pg_source AS (
  SELECT
    {{.PGSourceSelect}},
    3 AS source_tier_priority
  FROM postgres_scan('{{.PG_CONN}}', '{{.ChangeLogSchema}}', '{{.ChangeLogScanTable}}') cl
  JOIN postgres_scan('{{.PG_CONN}}',
    '{{.MainSchema}}',
    '{{.MainScanTable}}'
  ) m
    ON cl.schema_id = m.ltbase_schema_id
    AND cl.row_id = m.ltbase_row_id
  {{if .HasEAVPivot}}
  LEFT JOIN (
    SELECT row_id::VARCHAR as row_id, schema_id,
      {{.EAVPivotSelect}}
    FROM postgres_scan('{{.PG_CONN}}', '{{.EAVSchema}}', '{{.EAVScanTable}}')
    WHERE attr_id IN ({{.EAVPivotAttrs}})
    GROUP BY schema_id, row_id
  ) hot_vals ON hot_vals.schema_id = cl.schema_id AND hot_vals.row_id = cl.row_id::VARCHAR
  {{end}}
  WHERE cl.schema_id = {{.SCHEMA_ID}}
    AND (cl.flushed_at = 0 OR cl.flushed_at >= {{.FlushGraceCutoffMs}})
    AND m.ltbase_schema_id = {{.SCHEMA_ID}}
    AND ({{.PG_WHERE_CLAUSE}})
  GROUP BY {{.PGGroupBy}}
),
{{end}}
unified AS (
  SELECT * FROM s3_source{{if .HasHot}}
  UNION ALL
  SELECT * FROM pg_source{{end}}
),

ranked AS (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY row_id
      ORDER BY ver_ts DESC, source_tier_priority DESC, deleted_ts DESC, row_id ASC
    ) AS rn
  FROM unified
),

visible AS (
  SELECT *
  FROM ranked
  WHERE rn = 1
    AND (deleted_ts IS NULL OR deleted_ts = 0)
    AND ({{.LOGICAL_WHERE_CLAUSE}})
    -- Keyset cursor evaluates post-dedup: a WHERE in ranked would filter row
    -- versions before ROW_NUMBER, letting a superseded version win rn = 1
    -- and resurrect for cursors over any version-varying column — business
    -- attributes and created_at alike (#212), the keyset twin of #173. It
    -- renders after the logical clause so positional placeholder order keeps
    -- matching arg order (keyset args are appended last).
    {{if .HAS_KEYSET}}AND ({{.KEYSET_WHERE_CLAUSE}}){{end}}
)

SELECT
  {{.OuterSelect}},
  COUNT(*) OVER() AS total_records,
  CEIL(COUNT(*) OVER()::DOUBLE / NULLIF({{.PAGE_SIZE}}, 0))::BIGINT AS total_pages,
  {{if .HAS_KEYSET}}
  1::BIGINT AS current_page
  {{else}}
  (FLOOR({{.OFFSET}}::DOUBLE / NULLIF({{.PAGE_SIZE}}, 0)) + 1)::BIGINT AS current_page
  {{end}}
FROM visible
{{if .HAS_KEYSET}}
ORDER BY {{.ORDER_BY}}
LIMIT {{.PAGE_SIZE}}
{{else}}
ORDER BY {{.NON_KEYSET_ORDER_BY}}
LIMIT {{.PAGE_SIZE}} OFFSET {{.OFFSET}}
{{end}};
`))

AdvancedQueryTemplateDuckDB is the DuckDB SQL template used for federated queries. It accepts dynamically-generated SQL fragments for S3 source projection, PG source projection, EAV pivot, and final outer SELECT, supporting any schema layout. Metadata columns (total_records, total_pages, current_page) are rendered in the template directly so that PAGE_SIZE and OFFSET template vars are properly expanded. Resource pragmas (threads / memory_limit) are deliberately absent: they are connection-level configuration (DuckDBConfig via applyResourcePragmas), and a per-query PRAGMA would override the configured values on every execution. HasHot selects the tier form (#184): hot-excluded PreferredTiers drop the pg_source data CTE and its UNION ALL branch, while dirty_ids always renders — it is the consistency barrier, not a hot data source, so unflushed rows stay consistently invisible instead of resurfacing as stale parquet versions. Callers must always set HasHot (a missing map key renders as false and would silently prune pg_source). FlushGraceCutoffMs widens the dirty barrier (#252): rows flushed at or after the cutoff (the instant this query resolved its parquet path set, minus the configured clock-skew margin) count as dirty even though flushed_at != 0. The comparison is inclusive because millisecond stamps cannot order a mark and a path resolution landing in the same tick — the ambiguous tick must resolve toward visibility (review P1). The flush appends the manifest before marking, so a row flushed before path resolution already has its delta listed — the widening therefore catches exactly the rows racing this query, keeping them hot-readable instead of invisible. It renders only in the HasHot form: widening is safe solely because pg_source serves the discarded rows, so hot-excluded shapes keep the strict flushed_at = 0 barrier (their contract is flushed-data-only, and discarding without a hot server would drop rows whose delta IS listed). The cutoff is a server-generated int64 (or the compile-time sentinel that Bind splices); callers must always set it — FlushGraceCutoffDisabled (MaxInt64) restores the exact strict barrier.

View Source 
var ErrListInOrderBy = fmt.Errorf("LIST type attributes cannot be used in ORDER BY")

ErrListInOrderBy is returned when a LIST type attribute is used in ORDER BY.

Functions

func AppendDirtyExclusion 

func AppendDirtyExclusion(baseClause string, dirtyIDs []uuid.UUID) (string, []any)

AppendDirtyExclusion adds a NOT IN clause excluding dirty row ids. dirtyIDs are converted to strings for DuckDB parameterization using ? placeholders.

func BuildBenchmarkOuterSelect 

func BuildBenchmarkOuterSelect(schemaID int16) string

BuildBenchmarkOuterSelect builds an outer SELECT for benchmark parquet data that maps flat attribute columns to entity_main column descriptors.

func BuildBenchmarkS3Projection 

func BuildBenchmarkS3Projection(schemaID int16) string

BuildBenchmarkS3Projection builds an S3 projection for benchmark parquet data. Column-bound attributes are read as flat columns; EAV-only attributes are extracted from the attributes_json column to model production cold-tier costs. Columns are emitted in alphabetical order to match PG source projection ordering.

func BuildDuckClause 

func BuildDuckClause(cond forma.Condition, cache forma.SchemaAttributeCache) (string, []any, error)

func BuildDuckDBQuery 

func BuildDuckDBQuery(tpl *template.Template, params any, q *model.FederatedAttributeQuery, dirtyIDs []uuid.UUID, dual *DualClauses) (string, []any, error)

BuildDuckDBQuery prepares a DuckDB SQL string and its arguments for a federated query. It accepts optional DualClauses produced by ToDualClauses; when provided it will use the DuckClause and DuckArgs as the base where clause and inject PgMainClause into template params so the template (or tests) can observe the pushdown fragment. Dirty-ID exclusions are appended to the DuckDB clause regardless of source.

func BuildListPredicate 

func BuildListPredicate(column, operator, value string, elementType forma.ValueType) (string, any, error)

BuildListPredicate generates a DuckDB predicate for LIST column operations. Supported operators:

  • equals: list_contains(col, value) - checks if value is in the list
  • not_equals: NOT list_contains(col, value)
  • contains: list_any_match(col, x -> x LIKE '%value%')
  • starts_with: list_any_match(col, x -> x LIKE 'value%')
  • gt/gte/lt/lte: list_any_match(col, x -> x OP value)

Returns the SQL fragment and the parameter value to bind.

func BuildParquetScanSource 

func BuildParquetScanSource(pathsSQL string, missing []NullScanColumn) string

BuildParquetScanSource renders the parquet scan for the advanced template's two scan sites: the system-column guard above, plus (#255) a typed NULL for every current-schema column absent from every file in the set.

The guard renders unconditionally, so this no longer has a "bare read_parquet" idle state — the pre-#256 byte-identity of the zero-missing output is deliberately retired. Every rendered-SQL contract that pinned it (including the #214 design-doc guard and docs/federated-query/design.md §5) moves in lockstep.

func BuildPgMainClause 

func BuildPgMainClause(cond forma.Condition, cache forma.SchemaAttributeCache, paramIndex *int) (string, []any, error)

func BuildSchemaDrivenTemplateParams 

func BuildSchemaDrivenTemplateParams(sp *SchemaProjection, schemaID int16) map[string]any

BuildSchemaDrivenTemplateParams computes template parameters from a SchemaProjection.

func CastExpression 

func CastExpression(columnOrExpr string, v forma.ValueType) string

CastExpression returns a DuckDB-safe CAST expression for a column or expression. The caller is responsible for ensuring the identifier/expression is safe (e.g. using ident helper).

func ConvertPgMainValue 

func ConvertPgMainValue(valStr string, attr string, meta forma.AttributeMetadata) (any, error)

ConvertPgMainValue converts a string value to the appropriate Go type based on attribute metadata. It is the canonical value converter for Postgres main-table predicates, shared by the dual-path generator and the hybrid condition builder. Numeric-family literals keep their own type via TryParseNumber: a literal denoting an integer in int64 range binds as exact int64 in every accepted spelling ("42", "42.0", "9.007199254740993e15" — #357), lossless for bigint beyond 2^53; genuinely fractional literals bind as float64.

func DuckDBNullScanType 

func DuckDBNullScanType(vt forma.ValueType, itemsType forma.ValueType) string

DuckDBNullScanType maps an attribute's value type to the DuckDB type its parquet column would carry, for NULL::<type> augmentation. Kept in lockstep with buildEAVPivotExpr / eavElementCastExpr (hot leg) and cdc.castEAVValue (export leg): a mismatch would widen the UNION ALL and re-open #205.

func FederatedQueryHasHot 

func FederatedQueryHasHot(q *model.FederatedAttributeQuery) bool

FederatedQueryHasHot reports whether the hot tier participates in the advanced-template render (#184). Empty PreferredTiers means the default all-tier form (the HTTP layer and harness both normalize to all three, and legacy nil-query renders must keep the historical full shape); a non-empty list participates only when it names hot. Routing has already intercepted the hot-only cases (engine gate), so a false here always coexists with a parquet source. internal/queryplan mirrors this hasHot membership in the shape hash — the two must stay in lockstep or the plan cache serves the wrong skeleton.

func HashShapeParts 

func HashShapeParts(parts ...string) uint64

HashShapeParts fingerprints an ordered list of shape components with FNV-64a, inserting a separator so part boundaries cannot collide ("ab","c" vs "a","bc").

func IsBenchmarkSchemaID 

func IsBenchmarkSchemaID(schemaID int16) bool

func IsListType 

func IsListType(v forma.ValueType) bool

IsListType returns true if the ValueType represents a list/array.

func MapValueTypeToDuckDBType 

func MapValueTypeToDuckDBType(v forma.ValueType) string

MapValueTypeToDuckDBType maps forma.ValueType to a DuckDB SQL type string. For LIST types, returns the element type only (caller must wrap in LIST(...) if needed).

func MapValueTypeToListDuckDBType 

func MapValueTypeToListDuckDBType(elementType forma.ValueType) string

MapValueTypeToListDuckDBType returns the DuckDB LIST type for an array of the given element type. e.g., ValueTypeText -> "LIST(VARCHAR)", ValueTypeInteger -> "LIST(INTEGER)"

func MergeTemplateParamsWithDirtyIDs 

func MergeTemplateParamsWithDirtyIDs(params any, dirtyIDs []uuid.UUID) any

func ParquetAttrColumn 

func ParquetAttrColumn(attr string) string

ParquetAttrColumn maps a logical attribute name to its physical column name — the single naming contract shared by the CDC parquet writer (internal/cdc, which aliases every exported attribute through it) and the federated DuckDB reader (this package, whose unified CTE columns must carry the same names). The two sides cannot diverge: LOGICAL_WHERE_CLAUSE renders both against raw read_parquet (physical columns) and against the visible CTE (unified columns), so unified name and parquet name must be the same string. The mapping must also stay byte-stable across releases: parquet files already flushed to S3 were written with it.

func ParseDateValue 

func ParseDateValue(valStr string, meta forma.AttributeMetadata) (any, error)

func RenderDirtyIDsValuesCSV 

func RenderDirtyIDsValuesCSV(dirtyIDs []uuid.UUID) string

func RenderDuckDBQuery 

func RenderDuckDBQuery(tpl *template.Template, params any, whereArgs []any) (string, []any, error)

RenderDuckDBQuery renders a DuckDB SQL template (which uses "?" placeholders) and combines the provided whereArgs (typically from buildDuckClause) with the template-collected args. The order is: whereArgs first, then template args.

func RenderS3ParquetPath 

func RenderS3ParquetPath(tmpl string, schemaID int16) (string, error)

RenderS3ParquetPath interpolates a simple Go template for parquet path rendering. Example template: "s3://bucket/path/schema_{{.SchemaID}}/data.parquet"

func RenderSQLTemplate 

func RenderSQLTemplate(tpl *template.Template, data any) (string, []any, error)

Convenience helper: one-shot render

func ToDuckDBParam 

func ToDuckDBParam(value any, v forma.ValueType) (any, error)

ToDuckDBParam converts a Go value to the form expected by DuckDB drivers for the given value type. The predicate normalizer binds this result, not parseDuckDBRawParam's, so each numeric arm's output type is load-bearing:

  • uuid.UUID -> string
  • time.Time -> int64 epoch-ms (BIGINT)
  • smallint/integer -> an int64 passes through unchanged (#355); every other accepted input widens to float64
  • bigint/numeric -> a decimal string, never a number (see toDuckDBDecimalParam): exact for int/int64/string inputs, while a float64 input is rendered by decimalString's %.15g

func ValidateOrderByAttributesForListTypes 

func ValidateOrderByAttributesForListTypes(orderBy []model.AttributeOrder) error

ValidateOrderByAttributesForListTypes checks if resolved model.AttributeOrder entries are LIST types. This variant works with the internal model.AttributeOrder type used after attribute resolution.

func ValidateOrderByForListTypes 

func ValidateOrderByForListTypes(orderBy []forma.OrderBy, getValueType func(attrName string) (forma.ValueType, bool)) error

ValidateOrderByForListTypes checks if any ORDER BY attributes are LIST types. Returns an error if a LIST type is found (LIST columns cannot be used in ORDER BY). Uses forma.OrderBy (the DSL type with Attribute string field).

func ValidateParquetAttrColumns 

func ValidateParquetAttrColumns(cache forma.SchemaAttributeCache) error

ValidateParquetAttrColumns rejects attribute sets whose folded parquet column names land on a reserved system column or collide with each other (the fold is lossy: "contact.name" and "contact_name" both become contact_name). Schema registration calls it so an unusable schema is rejected before it accepts hot-tier writes; the CDC writer and the federated reader call it again as defense in depth. Plain operator error, never forma.ErrInvalidInput. Attributes are checked in sorted order so the error message is deterministic.

func WalkHybridCondition 

func WalkHybridCondition(cond forma.Condition, cache forma.SchemaAttributeCache, emit TypedLeafEmitter) (string, []any, error)

WalkHybridCondition normalizes cond with the hybrid leaf target (parse-once) and walks it with the hybrid composite style, emitting each leaf through the caller's typed emitter. It is the entrypoint for the hybrid condition builder: composite traversal and leaf parsing both live in the shared typed walker, so the builder only formats already-resolved leaves.

Types

type DualClausePlan 

type DualClausePlan struct {
	PgClause     string
	PgMainClause string
	DuckClause   string
	// ParamSpan is how many $N placeholders the PG fragments consume,
	// starting from the paramIndex the plan was built with.
	ParamSpan int
}

DualClausePlan is the shape-derived half of ToDualClauses (#142 phase 4): the three clause skeletons plus the $N span they consume. Values are bound per request through Bind, which re-runs the same walker decision paths with value-only emitters — argument order therefore matches ToDualClauses by construction, not by parallel bookkeeping.

func PlanDualClauses 

func PlanDualClauses(condition forma.Condition, eavTable string, schemaID int16, cache forma.SchemaAttributeCache, startParamIndex int) (*DualClausePlan, error)

PlanDualClauses compiles the shape-dependent clause skeletons for condition. startParamIndex must equal the paramIndex value Bind will later be called with (placeholder numbering is part of the skeleton text).

func (*DualClausePlan) Bind 

func (p *DualClausePlan) Bind(condition forma.Condition, cache forma.SchemaAttributeCache, paramIndex *int) (DualClauses, error)

Bind produces the per-request DualClauses for a condition tree with the same shape the plan was compiled from: cached skeletons plus freshly bound args. paramIndex is advanced by the plan's span, mirroring ToDualClauses. The condition is normalized into the typed predicate tree once; all three arg slices bind from that shared tree.

type DualClauses 

type DualClauses struct {
	PgClause     string // existing EAV-based clause (EXISTS...)
	PgArgs       []any
	PgMainClause string // predicates that can be pushed into entity_main (m.*)
	PgMainArgs   []any

	DuckClause string
	DuckArgs   []any
}

DualClauses contains SQL fragments and argument lists for both Postgres and DuckDB.

func ToDualClauses 

func ToDualClauses(
	condition forma.Condition,
	eavTable string,
	schemaID int16,
	cache forma.SchemaAttributeCache,
	paramIndex *int,
) (DualClauses, error)

ToDualClauses generates Postgres and DuckDB WHERE fragments for the given condition. - PgClause reuses existing SQLGenerator (EAV-based EXISTS expressions). - PgMainClause contains predicates suitable for entity_main pushdown. - DuckClause maps attributes to column names when available and emits a simple DuckDB-style clause. Placeholder styles: PgClause (the EAV EXISTS clause, genuinely Postgres-bound) uses "$n"; DuckClause and PgMainClause both use DuckDB positional "?", because PgMainClause is only ever embedded in the DuckDB federated template (see pgMainTypedEmitter and #161). Args are returned in order for all three.

The condition tree is normalized into the typed predicate IR exactly once (#143); the three emitters below consume that shared tree.

type DuckDBCompiledQuery 

type DuckDBCompiledQuery struct {
	Skeleton string
	TplArgs  []any
	HasDirty bool
	// HasHot records the tier form the skeleton was rendered with (#184):
	// hot-excluded skeletons have no pg_source CTE, so Bind must drop the
	// PgMainArgs occurrence to keep binds aligned with placeholders. The
	// shape hash splits on hasHot membership, so a cache hit never crosses
	// tier forms.
	HasHot bool
}

DuckDBCompiledQuery is the shape/scope-stable half of BuildDuckDBQuery for the production path (advanced template + dual clauses): the rendered SQL skeleton plus the template-collected args (#142 phase 5). Dirty IDs and all condition/keyset operands stay per-request and flow through Bind.

func CompileDuckDBQuery 

func CompileDuckDBQuery(tpl *template.Template, params map[string]any, q *model.FederatedAttributeQuery, dual *DualClauses, hasDirty bool) (*DuckDBCompiledQuery, error)

CompileDuckDBQuery renders the advanced-template skeleton once for a query shape. It mirrors BuildDuckDBQuery's dual-clause branch exactly, with the dirty-ID CSV replaced by a sentinel (hasDirty selects the template branch, so it must be part of the caller's cache key). It returns nil when the input is not the cacheable production path (non-advanced template or empty dual clause); callers must then fall back to BuildDuckDBQuery.

func (*DuckDBCompiledQuery) Bind 

func (c *DuckDBCompiledQuery) Bind(q *model.FederatedAttributeQuery, dual DualClauses, dirtyIDs []uuid.UUID, graceCutoffMs int64) (string, []any)

Bind produces the executable SQL and full argument list for a request whose shape matches the compiled skeleton: dirty CSV and flush-grace cutoff spliced, condition args in the advanced-template interleave (DuckArgs, PgMainArgs, DuckArgs), keyset cursor values, then the cached template args.

type DuckLeafPayload 

type DuckLeafPayload struct {
	Err       error
	Column    string
	SQLOp     string
	ValueType forma.ValueType
	TextLike  bool
	Param     any
}

DuckLeafPayload is the DuckDB emission payload (lenient parse policy). TextLike leaves bind the string value without a CAST wrapper.

type HybridLeafPayload 

type HybridLeafPayload struct {
	// Err is the main-branch resolution error (unsupported operator, unknown
	// main table column, or value conversion), surfaced verbatim at emit time
	// before any placeholder is consumed.
	Err    error
	IsMain bool

	// Main-table branch (lenient parse + descriptor-validated metadata).
	MainColumn string // physical column name, unsanitized
	MainSQLOp  string
	MainValue  any

	// EAV branch — computed identically to PgEavLeafPayload (strict parse).
	Eav PgEavLeafPayload
}

HybridLeafPayload carries the parse-once results the hybrid condition builder needs to emit either a main-table predicate or an EAV EXISTS subquery. Routing (IsMain) follows the hybrid column-resolution contract: raw entity_main columns and cache-bound columns emit against entity_main; every other attribute falls to the EAV EXISTS form. Placeholder assignment, identifier sanitization, table names, and the anchor alias remain the emitter's responsibility (they are request/builder state), mirroring the pg-main / pg-eav emitters.

type ListOperatorMapping 

type ListOperatorMapping struct {
	// Operator is the DSL operator name (equals, contains, starts_with, gt, etc.)
	Operator string
	// DuckDBExpr is a template for the DuckDB expression.
	// Placeholders: {{.Column}} for the column name, {{.Value}} for the parameter.
	DuckDBExpr string
	// RequiresLambda indicates if the expression uses a lambda (x -> predicate)
	RequiresLambda bool
}

ListOperatorMapping defines how DSL operators map to DuckDB LIST functions. DSL operators for LIST columns use list_contains or list_any_match patterns.

type NullScanColumn 

type NullScanColumn struct {
	// Name is the folded parquet column name (ParquetAttrColumn output).
	Name string
	// DuckDBType renders as NULL::<type>. It must type-unify with the
	// pg_source leg of the UNION ALL, so it mirrors the hot-tier EAV pivot
	// (buildEAVPivotExpr) and the CDC export (cdc.castEAVValue) — the #205
	// no-widening parity.
	DuckDBType string
}

NullScanColumn names one current-schema attribute column that is physically absent from EVERY parquet file in a query's resolved scan set (#255): an attribute added to the schema before its first flush. The scan source projects it as a typed NULL so both the explicit projection (S3SourceSelect) and the semijoin's logical clause bind, with exact SQL NULL semantics for filters, sorts, and keysets.

type PatternKind 

type PatternKind int

PatternKind classifies the LIKE wildcard normalization applied to a value. 

const (
	PatternKindNone     PatternKind = iota
	PatternKindPrefix               // starts_with -> "value%"
	PatternKindContains             // contains -> "%value%"
)

type PgEavLeafPayload 

type PgEavLeafPayload struct {
	Err         error
	AttrID      int16
	ValueColumn string
	SQLOp       string
	Value       any
}

PgEavLeafPayload is the EAV EXISTS emission payload (strict parse policy).

type PgMainLeafPayload 

type PgMainLeafPayload struct {
	Err    error
	Skip   bool
	Column string
	SQLOp  string
	Value  any
}

PgMainLeafPayload is the entity_main pushdown payload (lenient parse policy). Skip marks leaves pg-main silently drops (unknown attribute or no column binding).

type PredicateGroup 

type PredicateGroup struct {
	Logic    forma.Logic
	Children []PredicateNode
	// FullyPushable reports whether every leaf under this group can be
	// pushed to entity_main. The pg-main walker reads it to veto partial
	// OR branches without re-walking the subtree.
	FullyPushable bool
	// Source is the composite this group was normalized from, kept for the
	// legacy CompositeGuard contract.
	Source *forma.CompositeCondition
}

PredicateGroup mirrors a CompositeCondition.

type PredicateLeaf 

type PredicateLeaf struct {
	// Attr is the original attribute name; Source the leaf it came from.
	Attr   string
	Source *forma.KvCondition

	// HasMeta / Meta carry the schema cache lookup result (AttributeID,
	// ValueType, ColumnBinding).
	HasMeta bool
	Meta    forma.AttributeMetadata

	// Operator is the canonical lenient-parse operator; SQLOperator its SQL
	// form and Pattern the LIKE wildcarding, both zero when the operator is
	// unknown (the payloads carry the error).
	Operator    PredicateOperator
	SQLOperator string
	Pattern     PatternKind

	// Storage and Pushable summarize main-table placement: Storage is where
	// the leaf's value lives, Pushable whether pg-main may push it down.
	Storage  PredicateStorage
	Pushable bool

	PgEav  PgEavLeafPayload
	PgMain PgMainLeafPayload
	Duck   DuckLeafPayload
	Hybrid HybridLeafPayload
}

PredicateLeaf is a KvCondition parsed exactly once: attribute metadata, operator normalization, and the converted values for all three emission targets. Target payloads are computed per the entrypoint's target set and keep their own error so strict/lenient parse policies stay per-target.

type PredicateNode 

type PredicateNode interface {
	// contains filtered or unexported methods
}

PredicateNode is a node in the typed predicate tree produced by normalizePredicates (#143). The tree mirrors the forma.Condition AST one-to-one — composites become PredicateGroup, KvCondition leaves become PredicateLeaf, and nil / unknown-type nodes keep dedicated markers — so each walk style retains its distinct handling (error, skip, identity).

type PredicateOperator 

type PredicateOperator string

PredicateOperator is the canonical logical operator of a leaf, derived from the lenient "op:value" parse. Unknown tokens are kept verbatim; the per-target payloads carry any resulting error. 

const (
	PredicateOpEquals     PredicateOperator = "equals"
	PredicateOpNotEquals  PredicateOperator = "not_equals"
	PredicateOpGt         PredicateOperator = "gt"
	PredicateOpGte        PredicateOperator = "gte"
	PredicateOpLt         PredicateOperator = "lt"
	PredicateOpLte        PredicateOperator = "lte"
	PredicateOpStartsWith PredicateOperator = "starts_with"
	PredicateOpContains   PredicateOperator = "contains"
)

type PredicateStorage 

type PredicateStorage int

PredicateStorage says which physical storage a leaf resolves to. 

const (
	PredicateStorageEAV PredicateStorage = iota
	PredicateStorageMain
)

type ProjectionCache 

type ProjectionCache struct {
	// contains filtered or unexported fields
}

ProjectionCache caches BuildSchemaProjection results per schema (#142): the projection depends only on the schema's attribute metadata, which is immutable after registry construction. Cached *SchemaProjection values are shared — callers must treat them as read-only. Concurrent misses on one schema are single-flighted so exactly one build runs and `misses` equals builds.

func NewProjectionCache 

func NewProjectionCache() *ProjectionCache

NewProjectionCache creates an empty projection cache.

func (*ProjectionCache) GetOrBuild 

func (c *ProjectionCache) GetOrBuild(schemaID int16, build func() (*SchemaProjection, error)) (*SchemaProjection, bool, error)

GetOrBuild returns the cached projection for schemaID, or runs build and caches a non-nil, non-error result. Concurrent callers on the same missing schema build exactly once (waiters share the result and count as hits). The second return reports a cache hit.

func (*ProjectionCache) Reset 

func (c *ProjectionCache) Reset()

Reset drops all cached projections (schema-generation invalidation hook).

func (*ProjectionCache) Stats 

func (c *ProjectionCache) Stats() (hits, misses int64)

Stats returns the cumulative hit and miss counts.

type SQLGenerator 

type SQLGenerator struct{}

SQLGenerator converts parsed conditions into SQL fragments and argument lists.

func NewSQLGenerator 

func NewSQLGenerator() *SQLGenerator

NewSQLGenerator constructs a SQLGenerator.

func (*SQLGenerator) ToSQLClauses 

func (g *SQLGenerator) ToSQLClauses(
	condition forma.Condition,
	eavTable string,
	schemaID int16,
	cache forma.SchemaAttributeCache,
	paramIndex *int,
) (string, []any, error)

ToSQLClauses builds the SQL clause and arguments for a condition tree.

func (*SQLGenerator) ToSqlClauses 

func (g *SQLGenerator) ToSqlClauses(
	condition forma.Condition,
	eavTable string,
	schemaID int16,
	cache forma.SchemaAttributeCache,
	paramIndex *int,
) (string, []any, error)

ToSqlClauses is kept for backward compatibility.

type SQLRenderer 

type SQLRenderer struct {
	// contains filtered or unexported fields
}

SQLRenderer renders text/template SQL templates while collecting parameter values and providing a safe identifier helper to avoid SQL injection.

func NewSQLRenderer 

func NewSQLRenderer() *SQLRenderer

func (*SQLRenderer) Ident 

func (r *SQLRenderer) Ident(name string) (string, error)

Ident validates a SQL identifier (table/column) and returns it quoted.

func (*SQLRenderer) Param 

func (r *SQLRenderer) Param(v any) string

Param appends a value to the renderer's args and returns a "?" placeholder to be inserted into the template.

func (*SQLRenderer) Render 

func (r *SQLRenderer) Render(tpl *template.Template, data any) (string, []any, error)

Render executes tpl with data while providing the template functions:

  • param: adds a param and returns "?" placeholder
  • ident: validates and returns a quoted identifier

It returns the rendered SQL and the collected args slice.

type SchemaProjection 

type SchemaProjection struct {
	// S3SourceSelect is the SELECT projection for the s3_source CTE:
	//   row_id, created_at, ver_ts, deleted_ts, attr_col_1, attr_col_2, ...
	S3SourceSelect string

	// PGSourceSelect is the SELECT projection for the pg_source CTE, including
	// COALESCE expressions for entity_main columns and EAV pivot expressions.
	PGSourceSelect string

	// PGGroupBy is the GROUP BY clause for the pg_source CTE.
	PGGroupBy string

	// EAVPivotSelect is the EAV subquery SELECT for the pg_source LEFT JOIN,
	// containing MAX(CASE WHEN attr_id = X THEN value_Y END) AS attr_name lines.
	EAVPivotSelect string

	// EAVPivotAttrs is the comma-separated list of attr_ids for the EAV pivot WHERE.
	EAVPivotAttrs string

	// OuterSelect is the final outer SELECT that maps unified columns back to
	// entity_main column descriptors.
	OuterSelect string

	// UnifiedColumnNames lists the canonical column names produced by both
	// s3_source and pg_source (in order).
	UnifiedColumnNames []string

	// UnifiedColumnTypes maps column name to DuckDB type for CAST expressions.
	UnifiedColumnTypes map[string]forma.ValueType

	// AttrToMainColumn maps attribute name to entity_main column name.
	AttrToMainColumn map[string]string

	// EAVAttrs lists attribute names that are EAV-only (no column binding).
	EAVAttrs []string

	// HasEAVAttrs is true if the schema has any EAV-only attributes.
	HasEAVAttrs bool
	// contains filtered or unexported fields
}

SchemaProjection holds precomputed SQL fragments for the DuckDB federated query template, derived from a schema's attribute metadata cache.

func BuildBenchmarkProjections 

func BuildBenchmarkProjections(schemaID int16) *SchemaProjection

BuildBenchmarkProjections builds a SchemaProjection with matching S3 and PG sources for a benchmark schema. Both sources produce the same columns in the same order.

func BuildSchemaProjection 

func BuildSchemaProjection(schemaID int16, cache forma.SchemaAttributeCache) (*SchemaProjection, error)

BuildSchemaProjection computes SQL fragments for the DuckDB template from the schema attribute cache. It handles both production and benchmark parquet shapes.

func (*SchemaProjection) BuildPGGroupByNoEAV 

func (sp *SchemaProjection) BuildPGGroupByNoEAV() string

BuildPGGroupByNoEAV returns the GROUP BY clause for PG source without EAV pivot.

func (*SchemaProjection) BuildPGSelectNoEAV 

func (sp *SchemaProjection) BuildPGSelectNoEAV() string

BuildPGSelectNoEAV returns a PG source SELECT that uses only entity_main columns (no EAV pivot expressions), for use when all filter/sort attributes are column-bound.

type TypedLeafEmitter 

type TypedLeafEmitter = typedLeafEmitter

TypedLeafEmitter is the exported alias external packages (the hybrid condition builder) implement to consume typed leaves via WalkHybridCondition.

Source Files

View all Source files

Directories

Path Synopsis
Package sqlgentest provides shared helpers for tests that pin the postgres_scan contract on both sides: the runtime template (internal/sqlgen) and the executable §5 sketch in docs/federated-query/design.md (internal/federated).
 Package sqlgentest provides shared helpers for tests that pin the postgres_scan contract on both sides: the runtime template (internal/sqlgen) and the executable §5 sketch in docs/federated-query/design.md (internal/federated).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.