forma

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 25, 2026 License: MIT Imports: 8 Imported by: 0

README

Forma

Forma is a general-purpose data management system built on PostgreSQL. It uses JSON Schema for data definition and a dual storage model (Hot Fields Table + EAV Table) to handle highly dynamic data structures without schema migrations.

Prerequisites

  • Go 1.26+
  • Docker or Podman, plus Docker Compose compatibility (for local PostgreSQL and S3-compatible storage)
  • Bun (for E2E test scripts)
  • k6 (for load testing; Docker fallback available)

Quick Start

# Clone and enter the project
git clone https://github.com/lychee-technology/forma.git
cd forma

# Start PostgreSQL via Docker Compose
docker compose -f deploy/docker-compose.yml up -d

# Build all binaries
make build-all

# Initialize database tables (required once)
./build/tools init-db \
  --db-host localhost \
  --db-port 5432 \
  --db-name forma \
  --db-user postgres \
  --db-password postgres \
  --db-ssl-mode disable \
  --schema-dir cmd/server/schemas

# Start the server
SCHEMA_DIR=cmd/server/schemas ./build/server

Or use the convenience script that does all of the above:

./scripts/local_server.sh

The server listens on port 8080 by default. Configure via environment variables:

Variable Default Description
DB_HOST localhost PostgreSQL host
DB_PORT 5432 PostgreSQL port
DB_NAME forma Database name
DB_USER postgres Database user
DB_PASSWORD `` Database password
DB_SSL_MODE disable SSL mode
SCHEMA_DIR `` Directory containing schema JSON files
PORT 8080 HTTP listen port

API Reference

Method Path Description
POST /api/v1/{schema} Create records (single object or array)
GET /api/v1/{schema}/{row_id} Get a single record
GET /api/v1/{schema} Query records with pagination (?page=&items_per_page=&sort_by=&sort_order=&attrs=)
PUT /api/v1/{schema}/{row_id} Update a record
DELETE /api/v1/{schema} Batch delete (JSON body: array of row_id strings)
GET /api/v1/search Cross-schema search (?schemas=&q=&page=&items_per_page=)
POST /api/v1/advanced_query Advanced query with condition DSL (JSON body)

Testing

Unit & Integration Tests
# Run all unit and integration tests
make test

# Run with coverage report
make coverage

# Run linter
make lint
Go E2E Harness (container-based)

Uses Docker or Podman through testcontainers. Validates the three-tier federated query architecture (Postgres Hot + S3 Delta/Base → DuckDB merge-on-read).

# Auto-detect Docker or Podman, configure testcontainers, and run make test.
./scripts/test_with_container_runtime.sh

# Smoke test: verify infrastructure starts
go test -v ./internal/e2e_harness/... -timeout=5m

# Full federated suite (functional + consistency + failure modes)
go test -v ./internal/e2e_harness/federated/... -tags=e2e -timeout=30m

# Performance tests only (longer timeout)
go test -v ./internal/e2e_harness/federated/... -run TestPerformance -tags=e2e -timeout=60m

The runtime helper honors DOCKER_HOST. With rootless Podman it starts the user socket at $XDG_RUNTIME_DIR/podman/podman.sock, exports the Docker-compatible endpoint, disables the Ryuk reaper, and runs make test.

Bun E2E (black-box API validation)

Requires a running Forma server and PostgreSQL.

cd tests/e2e
cp .env.example .env
bun install

# Default pipeline: register schemas → generate data → CDC flush → federated check
bun run test

# Individual steps
bun run register-schemas
bun run gen-data -- --schema all --count 10000
bun run cdc-flush
bun run federated-check

# Extended steps
bun run cdc-init          # Backfill base parquet
bun run compactor -- --all # Merge delta into base
k6 Load Testing
cd tests/e2e
bun run build-k6

bun run k6-smoke   # 5 VUs, 30s
bun run k6-full    # 30 VUs, 2m
bun run k6-perf    # 100 VUs, 5m
Benchmarks
make benchmark-smoke       # CI smoke validation
make benchmark-regression  # Small live subset
make benchmark-heavy       # Heavy planning set

Documentation

Why Forma?

Forma targets the gap between rigid RDBMS schemas and schema-less NoSQL stores:

  • Zero-Downtime Schema Evolution — Add or modify fields by updating JSON Schema metadata; no ALTER TABLE required.
  • ACID on PostgreSQL — Inherits full transactional guarantees from Postgres.
  • Smart SQL Generation — CTE + JSON_AGG eliminates N+1 queries in EAV models.
  • Federated Query (Lakehouse) — PostgreSQL for OLTP, DuckDB + Parquet on S3 for OLAP. Anti-Join + Dirty Set ensures consistency across tiers.

Documentation

Index

Constants

View Source
const PartialReasonCorruptParquetExcluded = "corrupt_parquet_excluded"

PartialReasonCorruptParquetExcluded reports a #251 partial read: one or more verification-confirmed corrupt parquet objects were excluded and the page was answered from the readable remainder plus the hot tier.

Variables

View Source
var ErrConflict = errors.New("conflict")

ErrConflict is returned when an operation would violate a uniqueness constraint or produce a duplicate record.

View Source
var ErrInvalidInput = errors.New("invalid input")

ErrInvalidInput is returned when caller-supplied input fails validation (missing required fields, unsupported values, etc.).

View Source
var ErrManifestSchemaMismatch = errors.New("manifest schema id does not match the schema being read")

ErrManifestSchemaMismatch marks a manifest object whose recorded schema ID disagrees with the schema being read. A manifest addresses one schema by path convention alone; nothing downstream re-checks it — the parquet scan does not filter rows by schema (files are per-schema by path) and the projection stamps whatever it scans as the requested schema. So a path collision between two schemas would not merely under-read, it would serve another schema's rows under this schema's identity. Not degradable: cross-schema contamination is the opposite of a partial answer.

View Source
var ErrNoParquetPaths = errors.New("no parquet paths resolved")

ErrNoParquetPaths marks a DuckDB-routed federated read whose parquet path set resolved empty: no per-request render hint, and either no configured parquet source or a source whose manifest lists no files while the fallback glob is disabled. Every query that reaches the DuckDB engine wants warm and/or cold data (hot-only requests short-circuit to Postgres first), so an empty set cannot be answered honestly. Not degradable — a Postgres-only fallback would be silently short precisely where the cold tier was requested (#299).

View Source
var ErrNotFound = errors.New("not found")

ErrNotFound is returned when a requested entity does not exist.

View Source
var ErrParquetSetInconsistent = errors.New("parquet set inconsistent with manifest")

ErrParquetSetInconsistent marks a federated read whose manifest lists parquet objects that do not exist in storage. The manifest is the authoritative record of the schema's cold/warm tier, so a listed-but-absent object means that tier has lost data. Not degradable: surfacing it even under AllowPartialDegradedMode is the whole point — degrading here would return exactly the silently short answer this classification exists to make loud (#187 scenario 2).

It lives here rather than in internal/federated for the same reason as the two errors above, plus one specific to #301: internal/httpapi must classify it to redact its object keys, and cannot import internal/federated without pulling DuckDB CGO into a pure-Go test build.

Functions

func Conflictf added in v0.2.0

func Conflictf(format string, args ...any) error

Conflictf builds a client error classified by ErrConflict. Error() renders "<message>: conflict".

func HasOperatorDetail added in v0.2.0

func HasOperatorDetail(err error) bool

HasOperatorDetail reports whether err withholds detail from its published message. internal/httpapi keys a log level on it: a disclosed 4xx whose chain still holds operator text makes the log line the only copy of that text.

func InvalidInputf added in v0.2.0

func InvalidInputf(format string, args ...any) error

InvalidInputf builds a client error classified by ErrInvalidInput whose formatted message is published verbatim. Error() renders "<message>: invalid input".

func NotFoundf added in v0.2.0

func NotFoundf(format string, args ...any) error

NotFoundf builds a client error classified by ErrNotFound. Error() renders "<message>: not found".

func ResolvePublicMessage added in v0.2.0

func ResolvePublicMessage(err error) (string, bool)

ResolvePublicMessage returns the message err deliberately published, if any. It is the canonical resolution used by the HTTP boundary, by WrapPublicf/WithOperatorDetail to qualify their input, and by the decorators' own PublicMessage delegation — one traversal, so the publication a decorator carries is exactly the one the boundary would emit.

The walk is preorder, left-first (errors.As order, so wrap prefixes accumulate outermost-wins), and each node is matched the way errors.As matches: direct implementation or the node's As(any) bool protocol (#363 review, P2). A node qualifies only when a client sentinel is reachable from that node's OWN subtree (#362/#363 reviews, P1): two gates searching the whole tree independently let a mixed tree borrow — sentinel evidence from one branch, PublicMessage() from a foreign sibling — and decorating such a tree must not manufacture the same borrow one level up. Non-qualifying nodes are stepped over, not terminal, so a real carrier behind a foreign publisher still resolves. An empty publication is treated as no publication and the walk continues past it.

func WithOperatorDetail added in v0.2.0

func WithOperatorDetail(err error, detail error) error

WithOperatorDetail attaches operator-only detail to a client error. The detail joins the chain (errors.Is/As still see it) and its text reaches Error(), so logs keep it — but it never reaches PublicMessage(). Returns err unchanged when detail is nil or err carries no PublicError, and nil for a nil err.

func WrapPublicf added in v0.2.0

func WrapPublicf(err error, format string, args ...any) error

WrapPublicf adds caller-actionable context to err, prefixing BOTH its operator message and its published message: Error() is identical to fmt.Errorf("<prefix>: %w", err) either way. When err carries no PublicError the result IS exactly that plain wrap — an operator error can never gain a publication by being wrapped. Returns nil for a nil err.

Use it only where the wrap adds identification the caller can act on (a batch index, a caller-supplied name). A layer that adds operator context should stay a plain fmt.Errorf so its prefix stays out of the body.

Types

type AttributeMetadata added in v0.0.3

type AttributeMetadata struct {
	AttributeName string    `json:"attr_name"`  // attr_name, JSON Path
	AttributeID   int16     `json:"attr_id"`    // attr_id
	ValueType     ValueType `json:"value_type"` // 'text', 'numeric', 'date', 'bool'
	// ItemsType is the element value type when ValueType is ValueTypeList.
	// Empty means text (see EffectiveItemsType).
	ItemsType      ValueType      `json:"items_type,omitempty"`
	RequiredPolicy RequiredPolicy `json:"required_policy,omitempty"`
	// Required is kept for backward compatibility with older metadata readers.
	// New code should use RequiredPolicy.
	Required      bool               `json:"required,omitempty"`
	ColumnBinding *MainColumnBinding `json:"column_binding,omitempty"`
	// Retired marks an attribute removed from the schema whose entry stays in
	// the attributes file as the attributeID ledger (#342). Retired attributes
	// are excluded from every active cache — reads skip their EAV rows (#294),
	// writes preserve them — but their (id, name, valueType) still guards
	// against rebinding the freed id to a different attribute.
	Retired bool `json:"retired,omitempty"`
}

AttributeMetadata stores cached metadata from the attributes table.

func (AttributeMetadata) EffectiveItemsType added in v0.2.0

func (m AttributeMetadata) EffectiveItemsType() ValueType

EffectiveItemsType returns the element type of a list attribute, defaulting to text when items_type is not declared.

func (AttributeMetadata) EffectiveRequiredPolicy added in v0.0.27

func (m AttributeMetadata) EffectiveRequiredPolicy() RequiredPolicy

EffectiveRequiredPolicy returns the policy used at runtime. If RequiredPolicy is unset, it falls back to legacy Required semantics.

func (AttributeMetadata) IsInsideArray added in v0.0.3

func (m AttributeMetadata) IsInsideArray() bool

IsInsideArray infers if the attribute is inside an array based on its name.

func (AttributeMetadata) Location added in v0.0.3

Location returns where the attribute is stored. If ColumnBinding is nil, the attribute is in EAV; otherwise in main table.

type AttributeStorageLocation added in v0.0.3

type AttributeStorageLocation string

AttributeStorageLocation enumerates where the attribute physically resides.

const (
	AttributeStorageLocationUnknown AttributeStorageLocation = ""
	AttributeStorageLocationMain    AttributeStorageLocation = "main"
	AttributeStorageLocationEAV     AttributeStorageLocation = "eav"
)

type BatchConfig added in v0.0.2

type BatchConfig struct {
	EnableDynamicSizing      bool `json:"enableDynamicSizing"`
	EnableParallelProcessing bool `json:"enableParallelProcessing"`
	EnableBatchStreaming     bool `json:"enableBatchStreaming"`
	ParallelThreshold        int  `json:"parallelThreshold"`
	StreamingThreshold       int  `json:"streamingThreshold"`
	MaxParallelWorkers       int  `json:"maxParallelWorkers"`
	StreamingChunkSize       int  `json:"streamingChunkSize"`
	StreamingDelay           int  `json:"streamingDelay"` // milliseconds
	MaxComplexityPerBatch    int  `json:"maxComplexityPerBatch"`
	AttributeComplexityScore int  `json:"attributeComplexityScore"`
	OptimalChunkSize         int  `json:"optimalChunkSize"`
}

BatchConfig contains batch processing settings

type BatchOperation

type BatchOperation struct {
	Operations []EntityOperation `json:"operations"`
	Atomic     bool              `json:"atomic"` // Request all-or-nothing execution; may be rejected when unsupported.
}

BatchOperation represents batch entity operations

type BatchResult

type BatchResult struct {
	Successful []*DataRecord    `json:"successful"`
	Failed     []OperationError `json:"failed"`
	TotalCount int              `json:"totalCount"`
	Duration   int64            `json:"duration"` // microseconds
}

BatchResult represents results from batch operations

type CascadeAction added in v0.0.2

type CascadeAction string

CascadeAction defines the type of cascade action

const (
	CascadeActionDelete   CascadeAction = "delete"
	CascadeActionUpdate   CascadeAction = "update"
	CascadeActionNullify  CascadeAction = "nullify"
	CascadeActionRestrict CascadeAction = "restrict"
)

type CascadeRule added in v0.0.2

type CascadeRule struct {
	SourceSchema string        `json:"sourceSchema"`
	TargetSchema string        `json:"targetSchema"`
	Action       CascadeAction `json:"action"`
	MaxDepth     int           `json:"maxDepth,omitempty"`
}

CascadeRule defines cascade behavior for specific schema relationships

type CompositeCondition

type CompositeCondition struct {
	Logic      Logic       `json:"l"`
	Conditions []Condition `json:"c"`
}

--- 3. Composite Condition (Non-Leaf Node) ---

func (*CompositeCondition) IsLeaf

func (c *CompositeCondition) IsLeaf() bool

func (*CompositeCondition) UnmarshalJSON

func (c *CompositeCondition) UnmarshalJSON(data []byte) error

UnmarshalJSON customizes decoding so that nested conditions are turned into the appropriate concrete condition implementations.

type Condition

type Condition interface {
	IsLeaf() bool
}

--- 2. Interface (The Core) ---

type Config added in v0.0.2

type Config struct {
	Database       DatabaseConfig    `json:"database"`
	Query          QueryConfig       `json:"query"`
	Entity         EntityConfig      `json:"entity"`
	Transaction    TransactionConfig `json:"transaction"`
	Performance    PerformanceConfig `json:"performance"`
	Logging        LoggingConfig     `json:"logging"`
	Metrics        MetricsConfig     `json:"metrics"`
	Reference      ReferenceConfig   `json:"reference"`
	DuckDB         DuckDBConfig      `json:"duckdb"`
	SchemaRegistry SchemaRegistry    `json:"-"` // Custom schema registry implementation (optional)
}

Config consolidates settings from both modules

func DefaultConfig added in v0.0.2

func DefaultConfig(schemaRegistry SchemaRegistry) *Config

DefaultConfig returns a default configuration

func NewConfig added in v0.0.25

func NewConfig(opts ...Option) *Config

NewConfig creates a Config starting from the defaults produced by DefaultConfig(nil) and then applies each provided Option in order. The schema registry can be set via WithSchemaRegistry.

Example:

cfg := forma.NewConfig(
    forma.WithDatabase(forma.DatabaseConfig{Host: "db.example.com", Port: 5432, MaxConnections: 50}),
    forma.WithDuckDB(forma.DuckDBConfig{Enabled: true, DBPath: ":memory:"}),
)

func (*Config) Validate added in v0.0.2

func (c *Config) Validate() error

Validate validates the configuration

type ConfigError added in v0.0.2

type ConfigError struct {
	Field   string `json:"field"`
	Message string `json:"message"`
}

ConfigError represents a configuration validation error

func (*ConfigError) Error added in v0.0.2

func (e *ConfigError) Error() string

type CrossSchemaRequest

type CrossSchemaRequest struct {
	SchemaNames  []string  `json:"schema_names" validate:"required"`
	SearchTerm   string    `json:"search_term" validate:"required"`
	Page         int       `json:"page" validate:"min=1"`
	ItemsPerPage int       `json:"items_per_page" validate:"min=1,max=100"`
	Condition    Condition `json:"-"`               // Custom unmarshal, can be CompositeCondition or KvCondition
	Attrs        []string  `json:"attrs,omitempty"` // Attributes to return (field projection)
}

CrossSchemaRequest represents a cross-schema search request

func (CrossSchemaRequest) MarshalJSON

func (r CrossSchemaRequest) MarshalJSON() ([]byte, error)

MarshalJSON implements custom JSON marshaling for CrossSchemaRequest.

func (*CrossSchemaRequest) UnmarshalJSON

func (r *CrossSchemaRequest) UnmarshalJSON(data []byte) error

UnmarshalJSON implements custom JSON unmarshaling for CrossSchemaRequest. It allows the Condition field to be either a CompositeCondition or KvCondition.

type CursorQueryResult

type CursorQueryResult struct {
	Data          []*DataRecord `json:"data"`
	NextCursor    string        `json:"next_cursor,omitempty"`
	HasMore       bool          `json:"has_more"`
	ExecutionTime time.Duration `json:"execution_time"`
}

CursorQueryResult represents cursor-based pagination results.

type DataRecord

type DataRecord struct {
	SchemaName string         `json:"schema_name"`
	RowID      uuid.UUID      `json:"row_id"`
	Attributes map[string]any `json:"attributes"`
}

type DatabaseConfig added in v0.0.2

type DatabaseConfig struct {
	Host            string        `json:"host"`
	Port            int           `json:"port"`
	Database        string        `json:"database"`
	Username        string        `json:"username"`
	Password        string        `json:"password"`
	SSLMode         string        `json:"sslMode"`
	Schema          string        `json:"schema"`
	MaxConnections  int           `json:"maxConnections"`
	MaxIdleConns    int           `json:"maxIdleConns"`
	ConnMaxLifetime time.Duration `json:"connMaxLifetime"`
	ConnMaxIdleTime time.Duration `json:"connMaxIdleTime"`
	Timeout         time.Duration `json:"timeout"`
	TableNames      TableNames    `json:"tableNames"`
}

DatabaseConfig contains database connection settings

type DuckDBConfig added in v0.0.23

type DuckDBConfig struct {
	Enabled        bool          `json:"enabled"`
	DBPath         string        `json:"dbPath"`        // path to local DuckDB file (or ":memory:")
	MemoryLimitMB  int           `json:"memoryLimitMB"` // memory limit for DuckDB in MB
	EnableS3       bool          `json:"enableS3"`      // enable S3/http file system
	S3Endpoint     string        `json:"s3Endpoint"`    // custom S3 endpoint (for MinIO)
	S3AccessKey    string        `json:"s3AccessKey"`
	S3SecretKey    string        `json:"s3SecretKey"`
	S3Region       string        `json:"s3Region"`
	EnableParquet  bool          `json:"enableParquet"` // enable parquet extension
	Extensions     []string      `json:"extensions"`    // additional extensions to load
	MaxConnections int           `json:"maxConnections"`
	QueryTimeout   time.Duration `json:"queryTimeout"`   // per-query timeout for DuckDB access
	MaxParallelism int           `json:"maxParallelism"` // max threads/pragmas for DuckDB
	// Deprecated: ignored failure-rate threshold; use
	// CircuitBreakerFailureThreshold instead.
	CircuitBreakerThreshold float64 `json:"circuitBreakerThreshold"`

	// CircuitBreakerFailureThreshold is the number of consecutive failures that
	// opens the DuckDB circuit breaker. Zero means use the built-in default.
	CircuitBreakerFailureThreshold int `json:"circuitBreakerFailureThreshold"`
	// CircuitBreakerWindow is the sliding window for counting DuckDB failures.
	// Zero means use the built-in default.
	CircuitBreakerWindow time.Duration `json:"circuitBreakerWindow"`
	// CircuitBreakerOpenDuration is how long the DuckDB breaker stays open after
	// tripping. Zero means use the built-in default.
	CircuitBreakerOpenDuration time.Duration `json:"circuitBreakerOpenDuration"`

	// FlushVisibilityGraceMs is the #252 clock-skew margin for the federated
	// dirty barrier. Each query anchors its cutoff at the instant it resolved
	// its parquet path set: rows marked flushed after that instant (their
	// delta may be missing from the resolved set) stay hot-readable, minus
	// this margin to absorb CDC-host vs query-host clock skew. Zero (the
	// default) is the exact anchor — the steady state is never widened; a
	// positive value hot-serves rows flushed up to that long before the
	// query; a negative value disables the widening (the pre-#252 barrier).
	FlushVisibilityGraceMs int64 `json:"flushVisibilityGraceMs"`

	// S3Bucket is the bucket holding the lakehouse parquet files and the
	// manifests that index them. Required whenever ManifestTemplate is set.
	S3Bucket string `json:"s3Bucket"`
	// S3DataPrefix mirrors the CDC write side's S3Prefix (the prefix under
	// which delta/base parquet files are written). It is used *only* for the
	// legacy glob fallback covering schemas that have never been flushed and
	// therefore have no manifest object yet. Empty disables that fallback: a
	// schema without a manifest then resolves to zero parquet paths, and a
	// DuckDB-routed read of it fails fast with ErrNoParquetPaths — a
	// non-degradable read-path error naming the schema, distinct from the
	// transient failures AllowPartialDegradedMode absorbs (#299). Leave this
	// set unless every schema is known to be manifested.
	S3DataPrefix string `json:"s3DataPrefix"`
	// ManifestPrefix is the root prefix for manifest objects in S3. It must
	// match the CDC/compaction write side's ManifestPrefix.
	ManifestPrefix string `json:"manifestPrefix"`
	// ManifestTemplate is the manifest path template (e.g.
	// "manifest/{{.SchemaID}}.json") and doubles as the enable gate for
	// manifest-driven parquet resolution. It must be identical to the write
	// side's template: a writer-on/reader-off mismatch makes the reader miss
	// the cold tier entirely and lose those rows silently — no error is
	// raised, the result set is merely short. Empty (the default) keeps the
	// pre-existing glob-based read path.
	ManifestTemplate string `json:"manifestTemplate"`

	// AllowCallerParquetPaths opts the deployment into honoring the
	// per-request federated.s3_parquet_path_template field (#456). It defaults
	// to false: the field is a caller-controlled scan target, so on every
	// existing deployment a request carrying it is rejected. When true, a
	// caller template is still honored only for paths inside the configured
	// S3Bucket (see the engine's path resolver) — the flag is necessary, not
	// sufficient.
	AllowCallerParquetPaths bool `json:"allowCallerParquetPaths"`

	Routing RoutingPolicy `json:"routing"` // routing policy for federated queries
}

DuckDBConfig contains DuckDB connection and S3 settings for federated queries

func (DuckDBConfig) ManifestReadEnabled added in v0.2.0

func (d DuckDBConfig) ManifestReadEnabled() bool

ManifestReadEnabled reports whether manifest-driven parquet path resolution is configured. ManifestTemplate is the single enable gate.

The TrimSpace here is defense-in-depth, not normalization: a padded template is rejected outright by ValidateManifestRead (byte parity with the untrimmed write side is the contract), so no validated config ever reaches this method with surrounding whitespace. The trim only guards direct programmatic misuse — a caller that builds a DuckDBConfig by hand and skips validation — where reading a whitespace-only template as "manifest reads are on" would be strictly worse.

func (DuckDBConfig) ValidateCallerParquetPaths added in v0.2.0

func (d DuckDBConfig) ValidateCallerParquetPaths() error

ValidateCallerParquetPaths rejects the #456 caller-path opt-in when no bucket is configured to scope hints against. Honoring federated.s3_parquet_path_template requires a non-empty S3Bucket: validateHintPathScope would otherwise reject every such request at run time, turning an operator misconfiguration into invalid caller input. Like ValidateManifestRead, validation is independent of Enabled — whether the setting takes effect is the factory's gate, but an incoherent combination is always a configuration error. Both cmd/server and the factory call this beside ValidateManifestRead so the failure lands at startup, before any I/O, rather than on the first hint-bearing request.

func (DuckDBConfig) ValidateManifestRead added in v0.2.0

func (d DuckDBConfig) ValidateManifestRead() error

ValidateManifestRead validates the manifest read surface. It returns a *ConfigError naming the offending field, or nil when the combination is coherent. Validation is independent of Enabled: whether the settings take effect is the factory's gate, but an incoherent combination is always a configuration error.

type EntityBatchCreator added in v0.0.24

type EntityBatchCreator interface {
	BatchCreate(ctx context.Context, req *BatchOperation) (*BatchResult, error)
}

type EntityBatchOperator added in v0.0.24

type EntityBatchOperator interface {
	BatchCreate(ctx context.Context, req *BatchOperation) (*BatchResult, error)
	BatchUpdate(ctx context.Context, req *BatchOperation) (*BatchResult, error)
	BatchDelete(ctx context.Context, req *BatchOperation) (*BatchResult, error)
}

type EntityConfig added in v0.0.2

type EntityConfig struct {
	EnableReferenceValidation bool          `json:"enableReferenceValidation"`
	EnableCascadeDelete       bool          `json:"enableCascadeDelete"`
	BatchSize                 int           `json:"batchSize"`
	CacheEnabled              bool          `json:"cacheEnabled"`
	CacheTTL                  time.Duration `json:"cacheTTL"`
	MaxEntitySize             int           `json:"maxEntitySize"`
	EnableVersioning          bool          `json:"enableVersioning"`
	SchemaDirectory           string        `json:"schemaDirectory"`

	// ValidateUpdatesStrict makes update payloads that violate the entity's
	// JSON Schema fail with 4xx instead of only being logged (#314).
	//
	// Default false: rows written before schema enforcement may already violate
	// their schema, and rejecting on update would make them un-updatable — a
	// caller touching one unrelated field would be refused for a pre-existing
	// violation elsewhere. Creates are always enforced; they have no legacy data.
	//
	// Whether it is safe to flip this yet is answered by the #317 aggregate:
	// the "report-only schema validation violations reached a milestone" Warn
	// line (per schema, at the 1st and every 100th accepted violation) and the
	// entity_report_only_validation_violation_total telemetry counter. While
	// those lines keep appearing for a schema, its rows are not yet repaired.
	// Their absence is not proof of the converse: the signal fires only when a
	// violating row is written, and only every 100th time thereafter within one
	// process (strictly, per EntityManager — the shipped wiring builds exactly
	// one), so a schema whose bad rows are never updated stays silent.
	// Confirm with an e2e pass over real data before flipping
	// (docs/error-handling.md).
	ValidateUpdatesStrict bool `json:"validateUpdatesStrict"`
}

EntityConfig contains entity management settings

type EntityIdentifier

type EntityIdentifier struct {
	SchemaName string    `json:"schemaName"`
	RowID      uuid.UUID `json:"rowId"`
}

EntityIdentifier identifies an entity for operations

type EntityManager

type EntityManager interface {
	EntityWriter
	EntityReader
	EntityBatchOperator
	Close() error
}

EntityManager provides comprehensive entity and query operations.

Close releases resources the manager owns — for factory-built managers that is the embedded DuckDB client (#302). It is safe for concurrent use: teardown runs exactly once and every call returns the same result, so a failed close stays observable to late callers. Managers constructed without owned resources return nil. Embedders must Close the manager when done with it or the DuckDB instance lives until process exit.

type EntityOperation

type EntityOperation struct {
	EntityIdentifier
	Type    OperationType  `json:"type"`
	Data    map[string]any `json:"data,omitempty"`
	Updates map[string]any `json:"updates,omitempty"`
}

EntityOperation represents CRUD operations

type EntityReader added in v0.0.24

type EntityReader interface {
	Get(ctx context.Context, req *QueryRequest) (*DataRecord, error)
	Query(ctx context.Context, req *QueryRequest) (*QueryResult, error)
	CrossSchemaSearch(ctx context.Context, req *CrossSchemaRequest) (*QueryResult, error)
}

type EntityUpdate

type EntityUpdate struct {
	EntityIdentifier
	Updates any `json:"updates"`
}

EntityUpdate represents an update operation

type EntityWriter added in v0.0.24

type EntityWriter interface {
	Create(ctx context.Context, req *EntityOperation) (*DataRecord, error)
	Update(ctx context.Context, req *EntityOperation) (*DataRecord, error)
	Delete(ctx context.Context, req *EntityOperation) error
}

type ExecutionMerge added in v0.2.0

type ExecutionMerge struct {
	Strategy   string   `json:"strategy,omitempty"`
	PreferHot  bool     `json:"prefer_hot,omitempty"`
	DedupKeys  []string `json:"dedup_keys,omitempty"`
	DurationMs int64    `json:"duration_ms,omitempty"`
}

ExecutionMerge describes the tier-merge strategy the plan applied.

type ExecutionPlan added in v0.2.0

type ExecutionPlan struct {
	Routing ExecutionRouting  `json:"routing"`
	Sources []ExecutionSource `json:"sources,omitempty"`
	Merge   *ExecutionMerge   `json:"merge,omitempty"`
	Timings map[string]int64  `json:"timings,omitempty"`
}

ExecutionPlan is the JSON-serializable projection of the engine's internal execution plan surfaced on QueryResult. It carries only safe routing/tier metadata an external caller needs to understand the route.

SECURITY: the internal plan's SQL, bind params, and free-text notes are deliberately NOT projected here. Since #306, the database password is redacted at the source (plan SQL and failure notes carry password=***REDACTED***), but the rendered DuckDB SQL still embeds host/user/dbname and table internals; Params carry query arguments, and notes carry storage keys and engine internals. Exposing them on the HTTP API would leak internals to any caller of advanced_query. Only enum/numeric/static-string fields are surfaced. It also imports nothing so it can live in the public API.

type ExecutionRouting added in v0.2.0

type ExecutionRouting struct {
	UsedDuckDB bool     `json:"used_duckdb"`
	Tiers      []string `json:"tiers,omitempty"`
	Reason     string   `json:"reason,omitempty"`
}

ExecutionRouting reports the routing decision the engine committed to.

type ExecutionSource added in v0.2.0

type ExecutionSource struct {
	Tier              string `json:"tier"`
	Engine            string `json:"engine,omitempty"`
	RowEstimate       int64  `json:"row_estimate,omitempty"`
	ActualRows        int64  `json:"actual_rows,omitempty"`
	PredicatePushdown bool   `json:"predicate_pushdown,omitempty"`
	DurationMs        int64  `json:"duration_ms,omitempty"`
	Reason            string `json:"reason,omitempty"`
}

ExecutionSource describes one physical data source (tier) the plan read from. It intentionally omits the raw SQL and bind params (see ExecutionPlan).

type FederatedQueryRequest added in v0.1.0

type FederatedQueryRequest struct {
	Enabled                  bool     `json:"enabled,omitempty"`
	PreferredTiers           []string `json:"preferred_tiers,omitempty"`
	PreferHot                bool     `json:"prefer_hot,omitempty"`
	UseMainAsAnchor          bool     `json:"use_main_as_anchor,omitempty"`
	S3ParquetPathTemplate    string   `json:"s3_parquet_path_template,omitempty"`
	AllowPartialDegradedMode bool     `json:"allow_partial_degraded_mode,omitempty"`
	IncludeExecutionPlan     bool     `json:"include_execution_plan,omitempty"`
	ConsistencyMode          string   `json:"consistency_mode,omitempty"`
}

FederatedQueryRequest carries optional hints for routing QueryRequest through the federated repository path while preserving the normal API surface for callers that do not need DuckDB/S3-backed reads.

type FilterField

type FilterField string
const (
	FilterFieldAttributeName FilterField = "attr_name"
	FilterFieldValueText     FilterField = "value_text"
	FilterFieldValueNumeric  FilterField = "value_numeric"
	FilterFieldRowID         FilterField = "row_id"
	FilterFieldSchemaName    FilterField = "schema_name"
)

type FilterType

type FilterType string

FilterType defines supported filter operations

const (
	FilterEquals      FilterType = "equals"
	FilterNotEquals   FilterType = "not_equals"
	FilterStartsWith  FilterType = "starts_with"
	FilterContains    FilterType = "contains"
	FilterGreaterThan FilterType = "gt"
	FilterLessThan    FilterType = "lt"
	FilterGreaterEq   FilterType = "gte"
	FilterLessEq      FilterType = "lte"
	FilterIn          FilterType = "in"
	FilterNotIn       FilterType = "not_in"
)

type JSONSchema added in v0.0.15

type JSONSchema struct {
	ID         int16                      `json:"id"`
	Name       string                     `json:"name"`
	Version    int                        `json:"version"`
	Schema     string                     `json:"schema"`
	Properties map[string]*PropertySchema `json:"properties"`
	Required   []string                   `json:"required"`
	CreatedAt  int64                      `json:"created_at"`
}

JSONSchema represents a schema definition.

type KvCondition

type KvCondition struct {
	Attr  string `json:"a"`
	Value string `json:"v"`
}

--- 4. KvCondition (Leaf Node) ---

func (*KvCondition) IsLeaf

func (kv *KvCondition) IsLeaf() bool

func (*KvCondition) UnmarshalJSON

func (kv *KvCondition) UnmarshalJSON(data []byte) error

UnmarshalJSON ensures short-hand keys are present.

type LoggingConfig added in v0.0.2

type LoggingConfig struct {
	Level                  string        `json:"level"`
	Format                 string        `json:"format"`
	EnableStructured       bool          `json:"enableStructured"`
	EnablePerformance      bool          `json:"enablePerformance"`
	EnableQueryLogging     bool          `json:"enableQueryLogging"`
	LogSlowQueries         bool          `json:"logSlowQueries"`
	SlowQueryThreshold     time.Duration `json:"slowQueryThreshold"`
	MaxLogSize             int           `json:"maxLogSize"`
	LogRotation            bool          `json:"logRotation"`
	SanitizeParameters     bool          `json:"sanitizeParameters"`
	LogQueries             bool          `json:"logQueries"`
	LogErrors              bool          `json:"logErrors"`
	LogSecurityEvents      bool          `json:"logSecurityEvents"`
	LogPerformanceWarnings bool          `json:"logPerformanceWarnings"`
	LogAllOperations       bool          `json:"logAllOperations"`
	EnableDetailedLogging  bool          `json:"enableDetailedLogging"`
}

LoggingConfig contains logging settings

type Logic

type Logic string
const (
	LogicAnd Logic = "and"
	LogicOr  Logic = "or"
)

type MainColumn added in v0.0.3

type MainColumn string

MainColumn represents column names in the main entity table.

const (
	MainColumnText01     MainColumn = "text_01"
	MainColumnText02     MainColumn = "text_02"
	MainColumnText03     MainColumn = "text_03"
	MainColumnText04     MainColumn = "text_04"
	MainColumnText05     MainColumn = "text_05"
	MainColumnText06     MainColumn = "text_06"
	MainColumnText07     MainColumn = "text_07"
	MainColumnText08     MainColumn = "text_08"
	MainColumnText09     MainColumn = "text_09"
	MainColumnText10     MainColumn = "text_10"
	MainColumnSmallint01 MainColumn = "smallint_01"
	MainColumnSmallint02 MainColumn = "smallint_02"
	MainColumnInteger01  MainColumn = "integer_01"
	MainColumnInteger02  MainColumn = "integer_02"
	MainColumnInteger03  MainColumn = "integer_03"
	MainColumnBigint01   MainColumn = "bigint_01"
	MainColumnBigint02   MainColumn = "bigint_02"
	MainColumnBigint03   MainColumn = "bigint_03"
	MainColumnBigint04   MainColumn = "bigint_04"
	MainColumnBigint05   MainColumn = "bigint_05"
	MainColumnDouble01   MainColumn = "double_01"
	MainColumnDouble02   MainColumn = "double_02"
	MainColumnDouble03   MainColumn = "double_03"
	MainColumnDouble04   MainColumn = "double_04"
	MainColumnDouble05   MainColumn = "double_05"
	MainColumnUUID01     MainColumn = "uuid_01"
	MainColumnUUID02     MainColumn = "uuid_02"
	MainColumnCreatedAt  MainColumn = "ltbase_created_at"
	MainColumnUpdatedAt  MainColumn = "ltbase_updated_at"
	MainColumnDeletedAt  MainColumn = "ltbase_deleted_at"
	MainColumnCreatedBy  MainColumn = "ltbase_created_by"
	MainColumnUpdatedBy  MainColumn = "ltbase_updated_by"
	MainColumnDeletedBy  MainColumn = "ltbase_deleted_by"
	MainColumnSchemaID   MainColumn = "ltbase_schema_id"
	MainColumnRowID      MainColumn = "ltbase_row_id"
)

type MainColumnBinding added in v0.0.3

type MainColumnBinding struct {
	ColumnName MainColumn         `json:"col_name"`
	Encoding   MainColumnEncoding `json:"encoding,omitempty"`
}

MainColumnBinding describes how a schema attribute maps into a hot attribute column.

func (*MainColumnBinding) ColumnType added in v0.0.3

func (m *MainColumnBinding) ColumnType() MainColumnType

ColumnType derives the column type from the column name prefix.

type MainColumnEncoding added in v0.0.3

type MainColumnEncoding string

MainColumnEncoding represents special encoding for main column values.

const (
	MainColumnEncodingDefault  MainColumnEncoding = "default"
	MainColumnEncodingBoolText MainColumnEncoding = "bool_text" // "1"/"0" string in a text column
	MainColumnEncodingUnixMs   MainColumnEncoding = "unix_ms"
	MainColumnEncodingBoolInt  MainColumnEncoding = "bool_smallint"
	MainColumnEncodingISO8601  MainColumnEncoding = "iso8601"
)

type MainColumnType added in v0.0.3

type MainColumnType string

MainColumnType represents the data type of a main column.

const (
	MainColumnTypeText     MainColumnType = "text"
	MainColumnTypeSmallint MainColumnType = "smallint"
	MainColumnTypeInteger  MainColumnType = "integer"
	MainColumnTypeBigint   MainColumnType = "bigint"
	MainColumnTypeDouble   MainColumnType = "double"
	MainColumnTypeUUID     MainColumnType = "uuid"
)

type ManifestSchemaMismatchError added in v0.2.0

type ManifestSchemaMismatchError struct {
	RequestedSchemaID int16
	ManifestSchemaID  int16
	Path              string
}

ManifestSchemaMismatchError names both schema IDs and the manifest object, so an operator can see which template or object is misaddressed.

func (*ManifestSchemaMismatchError) Error added in v0.2.0

func (*ManifestSchemaMismatchError) Unwrap added in v0.2.0

func (e *ManifestSchemaMismatchError) Unwrap() error

type MetricsConfig added in v0.0.2

type MetricsConfig struct {
	Enabled                  bool              `json:"enabled"`
	Provider                 string            `json:"provider"` // prometheus, statsd, etc.
	Endpoint                 string            `json:"endpoint"`
	CollectionInterval       time.Duration     `json:"collectionInterval"`
	EnableHistograms         bool              `json:"enableHistograms"`
	EnableCounters           bool              `json:"enableCounters"`
	EnableGauges             bool              `json:"enableGauges"`
	Namespace                string            `json:"namespace"`
	Labels                   map[string]string `json:"labels"`
	MaxSamples               int               `json:"maxSamples"`
	EnableOperationMetrics   bool              `json:"enableOperationMetrics"`
	EnableTransactionMetrics bool              `json:"enableTransactionMetrics"`
	EnablePatternMetrics     bool              `json:"enablePatternMetrics"`
}

MetricsConfig contains metrics collection settings

type NoParquetPathsError added in v0.2.0

type NoParquetPathsError struct {
	SchemaID int16
	// SourceConfigured reports whether a parquet source was consulted and
	// returned nothing, as opposed to no source existing to consult.
	SourceConfigured bool
}

NoParquetPathsError names the schema whose path set came back empty and which resolution level was in play, because the two states have different remedies: a consulted-but-empty source needs its manifest repaired (or the fallback prefix set), while an absent source needs configuring at all.

func (*NoParquetPathsError) Error added in v0.2.0

func (e *NoParquetPathsError) Error() string

func (*NoParquetPathsError) Unwrap added in v0.2.0

func (e *NoParquetPathsError) Unwrap() error

type OperationError

type OperationError struct {
	Operation EntityOperation `json:"operation"`
	Error     string          `json:"error"`
	Code      string          `json:"code"`
	Details   map[string]any  `json:"details,omitempty"`
}

OperationError represents an error for a specific operation

type OperationType

type OperationType string

OperationType represents CRUD operations

const (
	OperationCreate OperationType = "create"
	OperationRead   OperationType = "read"
	OperationUpdate OperationType = "update"
	OperationDelete OperationType = "delete"
	OperationQuery  OperationType = "query"
)

type Option added in v0.0.25

type Option func(*Config)

Option is a functional option that mutates a Config. Use the With* constructors below to build option values, then pass them to NewConfig to obtain a fully-configured Config without touching the struct fields directly.

func WithDatabase added in v0.0.25

func WithDatabase(db DatabaseConfig) Option

WithDatabase replaces the DatabaseConfig section.

func WithDuckDB added in v0.0.25

func WithDuckDB(d DuckDBConfig) Option

WithDuckDB replaces the DuckDBConfig section.

func WithEntity added in v0.0.25

func WithEntity(e EntityConfig) Option

WithEntity replaces the EntityConfig section.

func WithLogging added in v0.0.25

func WithLogging(l LoggingConfig) Option

WithLogging replaces the LoggingConfig section.

func WithMetrics added in v0.0.25

func WithMetrics(m MetricsConfig) Option

WithMetrics replaces the MetricsConfig section.

func WithPerformance added in v0.0.25

func WithPerformance(p PerformanceConfig) Option

WithPerformance replaces the PerformanceConfig section.

func WithQuery added in v0.0.25

func WithQuery(q QueryConfig) Option

WithQuery replaces the QueryConfig section.

func WithReference added in v0.0.25

func WithReference(r ReferenceConfig) Option

WithReference replaces the ReferenceConfig section.

func WithSchemaRegistry added in v0.0.25

func WithSchemaRegistry(sr SchemaRegistry) Option

WithSchemaRegistry sets the schema registry on the config.

func WithTransaction added in v0.0.25

func WithTransaction(t TransactionConfig) Option

WithTransaction replaces the TransactionConfig section.

type OrderBy

type OrderBy struct {
	Attribute string    `json:"attribute"`
	SortOrder SortOrder `json:"sort_order,omitempty"`
}

type ParquetSetInconsistentError added in v0.2.0

type ParquetSetInconsistentError struct {
	SchemaID    int16
	MissingKeys []string
}

ParquetSetInconsistentError carries the schema and the missing object keys so the message names the offending state, per the read-path error style.

MissingKeys holds bucket-relative S3 object keys — operator detail that must not cross a public transport. internal/httpapi redacts it from response bodies (#301). That redaction is gated on sentinel evidence, not on the status: this error wraps no client sentinel, so it is redacted whatever status it is classified as — including a 4xx, since DuckDB renders a missing object as "404 (Not Found)". Any new transport owes the same treatment.

func (*ParquetSetInconsistentError) Error added in v0.2.0

func (*ParquetSetInconsistentError) Unwrap added in v0.2.0

func (e *ParquetSetInconsistentError) Unwrap() error

type PartialResultInfo added in v0.2.0

type PartialResultInfo struct {
	Reason              string `json:"reason"`
	ExcludedObjectCount int    `json:"excluded_object_count,omitempty"`
}

PartialResultInfo marks a QueryResult whose page was answered from a deliberately reduced data surface. It is the sanctioned HTTP-visible partial signal (#348): the #251 corrupt-parquet exclusion note lives in internal plan Notes, which never cross the HTTP boundary (#301/#306), so without this field an API consumer sees fewer rows and a smaller total_records with no explanation. Reason is a closed enum, not free text, and the excluded objects are identified only by count — storage keys stay internal. It is deliberately NOT Routing.Reason: the route does not change on a partial read.

type PerformanceConfig added in v0.0.2

type PerformanceConfig struct {
	EnableMonitoring          bool          `json:"enableMonitoring"`
	SlowQueryThreshold        time.Duration `json:"slowQueryThreshold"`
	SlowOperationThreshold    time.Duration `json:"slowOperationThreshold"`
	MetricsCollectionInterval time.Duration `json:"metricsCollectionInterval"`
	BatchSize                 int           `json:"batchSize"`
	MaxBatchSize              int           `json:"maxBatchSize"`
	Batch                     BatchConfig   `json:"batch"`

	// Unified monitoring settings
	MaxMetricsHistory      int           `json:"maxMetricsHistory"`
	MaxAlertsHistory       int           `json:"maxAlertsHistory"`
	MaxRecommendations     int           `json:"maxRecommendations"`
	EnableAlerting         bool          `json:"enableAlerting"`
	EnableRecommendations  bool          `json:"enableRecommendations"`
	AlertingInterval       time.Duration `json:"alertingInterval"`
	RecommendationInterval time.Duration `json:"recommendationInterval"`

	// Memory monitoring
	EnableMemoryMonitoring bool  `json:"enableMemoryMonitoring"`
	MemoryThreshold        int64 `json:"memoryThreshold"`

	// Correlation tracking
	EnableCorrelationTracking bool          `json:"enableCorrelationTracking"`
	CorrelationTTL            time.Duration `json:"correlationTTL"`
}

PerformanceConfig contains performance monitoring settings

type PropertySchema added in v0.0.15

type PropertySchema struct {
	Name       string                     `json:"name"`
	Type       string                     `json:"type"` // "string", "integer", "number", "boolean", "array", "object", "null"
	Format     string                     `json:"format,omitempty"`
	Items      *PropertySchema            `json:"items,omitempty"`
	Properties map[string]*PropertySchema `json:"properties,omitempty"`
	Required   bool                       `json:"required"`
	Default    any                        `json:"default,omitempty"`
	Enum       []any                      `json:"enum,omitempty"`
	Minimum    *float64                   `json:"minimum,omitempty"`
	Maximum    *float64                   `json:"maximum,omitempty"`
	MinLength  *int                       `json:"minLength,omitempty"`
	MaxLength  *int                       `json:"maxLength,omitempty"`
	Pattern    string                     `json:"pattern,omitempty"`
	Relation   *RelationSchema            `json:"x-relation,omitempty"`
	LTBaseType string                     `json:"x-ltbase-type,omitempty"`      // "virtual" for virtual fields that are populated dynamically
	LTBaseNote string                     `json:"x-ltbase-note-prop,omitempty"` // Reference to note field: "${note_id}", "${owner_id}", "${note_data}"
}

PropertySchema defines the schema for a single property.

type PublicError added in v0.2.0

type PublicError interface {
	error
	PublicMessage() string
}

PublicError is implemented by an error that carries a message deliberately published to API clients.

type QueryConfig added in v0.0.2

type QueryConfig struct {
	DefaultTimeout     time.Duration `json:"defaultTimeout"`
	MaxRows            int           `json:"maxRows"`
	DefaultPageSize    int           `json:"defaultPageSize"`
	MaxPageSize        int           `json:"maxPageSize"`
	EnableQueryPlan    bool          `json:"enableQueryPlan"`
	EnableOptimization bool          `json:"enableOptimization"`
	CacheQueryPlans    bool          `json:"cacheQueryPlans"`
	QueryPlanCacheTTL  time.Duration `json:"queryPlanCacheTTL"`
}

QueryConfig contains query execution settings

type QueryRequest

type QueryRequest struct {
	SchemaName   string    `json:"schema_name" validate:"required"`
	Page         int       `json:"page" validate:"min=1"`
	ItemsPerPage int       `json:"items_per_page" validate:"min=1,max=100"`
	Condition    Condition `json:"-"` // Custom unmarshal, can be CompositeCondition or KvCondition
	SortBy       []string  `json:"sort_by,omitempty"`
	SortOrder    SortOrder `json:"sort_order,omitempty"`
	// Sort carries per-key sort directions (#240). Mutually exclusive with
	// SortBy/SortOrder; an entry's empty SortOrder defaults to asc.
	Sort      []OrderBy              `json:"sort,omitempty"`
	RowID     *uuid.UUID             `json:"row_id,omitempty"` // For entity-specific operations
	Attrs     []string               `json:"attrs,omitempty"`  // Attributes to return (field projection)
	Federated *FederatedQueryRequest `json:"federated,omitempty"`
}

QueryRequest represents a pagination query request.

func (QueryRequest) MarshalJSON

func (r QueryRequest) MarshalJSON() ([]byte, error)

MarshalJSON implements custom JSON marshaling for QueryRequest.

func (*QueryRequest) UnmarshalJSON

func (r *QueryRequest) UnmarshalJSON(data []byte) error

UnmarshalJSON implements custom JSON unmarshaling for QueryRequest. It allows the Condition field to be either a CompositeCondition or KvCondition.

type QueryResult

type QueryResult struct {
	Data          []*DataRecord `json:"data"`
	TotalRecords  int           `json:"total_records"`
	TotalPages    int           `json:"total_pages"`
	CurrentPage   int           `json:"current_page"`
	ItemsPerPage  int           `json:"items_per_page"`
	HasNext       bool          `json:"has_next"`
	HasPrevious   bool          `json:"has_previous"`
	ExecutionTime time.Duration `json:"execution_time"`
	// ExecutionPlan is populated only for federated requests that set
	// federated.include_execution_plan; it reports the route the engine
	// actually took (DuckDB vs Postgres-only) and per-tier sources so callers
	// can distinguish federated reads from hot-path reads without guessing.
	ExecutionPlan *ExecutionPlan `json:"execution_plan,omitempty"`
	// Partial marks a response answered from an incomplete data surface
	// (#348); currently the only reason is the #251 corrupt-parquet
	// exclusion. Nil/omitted for complete answers. Unlike ExecutionPlan it
	// does not require federated.include_execution_plan.
	Partial *PartialResultInfo `json:"partial,omitempty"`
}

QueryResult represents paginated query results.

type Reference

type Reference struct {
	SourceSchemaName string        `json:"sourceSchemaName"`
	SourceRowID      uuid.UUID     `json:"sourceRowId"`
	SourceFieldName  string        `json:"sourceFieldName"`
	TargetSchemaName string        `json:"targetSchemaName"`
	TargetRowID      uuid.UUID     `json:"targetRowId"`
	ReferenceType    ReferenceType `json:"referenceType"`
}

Reference represents a reference from one entity to another

type ReferenceConfig added in v0.0.2

type ReferenceConfig struct {
	ValidateOnCreate bool                   `json:"validateOnCreate"`
	ValidateOnUpdate bool                   `json:"validateOnUpdate"`
	CheckIntegrity   bool                   `json:"checkIntegrity"`
	CascadeDelete    bool                   `json:"cascadeDelete"`
	CascadeUpdate    bool                   `json:"cascadeUpdate"`
	MaxCascadeDepth  int                    `json:"maxCascadeDepth"`
	CascadeRules     map[string]CascadeRule `json:"cascadeRules,omitempty"`
	EnableCaching    bool                   `json:"enableCaching"`
	CacheTTL         time.Duration          `json:"cacheTTL"`
	MaxCacheSize     int                    `json:"maxCacheSize"`
	BatchSize        int                    `json:"batchSize"`
}

ReferenceConfig contains reference management settings

type ReferenceType

type ReferenceType string

ReferenceType represents the type of reference

const (
	ReferenceTypeSingle ReferenceType = "single"
	ReferenceTypeArray  ReferenceType = "array"
	ReferenceTypeNested ReferenceType = "nested"
)

type RelationSchema added in v0.0.15

type RelationSchema struct {
	Target      string `json:"target"`       // Target schema name
	Type        string `json:"type"`         // "reference" for foreign key relationships
	KeyProperty string `json:"key_property"` // child-side foreign key attribute
}

RelationSchema defines reference relationships between objects.

type RequiredPolicy added in v0.0.27

type RequiredPolicy string

RequiredPolicy defines when an attribute is required.

const (
	RequiredPolicyOptional        RequiredPolicy = "optional"
	RequiredPolicyAlways          RequiredPolicy = "required_always"
	RequiredPolicyIfParentPresent RequiredPolicy = "required_if_parent_present"
)

type RoutingPolicy added in v0.0.23

type RoutingPolicy struct {
	Strategy          RoutingStrategy `json:"strategy"`          // "freshness-first", "cost-first", "hybrid"
	HotTTL            time.Duration   `json:"hotTTL"`            // TTL to consider data "hot"
	MaxDuckDBScanRows int             `json:"maxDuckDBScanRows"` // threshold for preferring cold scans
	AllowS3Fallback   bool            `json:"allowS3Fallback"`   // allow falling back to S3/DuckDB when PG not used
}

RoutingPolicy defines federated query routing behavior

type RoutingStrategy added in v0.0.25

type RoutingStrategy string

RoutingStrategy specifies the federated query routing algorithm.

const (
	// RoutingStrategyFreshnessFirst prefers the hot (PostgreSQL) tier for
	// queries that explicitly request fresh data (PreferHot flag).
	RoutingStrategyFreshnessFirst RoutingStrategy = "freshness-first"

	// RoutingStrategyCostFirst routes large scans to DuckDB to reduce
	// PostgreSQL load; small scans stay on the hot tier.
	RoutingStrategyCostFirst RoutingStrategy = "cost-first"

	// RoutingStrategyHybrid uses DuckDB by default but short-circuits to
	// PostgreSQL when the result set is expected to be small or hot data is
	// explicitly preferred.
	RoutingStrategyHybrid RoutingStrategy = "hybrid"
)

type SchemaAttributeCache added in v0.0.3

type SchemaAttributeCache map[string]AttributeMetadata

SchemaAttributeCache is a mapping of attr_name -> metadata. Strongly recommended to populate per schema_id at application startup.

type SchemaRegistry added in v0.0.3

type SchemaRegistry interface {
	// GetSchemaAttributeCacheByName retrieves schema ID and attribute cache by schema name
	GetSchemaAttributeCacheByName(name string) (int16, SchemaAttributeCache, error)
	// GetSchemaAttributeCacheByID retrieves schema name and attribute cache by schema ID
	GetSchemaAttributeCacheByID(id int16) (string, SchemaAttributeCache, error)

	GetSchemaByName(name string) (int16, JSONSchema, error)
	// GetSchemaAttributeCacheByID retrieves schema name and attribute cache by schema ID
	GetSchemaByID(id int16) (string, JSONSchema, error)
	ListSchemas() []string
}

SchemaRegistry provides schema lookup operations. Implementations can load schemas from files, databases, or other sources.

type SortOrder

type SortOrder string

SortOrder defines sort direction

const (
	SortOrderAsc  SortOrder = "asc"
	SortOrderDesc SortOrder = "desc"
)

type TableNames added in v0.0.2

type TableNames struct {
	SchemaRegistry string `json:"schemaRegistry"`
	EntityMain     string `json:"entityMain"`
	EAVData        string `json:"eavData"`
	ChangeLog      string `json:"changeLog"`
}

TableNames generates the table names for a specific client and project

type TransactionConfig added in v0.0.2

type TransactionConfig struct {
	DefaultTimeout           time.Duration `json:"defaultTimeout"`
	MaxTimeout               time.Duration `json:"maxTimeout"`
	MaxRetryAttempts         int           `json:"maxRetryAttempts"`
	RetryAttempts            int           `json:"retryAttempts"`
	RetryDelay               time.Duration `json:"retryDelay"`
	IsolationLevel           string        `json:"isolationLevel"`
	EnableDeadlockDetection  bool          `json:"enableDeadlockDetection"`
	DeadlockCheckInterval    time.Duration `json:"deadlockCheckInterval"`
	DeadlockMaxWaitTime      time.Duration `json:"deadlockMaxWaitTime"`
	SlowTransactionThreshold time.Duration `json:"slowTransactionThreshold"`
	MinSuccessRate           float64       `json:"minSuccessRate"`
	MaxAverageDuration       time.Duration `json:"maxAverageDuration"`
	MaxConnectionPoolUsage   float64       `json:"maxConnectionPoolUsage"`
}

TransactionConfig contains transaction settings

type ValueType added in v0.0.3

type ValueType string

ValueType represents supported attribute value types.

const (
	ValueTypeText     ValueType = "text"
	ValueTypeSmallInt ValueType = "smallint"
	ValueTypeInteger  ValueType = "integer"
	ValueTypeBigInt   ValueType = "bigint"
	ValueTypeNumeric  ValueType = "numeric"  // double precision
	ValueTypeDate     ValueType = "date"     // for JSON attributes with format `date`
	ValueTypeDateTime ValueType = "datetime" // for JSON attributes with format `date-time`
	ValueTypeUUID     ValueType = "uuid"
	ValueTypeBool     ValueType = "bool"
	ValueTypeList     ValueType = "list" // array type stored as DuckDB LIST in parquet
)

Directories

Path Synopsis
cmd
benchmark command
lambda command
Package main provides the AWS Lambda entry point for the Forma API server.
Package main provides the AWS Lambda entry point for the Forma API server.
sample command
server command
tools command
cdc
duckdbinit
Package duckdbinit builds and applies the session-scoped initialization (INSTALL/LOAD/SET/PRAGMA) that every pooled DuckDB connection must run on open.
Package duckdbinit builds and applies the session-scoped initialization (INSTALL/LOAD/SET/PRAGMA) that every pooled DuckDB connection must run on open.
e2e_harness/federated
Package federated provides custom assertions for E2E testing.
Package federated provides custom assertions for E2E testing.
e2e_harness/production
Package production is the reusable E2E test harness that exercises the REAL production stack end to end (#173, epic #172):
Package production is the reusable E2E test harness that exercises the REAL production stack end to end (#173, epic #172):
parquetcheck
Package parquetcheck defines the parquet export schema invariant shared by every Forma parquet consumer: the three system columns each generation carries regardless of attribute evolution, with the exact DuckDB types both exporters emit (delta flush and init/compaction base).
Package parquetcheck defines the parquet export schema invariant shared by every Forma parquet consumer: the three system columns each generation carries regardless of attribute evolution, with the exact DuckDB types both exporters emit (delta flush and init/compaction base).
pgdsn
Package pgdsn builds libpq keyword/value connection strings.
Package pgdsn builds libpq keyword/value connection strings.
queryplan
Package queryplan provides the plan-cache primitives for #142: a stable query-shape fingerprint, a composite cache key, and a concurrency-safe cache for compiled planning artifacts.
Package queryplan provides the plan-cache primitives for #142: a stable query-shape fingerprint, a composite cache key, and a concurrency-safe cache for compiled planning artifacts.
reconcile
Package reconcile diffs a schema's S3 parquet objects against its manifest (issue #203).
Package reconcile diffs a schema's S3 parquet objects against its manifest (issue #203).
redact
Package redact removes credential material from strings before they leave the process — into a log sink, or into an HTTP response body.
Package redact removes credential material from strings before they leave the process — into a log sink, or into an HTTP response body.
schemavalidate
Package schemavalidate resolves entity JSON Schemas once and validates write payloads against them.
Package schemavalidate resolves entity JSON Schemas once and validates write payloads against them.
sqlgen/sqlgentest
Package sqlgentest provides shared helpers for tests that pin the postgres_scan contract on both sides: the runtime template (internal/sqlgen) and the executable §5 sketch in docs/federated-query/design.md (internal/federated).
Package sqlgentest provides shared helpers for tests that pin the postgres_scan contract on both sides: the runtime template (internal/sqlgen) and the executable §5 sketch in docs/federated-query/design.md (internal/federated).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL