internal

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 25, 2026 License: MIT Imports: 33 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DecodeBase32ToUUID

func DecodeBase32ToUUID(s string) (uuid.UUID, error)

func DecodeFromBase32

func DecodeFromBase32(s string) ([]byte, error)

func EncodeToBase32

func EncodeToBase32(data []byte) string

func EncodeUUIDToBase32

func EncodeUUIDToBase32(id uuid.UUID) string

func FilterAttributes added in v0.0.18

func FilterAttributes(attributes map[string]any, attrs []string) map[string]any

FilterAttributes filters a map of attributes based on the requested attribute paths. If attrs is nil or empty, returns the original attributes unchanged. Supports nested paths like "contact.name" or "contact.phone".

func FilterDataRecord added in v0.0.18

func FilterDataRecord(record *forma.DataRecord, attrs []string) *forma.DataRecord

FilterDataRecord applies attribute filtering to a forma.DataRecord. Returns a new DataRecord with filtered attributes.

func MapKeys

func MapKeys[K comparable, V any](m map[K]V) []K

MapKeys extracts all keys from a map and returns them as a slice. The order of keys is non-deterministic due to map iteration.

func MapValues

func MapValues[K comparable, V any](m map[K]V) []V

MapValues extracts all values from a map and returns them as a slice. The order of values is non-deterministic due to map iteration.

func NewEntityManager

func NewEntityManager(
	transformer model.PersistentRecordTransformer,
	repository model.PersistentRecordRepository,
	federatedQueryEngine model.FederatedQueryEngine,
	registry forma.SchemaRegistry,
	config *forma.Config,
	validator *schemavalidate.Validator,
	opts ...EntityManagerOption,
) (forma.EntityManager, error)

NewEntityManager creates a new EntityManager instance, or fails.

The one failure it has is the relation index, and it is why this returns an error at all (#388). A manager holding no index does not merely lose a feature, and loses it for every schema rather than for the offending one:

  • StripComputedFields is an identity function on a nil receiver, so the relation subtree becomes caller-writable and persistable again — the exact state #318 exists to remove — and nothing says so at any point afterwards;
  • RelationRoots answers nil on the same receiver, and the required-policy check reads an empty set as "this attribute is not beneath a relation root" (transform.RelationRoots.Covers, attribute_converter.go), so #315's carve-out stops applying and payloads #314 ruled acceptable begin failing with missing-required errors.

Neither is visible to the caller on the returned manager, and both last for the process lifetime. So a registry fault that used to become a warning now becomes a refusal to build, and the caller decides what to do about it.

A successfully built manager therefore always holds a non-nil index, from one of two places: WithRelationIndex, when the caller has already built and validated one (the composition root has), or a load of the registry performed here. LoadRelationIndex answers an empty index rather than nil for a registry that lists nothing and for a nil registry, so neither of those is a failure and neither leaves the field nil.

The load is gated on the registry because the registry is what the index is built from. It used to be gated on config.Entity.SchemaDirectory, which stopped being the source: an embedder who registers a schema carrying x-relation but leaves SchemaDirectory empty gets stripping, enrichment and the #315 carve-out, all of which that schema asks for and none of which an unrelated path setting should have been deciding.

func PostgresHealthCheck added in v0.0.23

func PostgresHealthCheck(ctx context.Context, dsn string, timeout time.Duration) error

PostgresHealthCheck attempts to connect and ping a Postgres instance using a DSN. timeout may be 0 to use a sensible default (5s).

func S3HealthCheck added in v0.0.23

func S3HealthCheck(ctx context.Context, cfg forma.DuckDBConfig, timeout time.Duration) error

S3HealthCheck attempts a best-effort HTTP ping against the configured S3 endpoint. This is intentionally lightweight and non-authoritative: it will only succeed for endpoints that accept anonymous HEAD/GET requests (e.g., some MinIO setups). For AWS S3 this will often return 403 but is still useful to validate DNS/resolution and TLS.

func SnapshotSchemaDocuments added in v0.2.0

func SnapshotSchemaDocuments(registry forma.SchemaRegistry) forma.SchemaRegistry

SnapshotSchemaDocuments captures a registry's schema documents once and serves every later read of them from that capture.

It exists because forma.SchemaRegistry is a public extension point that promises nothing about repeated reads. An implementation serving documents from a database, a cache, or the network may answer differently on a second call, so two consumers built from independent reads can be handed two different documents. The two consumers that matter are the startup pair — schemavalidate.New and LoadRelationIndex — and the disagreement they can reach is exactly the state the relation guard exists to refuse: a validator that demands a relation root paired with an index that strips it, which makes the entity unwritable for the process lifetime behind a preflight that passed (#318 review). Building both from one snapshot removes the disagreement rather than making it unlikely.

What is captured, precisely: one ListSchemas call, and one GetSchemaByName call per name it reported. Each answer is stored whole — the error included — so a snapshot replays a registry failure exactly as the registry gave it, leaving each consumer's own failure handling and message unchanged.

What is NOT captured, and the boundaries are stated rather than implied:

  • the schema *directory*. schemavalidate.New resolves cross-file "$ref"s by reading sibling files off disk (its fileLoader), which nothing here can freeze. So this is one consistent view of the registry's documents, not an atomic view of everything a validator is built from.
  • a name ListSchemas did not report. Neither startup consumer asks for one — both iterate ListSchemas — so there is nothing captured to answer with, and such a read is passed to the wrapped registry.
  • the attribute caches. Neither startup consumer reads them; capturing them would add a read per schema at startup and a new startup failure class for data no consumer of this snapshot looks at. Those accessors delegate.

The result is immutable after construction, so it is safe to share and holds no lock. It is a startup value: it does not observe a registry that reloads, which is why the manager keeps the caller's registry rather than this (factory.buildSchemaGuards).

A nil registry yields a nil forma.SchemaRegistry — an untyped nil, not a boxed nil pointer — so a caller's own nil check still fires.

func ValidatePostgresConfig added in v0.0.23

func ValidatePostgresConfig(cfg forma.DatabaseConfig) error

ValidatePostgresConfig performs basic sanity checks on Postgres-related settings.

func ValidateS3Config added in v0.0.23

func ValidateS3Config(cfg forma.DuckDBConfig) error

ValidateS3Config performs basic sanity checks on S3-related DuckDB settings.

Types

type DBPersistentRecordRepository added in v0.0.23

type DBPersistentRecordRepository struct {
	// contains filtered or unexported fields
}

func NewDBPersistentRecordRepository added in v0.0.23

func NewDBPersistentRecordRepository(pool persistentRecordPool, metadataCache *schemameta.MetadataCache, opts ...RepoOption) *DBPersistentRecordRepository

func (*DBPersistentRecordRepository) BatchDeletePersistentRecords added in v0.0.24

func (r *DBPersistentRecordRepository) BatchDeletePersistentRecords(ctx context.Context, tables model.StorageTables, keys []model.PersistentRecordKey) error

func (*DBPersistentRecordRepository) BatchInsertPersistentRecords added in v0.0.24

func (r *DBPersistentRecordRepository) BatchInsertPersistentRecords(ctx context.Context, tables model.StorageTables, records []*model.PersistentRecord) error

func (*DBPersistentRecordRepository) BatchUpdatePersistentRecords added in v0.0.24

func (r *DBPersistentRecordRepository) BatchUpdatePersistentRecords(ctx context.Context, tables model.StorageTables, records []*model.PersistentRecord) error

func (*DBPersistentRecordRepository) BuildHybridConditions added in v0.2.0

func (r *DBPersistentRecordRepository) BuildHybridConditions(tables model.StorageTables, fq *model.FederatedAttributeQuery) (string, []any, error)

BuildHybridConditions builds the main-table/EAV hybrid WHERE clause for a federated query, for the federated engine's federated.PostgresFederatedSource seam.

func (*DBPersistentRecordRepository) DeletePersistentRecord added in v0.0.23

func (r *DBPersistentRecordRepository) DeletePersistentRecord(ctx context.Context, tables model.StorageTables, schemaID int16, rowID uuid.UUID) error

func (*DBPersistentRecordRepository) GetPersistentRecord added in v0.0.23

func (r *DBPersistentRecordRepository) GetPersistentRecord(ctx context.Context, tables model.StorageTables, schemaID int16, rowID uuid.UUID) (*model.PersistentRecord, error)

func (*DBPersistentRecordRepository) InsertPersistentRecord added in v0.0.23

func (r *DBPersistentRecordRepository) InsertPersistentRecord(ctx context.Context, tables model.StorageTables, record *model.PersistentRecord) error

func (*DBPersistentRecordRepository) QueryPersistentRecords added in v0.0.23

func (*DBPersistentRecordRepository) QueryPersistentRecordsByAttrValues added in v0.2.0

func (r *DBPersistentRecordRepository) QueryPersistentRecordsByAttrValues(
	ctx context.Context,
	tables model.StorageTables,
	schemaID int16,
	attr string,
	values []string,
	limit int,
) (*model.PersistentRecordPage, error)

QueryPersistentRecordsByAttrValues fetches full records whose attribute equals any of the given values through a single set-based anchor scan (#268). Relation enrichment previously expanded one equality condition per value into OR-of-N correlated EXISTS subqueries, which degenerated into a full EAV scan with N subplans per row.

func (*DBPersistentRecordRepository) RunOptimizedQuery added in v0.2.0

func (r *DBPersistentRecordRepository) RunOptimizedQuery(
	ctx context.Context,
	tables model.StorageTables,
	schemaID int16,
	clause string,
	args []any,
	limit, offset int,
	attributeOrders []model.AttributeOrder,
	useMainTableAsAnchor bool,
) ([]*model.PersistentRecord, int64, error)

RunOptimizedQuery exposes the optimized single-query path (prebuilt WHERE clause and args) for the federated engine's federated.PostgresFederatedSource seam.

func (*DBPersistentRecordRepository) StreamOptimizedQuery added in v0.0.23

func (r *DBPersistentRecordRepository) StreamOptimizedQuery(
	ctx context.Context,
	tables model.StorageTables,
	schemaID int16,
	clause string,
	args []any,
	limit, offset int,
	attributeOrders []model.AttributeOrder,
	useMainTableAsAnchor bool,
	rowHandler func(*model.PersistentRecord) error,
) (int64, error)

func (*DBPersistentRecordRepository) UpdatePersistentRecord added in v0.0.23

func (r *DBPersistentRecordRepository) UpdatePersistentRecord(ctx context.Context, tables model.StorageTables, record *model.PersistentRecord) error

type EntityManagerOption added in v0.2.0

type EntityManagerOption func(*entityManager)

EntityManagerOption customizes NewEntityManager construction.

func WithCloser added in v0.2.0

func WithCloser(c io.Closer) EntityManagerOption

WithCloser registers a resource the manager owns and must release on Close. Callers pass only non-nil resources; a typed-nil pointer boxed in io.Closer would not compare equal to nil here, so the guard lives at the call site.

func WithRelationIndex added in v0.2.0

func WithRelationIndex(idx *RelationIndex) EntityManagerOption

WithRelationIndex installs a relation index the caller has already built and validated, instead of letting the manager load its own.

It exists because the composition root has to check relation declarations before it builds anything, and a second, independent load afterwards is not guaranteed to see what the first one saw: forma.SchemaRegistry may serve documents from a database or over a network, so it can change or fail between the two reads. When it does, the manager's own load is the one that fails, and the caller is left holding a construction error over declarations its preflight had already approved. Handing the validated instance forward removes the second read entirely (#318 review).

A nil index is ignored rather than installed, so the option cannot become a way to switch stripping off by accident; the manager then self-loads as any direct constructor does. That self-load fails closed — NewEntityManager returns the error rather than warning and continuing with no index (#388) — so the option changes which read decides, never whether stripping happens.

type RelationDescriptor added in v0.0.18

type RelationDescriptor struct {
	ChildSchema        string
	ChildPath          string
	ParentSchema       string
	ParentPath         string
	ForeignKeyAttr     string
	ParentIDAttr       string
	ForeignKeyRequired bool
}

RelationDescriptor captures how a child schema derives fields from a parent schema.

type RelationIndex added in v0.0.18

type RelationIndex struct {
	// contains filtered or unexported fields
}

RelationIndex stores parent-child relations keyed by child schema name.

func LoadRelationIndex added in v0.0.18

func LoadRelationIndex(registry forma.SchemaRegistry) (*RelationIndex, error)

LoadRelationIndex builds a relation index from the schema documents the registry serves, one per name in registry.ListSchemas(), and fails closed on a relation declaration the runtime cannot honour.

Building and validating are one call because they are one walk: the guard's question — does this schema require a property the strip removes? — can only be asked of the descriptors the walk has just built. There is no separate validate-only entry point, and a caller that wants the check runs this and keeps the index (see WithRelationIndex for why keeping it matters).

The registry, not SCHEMA_DIR, is the source, and that is the point: the runtime validator is built from the same pair of accessors (schemavalidate.New), so neither is reading a document the other cannot see. forma.SchemaRegistry is a public extension point whose implementations may load from files, a database, or anything else, and only what it registers is ever validated or written to.

Same accessors is not the same as same bytes, and the difference is why the factory builds this index exactly once and hands it to NewEntityManager (WithRelationIndex). A registry is free to answer differently on a later call — it may be reading a database — so two independent loads can disagree, and the losing one would be the manager's, silently disabling stripping after a successful preflight. What this function guarantees is that one *load* sees a consistent view; keeping the validator and the manager on that same load is the caller's job.

A failure here reaches every construction site, because NewEntityManager returns it rather than continuing with no index (#388, #318). It has to: one offending schema fails the whole registry load, and a manager over a nil index strips nothing for any schema, not just the offender. What NewEntityManager still cannot enforce is that its load and the caller's preflight are the same load — that is what WithRelationIndex is for. Both shipped construction sites (the factory and the production e2e harness) therefore call this at a position where returning an error aborts startup, and pass the result forward.

What aborts startup, exactly:

  • a schema that declares a relation root the analysed fragment shows is required on every document — the root's own "required", or one in an allOf chain (see collectUnconditionalRootRequired for the fragment and why it is that small);
  • a schema the registry lists but cannot then serve, or whose document does not decode into any (see loadSchemaRelations for the decode).

Nothing is refused for being unanalysable. Outside the fragment the walk has no opinion, and a schema is booted whether or not it composes its root object out of "not", "anyOf", "oneOf", "if"/"then"/"else", the dependent* family or a root "$ref". Some of those really do demand a stripped relation root, and those writes fail at runtime with an operator-facing explanation attached (explainStrippedRelationRoots) rather than at startup.

The second cause is fatal rather than skipped because every name walked here came from ListSchemas: it names a schema the runtime resolves, validates writes against, and strips relation subtrees for. A document that cannot be read at all is therefore a fault in the registry itself, not a stray file left beside the ones that matter.

The boundary is the decode into any, stated exactly that way because the two looser descriptions of it are both wrong. It is not JSON *shape*: a document that decodes but is not an object is accepted, since it declares no properties and so has no relation roots, and drawing the line there made booleans fatal — a boolean is a legal JSON Schema. Nor is it JSON *syntax*, which is narrower than what actually fails:

{"properties":{"a":{"type":"string"},"b":{"const":1e999}}}

is well-formed JSON — json.Valid answers true, and RFC 8259 puts no bound on a number's magnitude — yet decoding it into any fails with "cannot unmarshal number 1e999 into Go value of type float64". So the fatal class is syntax errors *plus* numbers outside float64's range.

Observation, not justification: schemavalidate.New rejects every document this cause is fatal on, and both shipped call sites run it first, so in the shipped ordering the cause is unreachable. Measured for both halves, and it holds for the whole class rather than for the cases someone thought to try:

  • Syntax errors are target-independent. json.Unmarshal runs checkValid over the whole input before decoding anything (encoding/json/decode.go), so a malformed document fails unmarshalling into any target type.
  • Overflow is caught because jsonschema.Schema decodes the same bytes the same way this loader does. Schema.UnmarshalJSON tries a bool first (schema.go:379, which is why booleans are legal schemas) and otherwise calls unmarshalStructWithMap, which decodes the document twice: into its own struct, and into a map[string]any (util.go:367 and :372). A JSON object therefore passes every value through the same any decode this loader performs, so a value that fails here fails there — in the struct pass for a keyword the library models ("properties", "enum"), in the map pass for one it does not ("x-junk"). A document that is neither an object nor a boolean is rejected outright by the struct pass, which is why "[1,2,3]" and "\"x\"" are refused by the validator even though this loader accepts them.

The relationship is one-directional and deliberately so: this loader's fatal class is a strict subset of what schemavalidate.New refuses. The reason to refuse remains that the schema is registered.

A registry that lists no schemas, and a nil registry, both yield an empty index and no error.

func (*RelationIndex) RelationRootNames added in v0.2.0

func (idx *RelationIndex) RelationRootNames(schema string) []string

RelationRootNames returns schema's relation root names in sorted order, as diagnostic input for the write path (explainStrippedRelationRoots).

Sorted because bySchema is built by ranging a map of properties, so descriptor order is randomised per process; an operator comparing two log lines needs the list to be the same list.

It answers the roots alone, which is less than StripComputedFields removes — see RelationRoots for that distinction, which applies here identically. A nil receiver or a schema with no relations answers nil.

func (*RelationIndex) RelationRoots added in v0.2.0

func (idx *RelationIndex) RelationRoots(schema string) transform.RelationRoots

RelationRoots returns schema's relation root names, as the set transform.AttributeConverter.FromEAVRecords consults to skip required-policy enforcement beneath a relation root (#315). That check is not read-only — transform.ToAttributes reaches it on every create and update — so this is not read-path-only state.

It answers the roots alone, which is less than StripComputedFields removes: the strip also takes every dotted descendant of each root, by the prefix rule in coversRelationSubtree. Do not read this set as the strip's reach. A caller that needs the reach has to apply that prefix rule itself — transform.RelationRoots.Covers applies it for names strictly beneath a root, and deliberately excludes a name that is itself a root.

A nil receiver or a schema with no relations answers nil, which that set reads as "no relation roots".

func (*RelationIndex) Relations added in v0.0.18

func (idx *RelationIndex) Relations(schema string) []RelationDescriptor

Relations returns descriptors for a child schema.

func (*RelationIndex) StripComputedFields added in v0.0.18

func (idx *RelationIndex) StripComputedFields(schema string, data map[string]any) map[string]any

StripComputedFields removes the relation subtree from the payload before it is validated and persisted: the property carrying x-relation and everything beneath it, in either spelling.

The subtree is derived on read from the parent entity (entityRelationService.enrichDataRecords), which replaces it wholesale, so a caller-written value there is unreadable wherever enrichment applies. Enrichment does skip a record whose foreign key is missing or empty, whose parent row is not found, or whose parent fragment is nil, and a persisted value would survive those reads — but a caller cannot rely on that, and the next update deletes the value anyway. Dropping is silent — see TestCreateDropsDottedKeyBeneathRelationRoot.

type RepoOption added in v0.2.0

type RepoOption func(*DBPersistentRecordRepository)

RepoOption customizes optional repository collaborators.

func WithPlanCache added in v0.2.0

func WithPlanCache(c *queryplan.Cache) RepoOption

WithPlanCache injects a shared plan cache (#142).

type Set

type Set[T comparable] struct {
	// contains filtered or unexported fields
}

Set is a generic data structure that represents a collection of unique items. It uses a map internally for O(1) operations.

func NewSet

func NewSet[T comparable]() *Set[T]

NewSet creates and returns a new empty Set.

func (*Set[T]) Add

func (s *Set[T]) Add(item T)

Add inserts an item into the set. If the item already exists, it has no effect.

func (*Set[T]) Clear

func (s *Set[T]) Clear()

Clear removes all items from the set.

func (*Set[T]) Contains

func (s *Set[T]) Contains(item T) bool

Contains checks if an item exists in the set.

func (*Set[T]) Remove

func (s *Set[T]) Remove(item T)

Remove deletes an item from the set. If the item doesn't exist, it has no effect.

func (*Set[T]) Size

func (s *Set[T]) Size() int

Size returns the number of items in the set.

func (*Set[T]) ToSlice

func (s *Set[T]) ToSlice() []T

ToSlice converts the set to a slice containing all items. The order of items is non-deterministic due to map iteration.

Directories

Path Synopsis
Package duckdbinit builds and applies the session-scoped initialization (INSTALL/LOAD/SET/PRAGMA) that every pooled DuckDB connection must run on open.
Package duckdbinit builds and applies the session-scoped initialization (INSTALL/LOAD/SET/PRAGMA) that every pooled DuckDB connection must run on open.
federated
Package federated provides custom assertions for E2E testing.
Package federated provides custom assertions for E2E testing.
production
Package production is the reusable E2E test harness that exercises the REAL production stack end to end (#173, epic #172):
Package production is the reusable E2E test harness that exercises the REAL production stack end to end (#173, epic #172):
Package parquetcheck defines the parquet export schema invariant shared by every Forma parquet consumer: the three system columns each generation carries regardless of attribute evolution, with the exact DuckDB types both exporters emit (delta flush and init/compaction base).
Package parquetcheck defines the parquet export schema invariant shared by every Forma parquet consumer: the three system columns each generation carries regardless of attribute evolution, with the exact DuckDB types both exporters emit (delta flush and init/compaction base).
Package pgdsn builds libpq keyword/value connection strings.
Package pgdsn builds libpq keyword/value connection strings.
Package queryplan provides the plan-cache primitives for #142: a stable query-shape fingerprint, a composite cache key, and a concurrency-safe cache for compiled planning artifacts.
Package queryplan provides the plan-cache primitives for #142: a stable query-shape fingerprint, a composite cache key, and a concurrency-safe cache for compiled planning artifacts.
Package reconcile diffs a schema's S3 parquet objects against its manifest (issue #203).
Package reconcile diffs a schema's S3 parquet objects against its manifest (issue #203).
Package redact removes credential material from strings before they leave the process — into a log sink, or into an HTTP response body.
Package redact removes credential material from strings before they leave the process — into a log sink, or into an HTTP response body.
Package schemavalidate resolves entity JSON Schemas once and validates write payloads against them.
Package schemavalidate resolves entity JSON Schemas once and validates write payloads against them.
sqlgentest
Package sqlgentest provides shared helpers for tests that pin the postgres_scan contract on both sides: the runtime template (internal/sqlgen) and the executable §5 sketch in docs/federated-query/design.md (internal/federated).
Package sqlgentest provides shared helpers for tests that pin the postgres_scan contract on both sides: the runtime template (internal/sqlgen) and the executable §5 sketch in docs/federated-query/design.md (internal/federated).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL