durableindex

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	ContractCapability                = "contract: durable-index/v0"
	SegmentedSnapshotCapability       = "contract: segmented-snapshot/v0"
	ScopeProfileUnicodePrefixV0       = "unicode-prefix/v0"
	RowProfileObjectInventoryV0       = "object-inventory/v0"
	EncodingProfileParquetInventoryV0 = "parquet-inventory/v0"
	DefaultObjectNamespace            = "objects"

	KindIndexPlan          = "index_plan"
	KindIndexMetadata      = "index_metadata"
	KindIndexCutover       = "index_cutover"
	KindSnapshotPub        = "snapshot_publication"
	LogicalProjectionToken = "logical_projection"
	LogicalProfileID       = "example.index-logical/v0"
	LogicalProjectionMode  = "logical-only/v0"

	ModeBaseline = "baseline"
	ModeObserve  = "observe"
	ModeEnrich   = "enrich"

	AdmissionBaselineNotCompared = "baseline_not_compared"
	AdmissionNoLoss              = "no_loss"
	AdmissionLossAcknowledged    = "loss_acknowledged"
	AdmissionInherited           = "inherited"

	MaxEffectiveScopes         = 65536
	MaxTextUTF8Bytes           = 4096
	MaxEnrichmentBytes         = 65536
	MaxExactInteger            = int64((1 << 53) - 1) // 9007199254740991
	MaxAncestryDepth           = 64
	MaxMetadataBytes           = 64 * 1024 * 1024   // 67108864
	MaxDecompressedMemberBytes = 1024 * 1024 * 1024 // 1073741824
	InstantLayoutNanoUTC       = "2006-01-02T15:04:05.000000000Z"
)
View Source 
const DefaultMaxParquetDecompressedBytes int64 = 128 * 1024 * 1024

DefaultMaxParquetDecompressedBytes is the default 128 MiB per-member decompressed byte ceiling.

Variables

This section is empty.

Functions

func CompareRowKeys 

func CompareRowKeys(nsA, keyA, nsB, keyB string) int

CompareRowKeys compares two (namespace, key) pairs by unsigned UTF-8 byte order.

func CompareScopes 

func CompareScopes(a, b Scope) int

CompareScopes compares two scopes by unsigned UTF-8 byte order on (namespace, prefix).

func EncodeLogicalRowsNDJSON 

func EncodeLogicalRowsNDJSON(rows []LogicalRow) ([]byte, error)

EncodeLogicalRowsNDJSON serializes logical rows into deterministic UTF-8 NDJSON matching Crucible's twin format.

func EvaluateAdmission 

func EvaluateAdmission(
	mode string,
	setID string,
	parentPubSHA256 string,
	parentScopes []Scope,
	nextScopes []Scope,
	ack *TransitionAcknowledgment,
) (string, error)

EvaluateAdmission evaluates the I5 admission status for a baseline or successor transition.

func EvaluateCorpusCase 

func EvaluateCorpusCase(caseMap map[string]any) (map[string]any, error)

EvaluateCorpusCase evaluates a loaded corpus case and returns either the expected positive result map or {"refusal": reason}.

func ExportSliceDParquetTwins 

func ExportSliceDParquetTwins(dir string) error

ExportSliceDParquetTwins writes the deterministic Slice D physical Parquet twins, NDJSON logical twins, and manifest.json to dir.

func FormatInstant 

func FormatInstant(t time.Time) (string, error)

FormatInstant formats t as an exact 9-digit UTC timestamp string.

func HasScopeLoss 

func HasScopeLoss(parentScopes, nextScopes []Scope) (bool, error)

HasScopeLoss reports whether nextScopes loses any effective prefix coverage relative to parentScopes (I5). Every canonical parent scope must be covered by some canonical next scope in the same namespace.

func LoadCorpusCase 

func LoadCorpusCase(relPath string) (map[string]any, error)

LoadCorpusCase loads a positive or overlay reject case from schemas/durable-index/v0/<relPath>.

func NewRefusal 

func NewRefusal(reason, detail string) error

NewRefusal constructs a RefusalError with the given reason code and optional detail.

func PlanDigest 

func PlanDigest(setID string, scopes []Scope) (string, error)

PlanDigest computes the lowercase hex SHA-256 semantic plan_digest over PlanFramedBytes (I2).

func PlanFramedBytes 

func PlanFramedBytes(setID string, scopes []Scope) ([]byte, error)

PlanFramedBytes returns the semantic length-framed plan bytes (I2):

  1. ASCII "durable-index/v0/plan\x00"
  2. set_id framed as 4-byte big-endian UTF-8 byte length + bytes
  3. 4-byte big-endian effective scope count
  4. For each canonical scope, framed namespace + framed prefix.

func RefusalReason 

func RefusalReason(err error) (string, bool)

RefusalReason extracts the contract refusal reason from err if it wraps a *RefusalError.

func ScopeContains 

func ScopeContains(scope Scope, namespace, key string) bool

ScopeContains reports whether key in namespace belongs to scope by exact UTF-8 prefix bytes.

func ScopeCovers 

func ScopeCovers(outer, inner Scope) bool

ScopeCovers reports whether outer covers inner (same namespace and inner.Prefix starts with outer.Prefix).

func ScopesEqual 

func ScopesEqual(a, b []Scope) bool

ScopesEqual reports whether two Scope slices are identical in length and elements.

func StrictParseContractJSON 

func StrictParseContractJSON(data []byte) (any, error)

StrictParseContractJSON parses raw UTF-8 JSON bytes under contract strictness rules, returning a *RefusalError with reason "invalid_unicode", "duplicate_key", or "invalid_json".

func SubjectURI 

func SubjectURI(selection Selection) string

SubjectURI computes segmented-snapshot's immutable coverage-subject URI for selection.

func ValidateCanonicalScopes 

func ValidateCanonicalScopes(scopes []Scope) error

ValidateCanonicalScopes checks capacity bounds on scopes and refuses with noncanonical_plan if scopes is not already sorted, unique, and prefix-minimal.

func ValidateIdentifier 

func ValidateIdentifier(s string) bool

ValidateIdentifier reports whether s matches the contract's common id pattern.

func ValidateInstant 

func ValidateInstant(value string) error

ValidateInstant validates that value is an exact YYYY-MM-DDTHH:MM:SS.nnnnnnnnnZ Gregorian UTC timestamp (years 0001-9999, seconds 00-59, no leap seconds).

func ValidateParquetInventoryReaderAt 

func ValidateParquetInventoryReaderAt(r io.ReaderAt, size int64, maxDecompressedBytes int64) (*parquet.File, error)

ValidateParquetInventoryReaderAt validates a physical DDR-0004 parquet-inventory/v0 member (including magic, footer metadata, schema elements, actual page headers, and bounded ZSTD decompression) before row decoding.

func ValidateRows 

func ValidateRows(rows []LogicalRow, forbiddenSnapshotID string) error

ValidateRows checks a slice of logical rows for shape, capacity, strict primary-key order, uniqueness, stamp consistency, lineage order, and enrichment invariants (I4, I6).

func ValidateSHA256Hex 

func ValidateSHA256Hex(s string) bool

ValidateSHA256Hex reports whether s is a 64-character lowercase hex SHA-256 digest.

func ValidateTextUTF8 

func ValidateTextUTF8(s string) error

ValidateTextUTF8 checks that s is valid UTF-8 and at most MaxTextUTF8Bytes bytes.

func VerifyCutover 

func VerifyCutover(caseMap map[string]any) (map[string]any, error)

VerifyCutover validates an explicit baseline cutover receipt (index-cutover.schema.json).

func VerifyCutoverGraphs 

func VerifyCutoverGraphs(caseMap map[string]any) (map[string]any, error)

VerifyCutoverGraphs validates a complete cutover_graphs scenario.

func VerifyLogicalGraph 

func VerifyLogicalGraph(caseMap map[string]any) (map[string]any, error)

VerifyLogicalGraph validates a complete byte-bound logical projection graph (vectors/graphs/).

func VerifyPhysicalGraph 

func VerifyPhysicalGraph(pg PhysicalGraphCase) (map[string]any, error)

VerifyPhysicalGraph validates a complete segmented-snapshot/v0 + durable-index/v0 graph over physical parquet-inventory/v0 members (R2 when SourceEvidence is nil; W1/W2/W3 when SourceEvidence is supplied).

func VerifyRowMembers 

func VerifyRowMembers(caseMap map[string]any) (map[string]any, error)

VerifyRowMembers validates cross-member logical rows in manifest order.

func WriteParquetInventoryMember 

func WriteParquetInventoryMember(rows []LogicalRow, opts ParquetWriteOptions) ([]byte, error)

WriteParquetInventoryMember encodes validated logical rows into a self-contained DDR-0004 parquet-inventory/v0 member.

Types

type Binding 

type Binding struct {
	ID         string `json:"id"`
	Ref        string `json:"ref"`
	SHA256     string `json:"sha256"`
	ByteLength int64  `json:"byte_length"`
}

Binding pins an immutable artifact by id, logical ref, sha256, and byte_length.

type EnrichmentAttribute 

type EnrichmentAttribute struct {
	Name            string   `json:"name"`
	Value           *string  `json:"value"`
	Operation       string   `json:"operation"`
	ObservedAt      string   `json:"observed_at"`
	AgainstRevision Revision `json:"against_revision"`
}

EnrichmentAttribute represents a per-attribute enrichment bound to against_revision.

type EnrichmentUpdateInput 

type EnrichmentUpdateInput struct {
	Namespace string              `json:"namespace"`
	Key       string              `json:"key"`
	Attribute EnrichmentAttribute `json:"attribute"`
}

EnrichmentUpdateInput represents a per-key attribute update in a transition.

type IndexCutoverReceipt 

type IndexCutoverReceipt struct {
	Capabilities []string      `json:"capabilities"`
	Kind         string        `json:"kind"`
	Old          ParentBinding `json:"old"`
	New          ParentBinding `json:"new"`
	AuthorizedBy string        `json:"authorized_by"`
	AcceptedAt   string        `json:"accepted_at"`
	Rollback     Binding       `json:"rollback"`
}

IndexCutoverReceipt represents contract:durable-index/v0/index-cutover.schema.json.

func BuildCutoverReceipt 

func BuildCutoverReceipt(
	oldEndpoint ParentBinding,
	newEndpoint ParentBinding,
	authorizedBy string,
	acceptedAt string,
) (IndexCutoverReceipt, []byte, error)

BuildCutoverReceipt constructs and validates a canonical IndexCutoverReceipt.

type IndexPlan 

type IndexPlan struct {
	Capabilities []string `json:"capabilities"`
	Kind         string   `json:"kind"`
	SetID        string   `json:"set_id"`
	ScopeProfile string   `json:"scope_profile"`
	Scopes       []Scope  `json:"scopes"`
}

IndexPlan represents a contract:durable-index/v0/index-plan.schema.json document.

func BuildIndexPlan 

func BuildIndexPlan(setID string, scopes []Scope) (IndexPlan, []byte, string, error)

BuildIndexPlan canonicalizes scopes and builds a canonical IndexPlan document, its JSON bytes, and its semantic plan_digest.

type LogicalRow 

type LogicalRow struct {
	Namespace    string                `json:"namespace"`
	Key          string                `json:"key"`
	Revision     *Revision             `json:"revision"`
	SizeBytes    int64                 `json:"size_bytes"`
	LastModified *string               `json:"last_modified"`
	StorageClass *string               `json:"storage_class"`
	FirstSeen    Stamp                 `json:"first_seen"`
	LastSeen     Stamp                 `json:"last_seen"`
	LastChanged  Stamp                 `json:"last_changed"`
	Deleted      *Stamp                `json:"deleted"`
	Enrichments  []EnrichmentAttribute `json:"enrichments"`
}

LogicalRow represents a logical object-inventory/v0 row (index-row.schema.json).

func ReadParquetInventoryMember 

func ReadParquetInventoryMember(data []byte, maxDecompressedBytes int64) (rows []LogicalRow, err error)

ReadParquetInventoryMember validates a physical DDR-0004 parquet-inventory/v0 member and decodes its logical rows.

type LossAcknowledgment 

type LossAcknowledgment struct {
	Parent         ParentBinding `json:"parent"`
	PlanSHA256     string        `json:"plan_sha256"`
	AcknowledgedBy string        `json:"acknowledged_by"`
	AcknowledgedAt string        `json:"acknowledged_at"`
}

LossAcknowledgment is the wire loss_acknowledgment object in index_metadata.

type ObservationInput 

type ObservationInput struct {
	Namespace    string    `json:"namespace"`
	Key          string    `json:"key"`
	Revision     *Revision `json:"revision"`
	SizeBytes    int64     `json:"size_bytes"`
	LastModified *string   `json:"last_modified"`
	StorageClass *string   `json:"storage_class"`
}

ObservationInput represents a positive observation input to a transition.

type ParentBinding 

type ParentBinding struct {
	Selection   Selection `json:"selection"`
	Publication Binding   `json:"publication"`
}

ParentBinding binds a parent snapshot selection and publication pin.

type ParquetCodec 

type ParquetCodec string

ParquetCodec selects the DDR-0004 column compression codec. 

const (
	// ParquetCodecZSTD selects ZSTD column compression.
	ParquetCodecZSTD ParquetCodec = "zstd"
	// ParquetCodecUncompressed selects UNCOMPRESSED column encoding.
	ParquetCodecUncompressed ParquetCodec = "uncompressed"
)

type ParquetWriteOptions 

type ParquetWriteOptions struct {
	Codec               ParquetCodec
	DataPageVersion     int
	SizeBytesAnnotation SizeBytesAnnotation
	MaxRowsPerRowGroup  int64
}

ParquetWriteOptions configures deterministic DDR-0004 parquet-inventory/v0 member emission.

type PhysicalGraphCase 

type PhysicalGraphCase struct {
	Publication   Binding
	Store         map[string]string
	MemberParquet map[string][]byte
	// SourceEvidence is optional. When nil/empty, VerifyPhysicalGraph evaluates R2
	// (self-contained snapshot reader without historical parent transition checks).
	// When present, VerifyPhysicalGraph evaluates W1/W2/W3 transition parity.
	SourceEvidence map[string]any
	RequestExport  bool
}

PhysicalGraphCase holds a complete segmented-snapshot/v0 + durable-index/v0 graph with real parquet-inventory/v0 members.

func BuildPhysicalGraphFromLogical 

func BuildPhysicalGraphFromLogical(logicalInput map[string]any, opts ParquetWriteOptions) (PhysicalGraphCase, error)

BuildPhysicalGraphFromLogical converts a logical projection graph case (such as vectors/graphs/baseline.json, observe.json, or enrich.json) into a physical parquet-inventory/v0 graph case with real Parquet member bytes.

type RefusalError 

type RefusalError struct {
	Reason string
	Detail string
}

RefusalError represents a contract-defined refusal with its normative reason code.

func (*RefusalError) Error 

func (e *RefusalError) Error() string

type Revision 

type Revision struct {
	Kind  string `json:"kind"`
	Value string `json:"value"`
}

Revision represents an opaque {kind, value} object revision token.

type Scope 

type Scope struct {
	Namespace string `json:"namespace"`
	Prefix    string `json:"prefix"`
}

Scope represents a single {namespace, prefix} scope in an index_plan.

func CanonicalizeScopes 

func CanonicalizeScopes(scopes []Scope) ([]Scope, error)

CanonicalizeScopes sorts, deduplicates, and removes prefix-contained scopes without modifying individual prefix bytes (I2). Enforces the 65,536 scope count, identifier syntax on namespace, and 4,096 UTF-8 byte ceiling per namespace and prefix.

func VerifyBindings 

func VerifyBindings(caseMap map[string]any, ancestry []string) ([]Scope, error)

VerifyBindings validates an index_metadata/index_plan/coverage-attestation subgraph against source evidence (I1, I2, I3).

type Selection 

type Selection struct {
	NamespaceID string `json:"namespace_id"`
	SnapshotID  string `json:"snapshot_id"`
	ArtifactID  string `json:"artifact_id"`
}

Selection identifies a segmented-snapshot namespace, snapshot, and artifact.

type SizeBytesAnnotation 

type SizeBytesAnnotation string

SizeBytesAnnotation selects which DDR-0004 size_bytes schema annotation form to emit. 

const (
	// SizeBytesAnnotationAbsent emits physical INT64 size_bytes with no logical annotation.
	SizeBytesAnnotationAbsent SizeBytesAnnotation = "absent"
	// SizeBytesAnnotationInt64Signed emits physical INT64 size_bytes with INT(64, signed=true).
	SizeBytesAnnotationInt64Signed SizeBytesAnnotation = "int64_signed"
)

type SliceDTwinArtifact 

type SliceDTwinArtifact struct {
	ID                  string              `json:"id"`
	Codec               ParquetCodec        `json:"codec,omitempty"`
	DataPageVersion     int                 `json:"data_page_version,omitempty"`
	SizeBytesAnnotation SizeBytesAnnotation `json:"size_bytes_annotation,omitempty"`
	ParquetBytes        []byte              `json:"-"`
	ParquetSHA256       string              `json:"parquet_sha256"`
	ParquetByteLength   int64               `json:"parquet_byte_length"`
	LogicalRows         []LogicalRow        `json:"-"`
	LogicalNDJSON       []byte              `json:"-"`
	LogicalNDJSONSHA256 string              `json:"logical_ndjson_sha256,omitempty"`
	LogicalNDJSONBytes  int64               `json:"logical_ndjson_byte_length,omitempty"`
	RowCount            int64               `json:"row_count"`
	ExpectedRefusal     string              `json:"expected_refusal,omitempty"`
}

SliceDTwinArtifact describes one deterministic physical Parquet twin member or negative fixture.

func GenerateSliceDParquetTwins 

func GenerateSliceDParquetTwins() ([]SliceDTwinArtifact, error)

GenerateSliceDParquetTwins generates the complete deterministic DDR-0004 physical Parquet twin suite (3 positive + 20 negative).

type Stamp 

type Stamp struct {
	SnapshotID string `json:"snapshot_id"`
	At         string `json:"at"`
}

Stamp represents a {snapshot_id, at} lineage stamp.

type TransitionAcknowledgment 

type TransitionAcknowledgment struct {
	ParentSHA256   string `json:"parent_sha256"`
	PlanDigest     string `json:"plan_digest"`
	AcknowledgedBy string `json:"acknowledged_by"`
	AcknowledgedAt string `json:"acknowledged_at"`
}

TransitionAcknowledgment is the normalized acknowledgment record in transition inputs.

type TransitionInput 

type TransitionInput struct {
	Mode                 string                    `json:"mode"`
	SetID                string                    `json:"set_id"`
	SnapshotID           string                    `json:"snapshot_id"`
	Generation           int64                     `json:"generation"`
	ObservationAt        *string                   `json:"observation_at"`
	DiscoveryComplete    bool                      `json:"discovery_complete"`
	Parent               *TransitionParentInput    `json:"parent"`
	AdmittedParentSHA256 *string                   `json:"admitted_parent_sha256"`
	Scopes               []Scope                   `json:"scopes"`
	Acknowledgment       *TransitionAcknowledgment `json:"acknowledgment"`
	Observations         []ObservationInput        `json:"observations"`
	AbsenceAuthority     []Scope                   `json:"absence_authority"`
	Enrichments          []EnrichmentUpdateInput   `json:"enrichments"`
}

TransitionInput represents the full logical transition input.

type TransitionOutcome 

type TransitionOutcome struct {
	Rows                   []LogicalRow `json:"rows"`
	Admission              string       `json:"admission"`
	RetainedUnobservedRows int64        `json:"retained_unobserved_rows"`
}

TransitionOutcome is the computed result of FoldTransition.

func FoldTransition 

func FoldTransition(in TransitionInput) (TransitionOutcome, error)

FoldTransition executes the I4/I5/I6 logical transition fold over normalized inputs.

func VerifyBoundTransition 

func VerifyBoundTransition(caseMap map[string]any) (TransitionOutcome, error)

VerifyBoundTransition validates a bound_transitions vector or reject case (I1-I6).

type TransitionParentInput 

type TransitionParentInput struct {
	PublicationSHA256   string       `json:"publication_sha256"`
	SetID               string       `json:"set_id"`
	SnapshotID          string       `json:"snapshot_id"`
	Generation          int64        `json:"generation"`
	Scopes              []Scope      `json:"scopes"`
	Rows                []LogicalRow `json:"rows"`
	AncestorSnapshotIDs []string     `json:"ancestor_snapshot_ids,omitempty"`
}

TransitionParentInput represents the verified parent state in a transition.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.