internal

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 28, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	NullNode   = avro.SchemaNode{Type: atype.Null}
	BoolNode   = avro.SchemaNode{Type: atype.Boolean}
	IntNode    = avro.SchemaNode{Type: atype.Int}
	LongNode   = avro.SchemaNode{Type: atype.Long}
	FloatNode  = avro.SchemaNode{Type: atype.Float}
	DoubleNode = avro.SchemaNode{Type: atype.Double}
	StringNode = avro.SchemaNode{Type: atype.String}
	BytesNode  = avro.SchemaNode{Type: atype.Bytes}

	DateNode        = avro.SchemaNode{Type: atype.Int, LogicalType: atype.Date}
	TimeNode        = avro.SchemaNode{Type: atype.Long, LogicalType: atype.TimeMicros}
	TimestampNode   = avro.SchemaNode{Type: atype.Long, LogicalType: atype.TimestampMicros, Props: map[string]any{"adjust-to-utc": false}}
	TimestampTzNode = avro.SchemaNode{Type: atype.Long, LogicalType: atype.TimestampMicros, Props: map[string]any{"adjust-to-utc": true}}

	TimestampNsNode   = avro.SchemaNode{Type: atype.Long, LogicalType: atype.TimestampNanos, Props: map[string]any{"adjust-to-utc": false}}
	TimestampTzNsNode = avro.SchemaNode{Type: atype.Long, LogicalType: atype.TimestampNanos, Props: map[string]any{"adjust-to-utc": true}}

	UUIDNode = avro.SchemaNode{Type: atype.Fixed, Name: "uuid_fixed", Size: 16, LogicalType: atype.UUID}
)

Schema nodes for reuse in composition.

View Source 
var (
	NullSchema           = mustSchema(NullNode)
	BoolSchema           = mustSchema(BoolNode)
	NullableBoolSchema   = NullableSchema(BoolSchema)
	BinarySchema         = mustSchema(BytesNode)
	NullableBinarySchema = NullableSchema(BinarySchema)
	StringSchema         = mustSchema(StringNode)
	IntSchema            = mustSchema(IntNode)
	NullableIntSchema    = NullableSchema(IntSchema)
	LongSchema           = mustSchema(LongNode)
	NullableLongSchema   = NullableSchema(LongSchema)
	FloatSchema          = mustSchema(FloatNode)
	DoubleSchema         = mustSchema(DoubleNode)
	DateSchema           = mustSchema(DateNode)
	TimeSchema           = mustSchema(TimeNode)
	TimestampSchema      = mustSchema(TimestampNode)
	TimestampTzSchema    = mustSchema(TimestampTzNode)
	UUIDSchema           = mustSchema(UUIDNode)
)

Compiled schemas for direct encoding/decoding use.

View Source 
var AvroSchemaMap = make(map[string]*avro.Schema)

AvroSchemaMap stores prebuilt schemas by name for lookup.

Functions

func BorrowedDataFileBounds added in v0.7.0 

func BorrowedDataFileBounds(file DataFileStats) (lowerBounds, upperBounds map[int][]byte)

BorrowedDataFileBounds returns the lower and upper bounds without copying for the built-in data file and falls back to the public getters for external implementations. The returned maps and byte slices are read-only borrows.

func BorrowedDataFileCollections added in v0.7.0 

func BorrowedDataFileCollections(file DataFileCollections) (
	columnSizes map[int]int64,
	keyMetadata []byte,
	splitOffsets []int64,
	equalityFieldIDs []int,
)

func BorrowedDataFilePartition added in v0.7.0 

func BorrowedDataFilePartition(file DataFilePartition) map[int]any

BorrowedDataFilePartition returns a partition map without copying for the built-in data file and falls back to the public getter for external implementations. The returned map and its values must be treated as read-only and must not escape the current operation.

func BorrowedDataFilePointers added in v0.7.0 

func BorrowedDataFilePointers(file DataFilePointers) (
	sortOrderID *int,
	firstRowID *int64,
	referencedDataFile *string,
	contentOffset *int64,
	contentSizeInBytes *int64,
)

BorrowedDataFilePointers returns pointer-valued metadata without copying for the built-in data file and falls back to the public getters for external implementations. The returned pointers must only be dereferenced during the current operation and must not be retained.

func BorrowedDataFileReferencedDataFile added in v0.7.0 

func BorrowedDataFileReferencedDataFile(file DataFileReferencedDataFile) *string

BorrowedDataFileReferencedDataFile returns the optional referenced path without copying for the built-in data file and falls back to the public getter for external implementations. The pointer must only be dereferenced during the current operation and must not be retained.

func BorrowedDataFileStats added in v0.7.0 

func BorrowedDataFileStats(file DataFileStats) (
	valueCounts map[int]int64,
	nullCounts map[int]int64,
	nanCounts map[int]int64,
	lowerBounds map[int][]byte,
	upperBounds map[int][]byte,
)

BorrowedDataFileStats returns statistics without copying for the built-in data file and falls back to the public getters for external implementations. The returned values must be treated as read-only and must not escape the current operation.

func BorrowedManifestFilePartitions added in v0.7.0 

func BorrowedManifestFilePartitions[T any](file ManifestFilePartitions[T]) []T

BorrowedManifestFilePartitions returns partition summaries without copying for the built-in manifest file and falls back to the public getter for external implementations. The returned slice and all nested values are read-only borrows that must not escape the current operation.

func CheckedClose added in v0.4.0 

func CheckedClose(c io.Closer, err *error)

CheckedClose is a helper function to close a resource and return an error if it fails. It is intended to be used in a defer statement.

func Counter added in v0.3.0 

func Counter(start int) iter.Seq[int]

func DecimalNode added in v0.6.0 

func DecimalNode(precision, scale int) avro.SchemaNode

func DecimalRequiredBytes added in v0.2.0 

func DecimalRequiredBytes(precision int) int

DecimalRequiredBytes returns the required number of bytes to store a decimal value of the given precision. If the precision is outside the range (0, 40], this returns -1 as it is invalid.

func DecimalSchema added in v0.2.0 

func DecimalSchema(precision, scale int) *avro.Schema

func DecodeDefaultBytes added in v0.7.0 

func DecodeDefaultBytes(value string, fixedLen int) ([]byte, error)

DecodeDefaultBytes decodes the Iceberg hex representation for binary and fixed defaults. Base64 is accepted only for compatibility with metadata written by iceberg-go v0.6.0. Hex wins when both encodings are valid. fixedLen is -1 for binary and the required width for fixed values.

func Difference added in v0.2.0 

func Difference(a, b []string) []string

Helper function to find the difference between two slices (a - b).

func EncodeDefaultBytes added in v0.7.0 

func EncodeDefaultBytes(value []byte) string

EncodeDefaultBytes returns the Iceberg JSON single-value representation for binary and fixed defaults.

func FixedNode added in v0.6.0 

func FixedNode(size int) avro.SchemaNode

func FixedSchema added in v0.6.0 

func FixedSchema(size int) *avro.Schema

func FloorDiv added in v0.7.0 

func FloorDiv[T constraints.Integer](a, b T) T

FloorDiv performs floored integer division, rounding toward negative infinity. This matches Java's Math.floorDiv behavior for negative dividends.

func MissingRequiredField added in v0.7.0 

func MissingRequiredField(sentinel error, name string) error

func Must added in v0.2.0 

func Must[T any](v T, err error) T

func NewManifestEntrySchema added in v0.2.0 

func NewManifestEntrySchema(partitionType *avro.Schema, version int) (*avro.Schema, error)

func NewManifestFileSchema added in v0.2.0 

func NewManifestFileSchema(version int) (*avro.Schema, error)

func NullableNode added in v0.6.0 

func NullableNode(node avro.SchemaNode) avro.SchemaNode

func NullableSchema added in v0.2.0 

func NullableSchema(schema *avro.Schema) *avro.Schema

func PackingIterator added in v0.3.0 

func PackingIterator[T any](itr iter.Seq[T], targetWeight int64, lookback int, weightFunc func(T) int64, largestBinFirst bool) iter.Seq[[]T]

func RecoverError added in v0.2.0 

func RecoverError(err *error)

func SingleErrorIter added in v0.6.0 

func SingleErrorIter[T any](err error) iter.Seq2[T, error]

func SliceEqualHelper added in v0.5.0 

func SliceEqualHelper[T interface{ Equals(T) bool }](s1, s2 []T) bool

SliceEqualHelper compares the equality of two slices whose elements have an Equals method

func WithFieldID added in v0.2.0 

func WithFieldID(id int) map[string]any

WithFieldID returns a props map for use with SchemaField for Iceberg field IDs.

Types

type Bin added in v0.2.0 

type Bin[T any] struct {
	// contains filtered or unexported fields
}

func (*Bin[T]) Add added in v0.2.0 

func (b *Bin[T]) Add(item T, weight int64)

func (*Bin[T]) CanAdd added in v0.2.0 

func (b *Bin[T]) CanAdd(weight int64) bool

func (*Bin[T]) Weight added in v0.2.0 

func (b *Bin[T]) Weight() int64

type BoundPredicateRef added in v0.7.0 

type BoundPredicateRef struct{}

BoundPredicateRef authorizes the zero-copy bound-predicate visitor path for trusted visitors in this module.

type CountingWriter added in v0.2.0 

type CountingWriter struct {
	Count int64
	W     io.Writer
}

func (*CountingWriter) Write added in v0.2.0 

func (w *CountingWriter) Write(p []byte) (int, error)

type DataFileCollections added in v0.7.0 

type DataFileCollections interface {
	ColumnSizes() map[int]int64
	KeyMetadata() []byte
	SplitOffsets() []int64
	EqualityFieldIDs() []int
}

type DataFileCollectionsRef added in v0.7.0 

type DataFileCollectionsRef interface {
	DataFileCollectionsRef(DataFileRef) (
		columnSizes map[int]int64,
		keyMetadata []byte,
		splitOffsets []int64,
		equalityFieldIDs []int,
	)
}

type DataFilePartition added in v0.7.0 

type DataFilePartition interface {
	Partition() map[int]any
}

DataFilePartition is the public partition surface needed by the internal borrowed-partition helper. It avoids importing the root package here.

type DataFilePartitionRef added in v0.7.0 

type DataFilePartitionRef interface {
	DataFilePartitionRef(DataFileRef) map[int]any
}

DataFilePartitionRef is implemented by the built-in data file to expose its immutable partition map to trusted in-module callers.

type DataFilePointers added in v0.7.0 

type DataFilePointers interface {
	SortOrderID() *int
	FirstRowID() *int64
	ReferencedDataFile() *string
	ContentOffset() *int64
	ContentSizeInBytes() *int64
}

DataFilePointers is the pointer-valued DataFile surface used by internal readers. It avoids importing the root package here.

type DataFilePointersRef added in v0.7.0 

type DataFilePointersRef interface {
	DataFilePointersRef(DataFileRef) (
		sortOrderID *int,
		firstRowID *int64,
		referencedDataFile *string,
		contentOffset *int64,
		contentSizeInBytes *int64,
	)
}

DataFilePointersRef is implemented by the built-in data file to expose its pointer-valued fields without allocating defensive copies.

type DataFileRef added in v0.7.0 

type DataFileRef struct{}

DataFileRef authorizes zero-copy access to immutable DataFile state from trusted packages within this module. Go's internal-package rule prevents external callers from constructing this token.

type DataFileReferencedDataFile added in v0.7.0 

type DataFileReferencedDataFile interface {
	ReferencedDataFile() *string
}

DataFileReferencedDataFile is the public optional reference surface needed by the internal borrowed-reference helper.

type DataFileReferencedDataFileRef added in v0.7.0 

type DataFileReferencedDataFileRef interface {
	DataFileReferencedDataFileRef(DataFileRef) *string
}

DataFileReferencedDataFileRef is implemented by the built-in data file to expose its optional referenced path without allocating a defensive pointer.

type DataFileStats added in v0.7.0 

type DataFileStats interface {
	ValueCounts() map[int]int64
	NullValueCounts() map[int]int64
	NaNValueCounts() map[int]int64
	LowerBoundValues() map[int][]byte
	UpperBoundValues() map[int][]byte
}

DataFileStats is the public statistics surface needed by the internal borrowed-statistics helper. It is intentionally smaller than iceberg.DataFile so this low-level package does not import the root package.

type DataFileStatsRef added in v0.7.0 

type DataFileStatsRef interface {
	DataFileStatsRef(DataFileRef) (
		valueCounts map[int]int64,
		nullCounts map[int]int64,
		nanCounts map[int]int64,
		lowerBounds map[int][]byte,
		upperBounds map[int][]byte,
	)
}

DataFileStatsRef is implemented by the built-in data file to expose its immutable statistics maps to trusted in-module callers.

type ManifestFilePartitions added in v0.7.0 

type ManifestFilePartitions[T any] interface {
	Partitions() []T
}

ManifestFilePartitions is the public partition-summary surface needed by the internal borrowed-partition helper. It is parameterized so this package does not need to import the root package.

type ManifestFileRef added in v0.7.0 

type ManifestFileRef struct{}

ManifestFileRef authorizes zero-copy access to immutable manifest state from trusted packages within this module. Data obtained with this token is borrowed, must be treated as read-only, and must not be retained beyond the current operation. Go's internal-package rule prevents external callers from constructing this token.

type ManifestPartitionBorrower added in v0.7.0 

type ManifestPartitionBorrower[T any] interface {
	ManifestFilePartitionRef(ManifestFileRef) []T
}

ManifestPartitionBorrower is implemented by the built-in manifest file to expose its partition summaries to trusted in-module callers.

type MockFS 

type MockFS struct {
	mock.Mock
}

func (*MockFS) Create added in v0.2.0 

func (m *MockFS) Create(name string) (io.FileWriter, error)

func (*MockFS) Open 

func (m *MockFS) Open(name string) (io.File, error)

func (*MockFS) Remove 

func (m *MockFS) Remove(name string) error

func (*MockFS) WriteFile added in v0.2.0 

func (m *MockFS) WriteFile(name string, content []byte) error

type MockFSReadFile 

type MockFSReadFile struct {
	MockFS
	ErrOnClose bool
}

func (*MockFSReadFile) ReadFile 

func (m *MockFSReadFile) ReadFile(name string) ([]byte, error)

type MockFile 

type MockFile struct {
	Contents   *bytes.Reader
	ErrOnClose bool
	// contains filtered or unexported fields
}

func (*MockFile) Close 

func (m *MockFile) Close() error

func (*MockFile) Read 

func (m *MockFile) Read(p []byte) (int, error)

func (*MockFile) ReadAt 

func (m *MockFile) ReadAt(p []byte, off int64) (n int, err error)

func (*MockFile) ReadFrom added in v0.4.0 

func (m *MockFile) ReadFrom(r sio.Reader) (n int64, err error)

func (*MockFile) Seek 

func (m *MockFile) Seek(offset int64, whence int) (n int64, err error)

func (*MockFile) Stat 

func (m *MockFile) Stat() (fs.FileInfo, error)

func (*MockFile) Write added in v0.4.0 

func (m *MockFile) Write(p []byte) (n int, err error)

type PartitionSpecRef added in v0.7.0 

type PartitionSpecRef struct{}

PartitionSpecRef marks partition-spec access that may return references to internal state. It is restricted to packages within this module by Go's internal package rules.

type RequirementWire added in v0.7.0 

type RequirementWire struct {
	Type *string `json:"type"`
}

type SchemaRef added in v0.7.0 

type SchemaRef struct{}

SchemaRef marks schema access that may return references to internal state. It is restricted to packages within this module by Go's internal package rules.

type SlicePacker added in v0.2.0 

type SlicePacker[T any] struct {
	TargetWeight    int64
	Lookback        int
	LargestBinFirst bool
}

func (*SlicePacker[T]) Pack added in v0.2.0 

func (s *SlicePacker[T]) Pack(items []T, weightFunc func(T) int64) [][]T

func (*SlicePacker[T]) PackEnd added in v0.2.0 

func (s *SlicePacker[T]) PackEnd(items []T, weightFunc func(T) int64) [][]T

Source Files

View all Source files

Directories

Path Synopsis
Package datafileavro is an internal bridge that lets the github.com/apache/iceberg-go/codec package consume the iceberg package's manifest-entry Avro decoder without iceberg having to export it publicly.
 Package datafileavro is an internal bridge that lets the github.com/apache/iceberg-go/codec package consume the iceberg package's manifest-entry Avro decoder without iceberg having to export it publicly.
Package scanmetrics instruments client-side scan planning.
 Package scanmetrics instruments client-side scan planning.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.