Help us shape the future of Go! Take the 2026 Go Developer Survey

iceberg

package
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package iceberg reads and writes Apache Iceberg v2 table metadata, manifest lists and manifests (the table spec, https://iceberg.apache.org/spec/).

Index

Constants

View Source 
const (
	StatusExisting = 0
	StatusAdded    = 1
	StatusDeleted  = 2

	ContentData            = 0
	ContentPositionDeletes = 1
	ContentEqualityDeletes = 2
)

Manifest entry statuses and content types from the table spec.

View Source 
const (
	// FormatVersion is the only table format version this package writes.
	FormatVersion = 2
	// NoSnapshot is the current-snapshot-id of a table without snapshots.
	NoSnapshot = int64(-1)
	// MainBranch is the default branch ref name.
	MainBranch = "main"
)

Variables

View Source 
var (
	// ErrInvalidMetadata is returned for a metadata document that is not valid Iceberg table metadata.
	ErrInvalidMetadata = errors.New("invalid iceberg table metadata")
	// ErrNoSnapshot is returned when a snapshot id or timestamp matches no snapshot.
	ErrNoSnapshot = errors.New("iceberg snapshot not found")
)
View Source 
var ErrInvalidManifest = errors.New("invalid iceberg manifest")

ErrInvalidManifest is returned for an Avro manifest or manifest list that cannot be decoded.

View Source 
var ErrUnsupportedType = errors.New("unsupported iceberg type")

ErrUnsupportedType is returned for a type that cannot be converted or written.

Functions

func AvroTypeOf 

func AvroTypeOf(icebergType string) (string, error)

AvroTypeOf is the Avro primitive a value of the given Iceberg primitive is stored as in a manifest.

func EncodeManifest 

func EncodeManifest(meta *TableMetadata, spec PartitionSpec, content int, entries []ManifestEntry) ([]byte, error)

EncodeManifest writes a manifest of entries written under spec.

func EncodeManifestList 

func EncodeManifestList(snapshotID int64, parent *int64, seq int64, files []ManifestFile) ([]byte, error)

EncodeManifestList writes a manifest list for a snapshot.

func EncodeMetadata 

func EncodeMetadata(m *TableMetadata) ([]byte, error)

EncodeMetadata renders the metadata.json document.

func FromHive 

func FromHive(hive string) (string, error)

FromHive maps an Athena/Hive column type to an Iceberg primitive type name. Nested types are not writable and return ErrUnsupportedType.

func HiveType 

func HiveType(raw json.RawMessage) (string, error)

HiveType renders an Iceberg type as the Athena/Hive type name used in the Glue catalog (long -> bigint, list -> array<..>, struct -> struct<a:int,..>).

func InitialMetadataLocation 

func InitialMetadataLocation(location string) string

InitialMetadataLocation names the first metadata file of a new table.

func NewSnapshotID 

func NewSnapshotID() int64

NewSnapshotID returns a random positive snapshot id.

func PrimitiveName 

func PrimitiveName(raw json.RawMessage) string

PrimitiveName returns the primitive type name of a field type ("" for nested types).

Types

type AppendPlan 

type AppendPlan struct {
	Metadata         *TableMetadata
	MetadataLocation string
	Objects          []Object
}

AppendPlan is a prepared append commit: the metadata, manifest and manifest list objects to store, and the metadata location to commit to the catalog.

type ColumnSpec 

type ColumnSpec struct {
	Name    string
	Type    string
	Comment string
}

ColumnSpec is a column of a table about to be created.

type DataFile 

type DataFile struct {
	Partition   map[string]any
	ValueCounts map[int]int64
	NullCounts  map[int]int64
	Path        string
	Format      string
	EqualityIDs []int
	Content     int
	RecordCount int64
	FileSize    int64
}

DataFile is one data or delete file of a manifest entry.

type Field 

type Field struct {
	Name     string          `json:"name"`
	Doc      string          `json:"doc,omitempty"`
	Type     json.RawMessage `json:"type"`
	ID       int             `json:"id"`
	Required bool            `json:"required"`
}

Field is one schema field; Type is a primitive name or a nested type object.

type ManifestEntry 

type ManifestEntry struct {
	SnapshotID     *int64
	SequenceNumber *int64
	DataFile       DataFile
	Status         int
}

ManifestEntry is a manifest row.

func ReadManifest 

func ReadManifest(data []byte) ([]ManifestEntry, error)

ReadManifest decodes a manifest.

type ManifestFile 

type ManifestFile struct {
	Path              string
	Length            int64
	SequenceNumber    int64
	MinSequenceNumber int64
	AddedSnapshotID   int64
	AddedRows         int64
	ExistingRows      int64
	DeletedRows       int64
	SpecID            int
	Content           int
	AddedFiles        int
	ExistingFiles     int
	DeletedFiles      int
}

ManifestFile is a manifest list row.

func ReadManifestList 

func ReadManifestList(data []byte) ([]ManifestFile, error)

ReadManifestList decodes a manifest list (v1 and v2 column names).

type MetadataLogEntry 

type MetadataLogEntry struct {
	MetadataFile string `json:"metadata-file"`
	TimestampMS  int64  `json:"timestamp-ms"`
}

MetadataLogEntry records a previous metadata file.

type NewParams 

type NewParams struct {
	Now        time.Time
	Properties map[string]string
	Location   string
	SortOrder  json.RawMessage
	Schema     Schema
	Spec       PartitionSpec
}

NewParams describes a table to create.

type Object 

type Object struct {
	Location string
	Body     []byte
}

Object is a file to store alongside a table commit.

type PartitionField 

type PartitionField struct {
	Name      string `json:"name"`
	Transform string `json:"transform"`
	SourceID  int    `json:"source-id"`
	FieldID   int    `json:"field-id"`
}

PartitionField is one partition spec field.

type PartitionSpec 

type PartitionSpec struct {
	Fields []PartitionField `json:"fields"`
	SpecID int              `json:"spec-id"`
}

PartitionSpec is a partitioning of a table.

type Ref 

type Ref struct {
	Type       string `json:"type"`
	SnapshotID int64  `json:"snapshot-id"`
}

Ref is a named snapshot reference.

type Schema 

type Schema struct {
	Type               string  `json:"type"`
	Fields             []Field `json:"fields"`
	IdentifierFieldIDs []int   `json:"identifier-field-ids,omitempty"`
	SchemaID           int     `json:"schema-id"`
}

Schema is an Iceberg struct schema.

func SchemaFromColumns 

func SchemaFromColumns(cols []ColumnSpec) (Schema, error)

SchemaFromColumns builds a schema with sequential field ids; columns are optional (nullable).

type Snapshot 

type Snapshot struct {
	Summary          map[string]string `json:"summary,omitempty"`
	ParentSnapshotID *int64            `json:"parent-snapshot-id,omitempty"`
	SchemaID         *int              `json:"schema-id,omitempty"`
	ManifestList     string            `json:"manifest-list,omitempty"`
	SnapshotID       int64             `json:"snapshot-id"`
	SequenceNumber   int64             `json:"sequence-number"`
	TimestampMS      int64             `json:"timestamp-ms"`
}

Snapshot is one table snapshot.

type SnapshotLogEntry 

type SnapshotLogEntry struct {
	TimestampMS int64 `json:"timestamp-ms"`
	SnapshotID  int64 `json:"snapshot-id"`
}

SnapshotLogEntry records when a snapshot became current.

type TableMetadata 

type TableMetadata struct {
	Properties map[string]string `json:"properties"`
	Refs       map[string]Ref    `json:"refs,omitempty"`

	TableUUID          string             `json:"table-uuid"`
	Location           string             `json:"location"`
	Schemas            []Schema           `json:"schemas"`
	PartitionSpecs     []PartitionSpec    `json:"partition-specs"`
	SortOrders         []json.RawMessage  `json:"sort-orders"`
	Snapshots          []Snapshot         `json:"snapshots"`
	SnapshotLog        []SnapshotLogEntry `json:"snapshot-log"`
	MetadataLog        []MetadataLogEntry `json:"metadata-log"`
	LastSequenceNumber int64              `json:"last-sequence-number"`
	LastUpdatedMS      int64              `json:"last-updated-ms"`
	CurrentSnapshotID  int64              `json:"current-snapshot-id"`
	FormatVersion      int                `json:"format-version"`
	LastColumnID       int                `json:"last-column-id"`
	CurrentSchemaID    int                `json:"current-schema-id"`
	DefaultSpecID      int                `json:"default-spec-id"`
	LastPartitionID    int                `json:"last-partition-id"`
	DefaultSortOrderID int                `json:"default-sort-order-id"`
	// contains filtered or unexported fields
}

TableMetadata is a table's metadata.json document. Fields this package does not model survive a decode/encode round trip.

func NewTableMetadata 

func NewTableMetadata(p NewParams) *TableMetadata

NewTableMetadata builds the v2 metadata of a table with no snapshots.

func ParseMetadata 

func ParseMetadata(data []byte) (*TableMetadata, error)

ParseMetadata decodes and validates a metadata.json document.

func (*TableMetadata) AppendSnapshot 

func (m *TableMetadata) AppendSnapshot(snap Snapshot, previousMetadataFile string, now time.Time) *TableMetadata

AppendSnapshot returns a copy of m with snap added as the current snapshot of main. previousMetadataFile is recorded in the metadata log.

func (*TableMetadata) CurrentSchema 

func (m *TableMetadata) CurrentSchema() Schema

CurrentSchema is the table's current schema.

func (*TableMetadata) CurrentSnapshot 

func (m *TableMetadata) CurrentSnapshot() (Snapshot, bool)

CurrentSnapshot returns the current snapshot, if the table has one.

func (*TableMetadata) DefaultSpec 

func (m *TableMetadata) DefaultSpec() PartitionSpec

DefaultSpec is the spec new data is written with.

func (*TableMetadata) MarshalJSON 

func (m *TableMetadata) MarshalJSON() ([]byte, error)

MarshalJSON encodes the modeled members over the preserved extras.

func (*TableMetadata) NextMetadataLocation 

func (m *TableMetadata) NextMetadataLocation() string

NextMetadataLocation names this metadata's own file, numbered by its metadata-log length.

func (*TableMetadata) PlanAppend 

func (m *TableMetadata) PlanAppend(
	currentLocation string, prev []ManifestFile, files []DataFile, now time.Time,
) (*AppendPlan, error)

PlanAppend prepares a commit appending files to the table. currentLocation is the metadata file m was read from; prev are the current snapshot's manifests.

func (*TableMetadata) SchemaByID 

func (m *TableMetadata) SchemaByID(id int) (Schema, bool)

SchemaByID returns the schema with the given id.

func (*TableMetadata) SnapshotAsOf 

func (m *TableMetadata) SnapshotAsOf(t time.Time) (Snapshot, error)

SnapshotAsOf returns the snapshot that was current at t: the last snapshot-log entry at or before t.

func (*TableMetadata) SnapshotByID 

func (m *TableMetadata) SnapshotByID(id int64) (Snapshot, bool)

SnapshotByID returns the snapshot with the given id.

func (*TableMetadata) SpecByID 

func (m *TableMetadata) SpecByID(id int) (PartitionSpec, bool)

SpecByID returns the partition spec with the given id.

func (*TableMetadata) UnmarshalJSON 

func (m *TableMetadata) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes a v1 or v2 document and keeps unmodeled members.

type Transform 

type Transform struct {
	Name  string
	Width int
}

Transform is a parsed partition transform.

func ParseTransform 

func ParseTransform(s string) (Transform, error)

ParseTransform parses identity, year, month, day, hour, void, bucket[N] and truncate[W].

func (Transform) Apply 

func (t Transform) Apply(source string, raw *string) (any, error)

Apply transforms the textual value of a source column. A nil result is null. Results are bool, int32, int64, float32, float64 or string.

func (Transform) ResultType 

func (t Transform) ResultType(source string) string

ResultType is the Iceberg type of the transform's output for a source type.

func (Transform) String 

func (t Transform) String() string

String renders the transform as it appears in a partition spec.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.