Help us shape the future of Go! Take the 2026 Go Developer Survey

paimon

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package paimon provides a Go binding for Apache Paimon Rust.

This binding uses purego and libffi to call into the paimon-c shared library. The pre-built shared library is embedded in the package and automatically loaded at runtime — no manual build step needed.

This package requires CGO because it imports the arrow-go cdata package for Arrow C Data Interface support.

Basic usage: 

// Create a catalog with options
catalog, err := paimon.NewCatalog(map[string]string{
	"warehouse": "/path/to/warehouse",
})
if err != nil { log.Fatal(err) }
defer catalog.Close()

table, err := catalog.GetTable(paimon.NewIdentifier("default", "my_table"))
if err != nil { log.Fatal(err) }
defer table.Close()

For S3 or OSS warehouses, pass the appropriate credentials: 

// S3
catalog, _ := paimon.NewCatalog(map[string]string{
	"warehouse":            "s3://bucket/warehouse",
	"s3.access-key-id":     "...",
	"s3.secret-access-key": "...",
	"s3.region":            "us-east-1",
})

// OSS
catalog, _ := paimon.NewCatalog(map[string]string{
	"warehouse":            "oss://bucket/warehouse",
	"fs.oss.accessKeyId":     "...",
	"fs.oss.accessKeySecret": "...",
	"fs.oss.endpoint":        "oss-cn-hangzhou.aliyuncs.com",
})

Index

Constants

This section is empty.

Variables

View Source 
var ErrClosed = errors.New("paimon: use of closed resource")

ErrClosed is returned when an operation is attempted on a closed resource.

View Source 
var ErrNegativeLimit = errors.New("paimon: read-builder limit must not be negative")

ErrNegativeLimit is returned by ReadBuilder.WithLimit when the limit is negative. A negative Go int would wrap to a huge value through the unsigned C boundary and make scan planning stop after the first split, silently dropping rows, so it is rejected up front.

Functions

func StringBlobMapDescriptors added in v0.4.0 

func StringBlobMapDescriptors(column arrow.Array, row int) (map[string][]byte, error)

StringBlobMapDescriptors returns one MAP<STRING, BLOB> row.

Types

type BlobReader added in v0.4.0 

type BlobReader struct {
	// contains filtered or unexported fields
}

BlobReader resolves serialized BlobDescriptors.

func NewBlobReader added in v0.4.0 

func NewBlobReader(storageOptions map[string]string) (*BlobReader, error)

NewBlobReader creates a descriptor reader with FileIO options.

func (*BlobReader) Close added in v0.4.0 

func (r *BlobReader) Close()

Close releases the reader and is idempotent.

func (*BlobReader) OpenBlob added in v0.4.0 

func (r *BlobReader) OpenBlob(descriptor []byte) (*BlobStream, error)

OpenBlob opens one descriptor without reading its contents.

func (*BlobReader) ReadBlob added in v0.4.0 

func (r *BlobReader) ReadBlob(descriptor []byte) ([]byte, error)

ReadBlob resolves one descriptor.

func (*BlobReader) ReadBlobs added in v0.4.0 

func (r *BlobReader) ReadBlobs(descriptors [][]byte) ([][]byte, error)

ReadBlobs resolves a batch in input order.

type BlobStream added in v0.4.0 

type BlobStream struct {
	// contains filtered or unexported fields
}

BlobStream incrementally reads one BlobDescriptor.

func (*BlobStream) Close added in v0.4.0 

func (s *BlobStream) Close() error

Close releases the stream and is idempotent.

func (*BlobStream) Read added in v0.4.0 

func (s *BlobStream) Read(buffer []byte) (int, error)

Read implements io.Reader.

func (*BlobStream) Seek added in v0.4.0 

func (s *BlobStream) Seek(offset int64, whence int) (int64, error)

Seek implements io.Seeker within the descriptor's range.

type Bytes 

type Bytes []byte

Bytes represents a binary value. Usage: pb.Eq("data", paimon.Bytes(someSlice))

type Catalog 

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog wraps a paimon Catalog.

func NewCatalog 

func NewCatalog(options map[string]string) (*Catalog, error)

NewCatalog creates a new Catalog using the CatalogFactory with the given options. The catalog type is determined by the "metastore" option (default: "filesystem").

Common options:

  • "warehouse": The warehouse path (required)
  • "metastore": Catalog type - "filesystem" (default) or "rest"
  • "uri": REST catalog server URI (required for REST catalog)
  • "s3.access-key-id", "s3.secret-access-key", "s3.region": S3 credentials
  • "fs.oss.accessKeyId", "fs.oss.accessKeySecret", "fs.oss.endpoint": OSS credentials

func (*Catalog) Close 

func (c *Catalog) Close()

Close releases the catalog resources. Safe to call multiple times.

func (*Catalog) CreateTag added in v0.4.0 

func (c *Catalog) CreateTag(
	id Identifier,
	tagName string,
	snapshotID *int64,
	ignoreIfExists bool,
) error

CreateTag creates a tag for snapshotID, or the latest snapshot when snapshotID is nil.

func (*Catalog) DeleteTag added in v0.4.0 

func (c *Catalog) DeleteTag(id Identifier, tagName string) error

DeleteTag deletes a tag.

func (*Catalog) GetTable 

func (c *Catalog) GetTable(id Identifier) (*Table, error)

GetTable retrieves a table from the catalog using the given identifier.

func (*Catalog) GetTag added in v0.4.0 

func (c *Catalog) GetTag(id Identifier, tagName string) (GetTagResponse, error)

GetTag returns a tag and its snapshot metadata.

func (*Catalog) ListDatabases added in v0.4.0 

func (c *Catalog) ListDatabases() ([]string, error)

ListDatabases returns the names of all databases in the catalog.

func (*Catalog) ListTables added in v0.4.0 

func (c *Catalog) ListTables(database string) ([]string, error)

ListTables returns the names of all tables in the given database.

type CommitKind added in v0.4.0 

type CommitKind string

CommitKind identifies the change represented by a snapshot. 

const (
	CommitKindAppend    CommitKind = "APPEND"
	CommitKindCompact   CommitKind = "COMPACT"
	CommitKindOverwrite CommitKind = "OVERWRITE"
	CommitKindAnalyze   CommitKind = "ANALYZE"
)

type CommitMessages added in v0.4.0 

type CommitMessages struct {
	// contains filtered or unexported fields
}

CommitMessages contains the files produced by one or more writers. It is a process-local native handle and cannot be transferred between processes.

func (*CommitMessages) Close added in v0.4.0 

func (m *CommitMessages) Close()

Close releases the commit messages. Safe to call multiple times.

func (*CommitMessages) Merge added in v0.4.0 

func (m *CommitMessages) Merge(source *CommitMessages) error

Merge appends a copy of source's messages; both handles stay valid and must share a table and commit user. It does not establish fixed-bucket ownership.

type DataSplit 

type DataSplit struct {
	// contains filtered or unexported fields
}

DataSplit identifies a single data split within a plan. DataSplits keep the underlying plan data alive via GC-attached reference counting, so they are safe to use independently.

type Date 

type Date int32

Date represents a date value as epoch days since 1970-01-01. Usage: pb.Eq("dt", paimon.Date(19000))

type Datum 

type Datum struct {
	// contains filtered or unexported fields
}

Datum is a typed literal value for predicate comparison. The internal representation is hidden to allow future changes (e.g. switching to opaque handles) without breaking callers.

func BoolDatum 

func BoolDatum(v bool) Datum

BoolDatum creates a boolean datum.

func DoubleDatum 

func DoubleDatum(v float64) Datum

DoubleDatum creates a double datum.

func FloatDatum 

func FloatDatum(v float32) Datum

FloatDatum creates a float datum.

func IntDatum 

func IntDatum(v int32) Datum

IntDatum creates an int datum.

func LongDatum 

func LongDatum(v int64) Datum

LongDatum creates a long (bigint) datum.

func SmallIntDatum 

func SmallIntDatum(v int16) Datum

SmallIntDatum creates a smallint datum.

func StringDatum 

func StringDatum(v string) Datum

StringDatum creates a string datum.

func TinyIntDatum 

func TinyIntDatum(v int8) Datum

TinyIntDatum creates a tinyint datum.

type Decimal 

type Decimal struct {
	Lo        int64 // low 64 bits of unscaled i128 (unsigned interpretation)
	Hi        int64 // high 64 bits of unscaled i128 (sign extension)
	Precision uint32
	Scale     uint32
}

Decimal represents a fixed-precision decimal value up to DECIMAL(38, s).

The unscaled value is stored as a little-endian i128 split into two int64 halves: Lo (low 64 bits, unsigned interpretation) and Hi (high 64 bits, sign-extended). For values that fit in int64, use NewDecimal.

Usage: 

paimon.NewDecimal(12345, 10, 2)          // 123.45 as DECIMAL(10,2)
paimon.Decimal{Lo: lo, Hi: hi, ...}      // full i128

func NewDecimal 

func NewDecimal(unscaled int64, precision, scale uint32) Decimal

NewDecimal creates a Decimal from an int64 unscaled value. For unscaled values that exceed int64 range, construct Decimal directly with Lo/Hi fields.

type Error 

type Error struct {
	// contains filtered or unexported fields
}

Error represents a paimon error with code and message.

func (*Error) Code 

func (e *Error) Code() ErrorCode

Code returns the error code.

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Message 

func (e *Error) Message() string

Message returns the error message.

type ErrorCode 

type ErrorCode int32

ErrorCode represents categories of errors from paimon. 

const (
	CodeUnexpected        ErrorCode = 0
	CodeUnsupported       ErrorCode = 1
	CodeNotFound          ErrorCode = 2
	CodeAlreadyExist      ErrorCode = 3
	CodeInvalidInput      ErrorCode = 4
	CodeIoError           ErrorCode = 5
	CodeResourceExhausted ErrorCode = 6
)

type FFI 

type FFI[T any] struct {
	// contains filtered or unexported fields
}

FFI is a generic type-safe wrapper for a foreign function.

type GetTagResponse added in v0.4.0 

type GetTagResponse struct {
	TagName         string   `json:"tagName"`
	Snapshot        Snapshot `json:"snapshot"`
	TagCreateTime   *int64   `json:"tagCreateTime,omitempty"`
	TagTimeRetained *string  `json:"tagTimeRetained,omitempty"`
}

GetTagResponse contains a tag and its snapshot metadata.

type Identifier 

type Identifier struct {
	// contains filtered or unexported fields
}

Identifier identifies a table by database and object name.

func NewIdentifier 

func NewIdentifier(database, object string) Identifier

NewIdentifier creates a new Identifier with the given database and object name.

type LocalZonedTimestamp 

type LocalZonedTimestamp struct {
	Millis int64
	Nanos  int32
}

LocalZonedTimestamp represents a timestamp with local timezone semantics. Usage: pb.Eq("lzts", paimon.LocalZonedTimestamp{Millis: 1700000000000, Nanos: 0})

type Plan 

type Plan struct {
	// contains filtered or unexported fields
}

Plan holds the result of a table scan, containing data splits to read.

func (*Plan) Close 

func (p *Plan) Close()

Close releases the plan resources. Safe to call multiple times. DataSplits obtained from Splits() remain valid after Close.

func (*Plan) NumSplits 

func (p *Plan) NumSplits() int

NumSplits returns the number of data splits in this plan.

func (*Plan) Splits 

func (p *Plan) Splits() []DataSplit

Splits returns all data splits in this plan. The returned DataSplits keep the underlying plan data alive via GC-attached reference counting, so they remain valid even after Plan.Close() is called.

type PostponeFixedBucketCommitMessages added in v0.4.0 

type PostponeFixedBucketCommitMessages struct {
	// contains filtered or unexported fields
}

PostponeFixedBucketCommitMessages contains files produced by fixed-bucket writers. It is a process-local native handle and cannot be transferred between processes or passed to a standard TableCommit.

func (*PostponeFixedBucketCommitMessages) Close added in v0.4.0 

func (m *PostponeFixedBucketCommitMessages) Close()

Close releases the messages. Safe to call multiple times.

func (*PostponeFixedBucketCommitMessages) Merge added in v0.4.0 

func (m *PostponeFixedBucketCommitMessages) Merge(
	source *PostponeFixedBucketCommitMessages,
) error

Merge appends a copy of source's messages. Both handles must belong to the same process, and both builders must use the same table, commit user, and overwrite mode.

type PostponeFixedBucketTableCommit added in v0.4.0 

type PostponeFixedBucketTableCommit struct {
	// contains filtered or unexported fields
}

PostponeFixedBucketTableCommit commits fixed-bucket messages using the mode selected on its builder.

func (*PostponeFixedBucketTableCommit) Abort added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) Abort(
	messages *PostponeFixedBucketCommitMessages,
) error

Abort deletes files for an explicitly abandoned, known-uncommitted write. Never call when the commit outcome is unknown.

func (*PostponeFixedBucketTableCommit) Close added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) Close()

Close releases the committer resources. Safe to call multiple times.

func (*PostponeFixedBucketTableCommit) Commit added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) Commit(
	messages *PostponeFixedBucketCommitMessages,
) error

Commit persists fixed-bucket messages using the builder's append or overwrite mode.

func (*PostponeFixedBucketTableCommit) CommitWithIdentifier added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) CommitWithIdentifier(
	messages *PostponeFixedBucketCommitMessages,
	commitIdentifier int64,
) error

CommitWithIdentifier commits with a stable identifier.

func (*PostponeFixedBucketTableCommit) FilterAndCommitWithIdentifier added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) FilterAndCommitWithIdentifier(
	messages *PostponeFixedBucketCommitMessages,
	commitIdentifier int64,
) error

FilterAndCommitWithIdentifier makes a retry idempotent.

func (*PostponeFixedBucketTableCommit) TruncateTable added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) TruncateTable() error

TruncateTable removes all table data.

func (*PostponeFixedBucketTableCommit) TruncateTableWithIdentifier added in v0.4.0 

func (tc *PostponeFixedBucketTableCommit) TruncateTableWithIdentifier(
	commitIdentifier int64,
) error

TruncateTableWithIdentifier removes all table data with a stable identifier.

type PostponeFixedBucketTableWrite added in v0.4.0 

type PostponeFixedBucketTableWrite struct {
	// contains filtered or unexported fields
}

PostponeFixedBucketTableWrite writes rows according to a resolved bucket plan.

func (*PostponeFixedBucketTableWrite) Close added in v0.4.0 

func (tw *PostponeFixedBucketTableWrite) Close()

Close releases the writer resources. Safe to call multiple times.

func (*PostponeFixedBucketTableWrite) PrepareCommit added in v0.4.0 

func (tw *PostponeFixedBucketTableWrite) PrepareCommit() (*PostponeFixedBucketCommitMessages, error)

PrepareCommit finalizes pending writes and returns fixed-bucket messages. It consumes the writer; subsequent operations return ErrClosed.

func (*PostponeFixedBucketTableWrite) WriteArrowBatch added in v0.4.0 

func (tw *PostponeFixedBucketTableWrite) WriteArrowBatch(record arrow.Record) error

WriteArrowBatch writes one Arrow record batch. The caller retains ownership.

type PostponeFixedBucketWriteBuilder added in v0.4.0 

type PostponeFixedBucketWriteBuilder struct {
	// contains filtered or unexported fields
}

PostponeFixedBucketWriteBuilder creates fixed-bucket writers for bucket=-2 tables. A resolved bucket plan must be supplied before NewWrite.

func (*PostponeFixedBucketWriteBuilder) Close added in v0.4.0 

func (wb *PostponeFixedBucketWriteBuilder) Close()

Close releases the builder resources. Safe to call multiple times.

func (*PostponeFixedBucketWriteBuilder) NewCommit added in v0.4.0 

func (wb *PostponeFixedBucketWriteBuilder) NewCommit() (*PostponeFixedBucketTableCommit, error)

NewCommit creates a committer using this builder's commit identity and mode.

func (*PostponeFixedBucketWriteBuilder) NewWrite added in v0.4.0 

func (wb *PostponeFixedBucketWriteBuilder) NewWrite() (*PostponeFixedBucketTableWrite, error)

NewWrite creates a fixed-bucket writer. WithBucketPlan must be called first.

func (*PostponeFixedBucketWriteBuilder) WithBucketPlan added in v0.4.0 

func (wb *PostponeFixedBucketWriteBuilder) WithBucketPlan(plan arrow.Record) error

WithBucketPlan sets a resolved partition-to-bucket-count plan. The plan must contain the table partition columns followed by a non-null Int32 total_buckets column. The caller retains ownership of plan.

func (*PostponeFixedBucketWriteBuilder) WithOverwrite added in v0.4.0 

func (wb *PostponeFixedBucketWriteBuilder) WithOverwrite() error

WithOverwrite enables overwrite mode for both writers and committers created by this builder.

func (*PostponeFixedBucketWriteBuilder) WithResources added in v0.4.0 

func (wb *PostponeFixedBucketWriteBuilder) WithResources(resources *ResourceContext) error

WithResources shares the context's reservation budget with writers from this builder. The builder retains the context after the caller closes its handle.

type Predicate 

type Predicate struct {
	// contains filtered or unexported fields
}

Predicate is an opaque filter predicate for scan planning.

func (*Predicate) And 

func (p *Predicate) And(other *Predicate) (*Predicate, error)

And combines this predicate with another using AND. Consumes both predicates (callers must NOT close either after this call).

func (*Predicate) Close 

func (p *Predicate) Close()

Close releases the predicate resources. Safe to call multiple times. Note: predicates passed to WithFilter or combinators (And/Or/Not) are consumed and should NOT be closed by the caller.

func (*Predicate) Not 

func (p *Predicate) Not() (*Predicate, error)

Not negates this predicate. Consumes the input (caller must NOT close it after this call).

func (*Predicate) Or 

func (p *Predicate) Or(other *Predicate) (*Predicate, error)

Or combines this predicate with another using OR. Consumes both predicates (callers must NOT close either after this call).

type PredicateBuilder 

type PredicateBuilder struct {
	// contains filtered or unexported fields
}

PredicateBuilder creates filter predicates for a table. It holds a Go-level reference to the Table and does not own any C resources, so there is no Close() method.

func (*PredicateBuilder) Between added in v0.4.0 

func (pb *PredicateBuilder) Between(column string, low, high any) (*Predicate, error)

Between creates an inclusive range predicate: low <= column <= high.

func (*PredicateBuilder) Contains added in v0.4.0 

func (pb *PredicateBuilder) Contains(column string, substr string) (*Predicate, error)

Contains creates a predicate matching rows where the string column contains substr.

func (*PredicateBuilder) EndsWith added in v0.4.0 

func (pb *PredicateBuilder) EndsWith(column string, suffix string) (*Predicate, error)

EndsWith creates a predicate matching rows where the string column ends with suffix.

func (*PredicateBuilder) Eq 

func (pb *PredicateBuilder) Eq(column string, value any) (*Predicate, error)

Eq creates an equality predicate: column = value.

func (*PredicateBuilder) Ge 

func (pb *PredicateBuilder) Ge(column string, value any) (*Predicate, error)

Ge creates a greater-or-equal predicate: column >= value.

func (*PredicateBuilder) Gt 

func (pb *PredicateBuilder) Gt(column string, value any) (*Predicate, error)

Gt creates a greater-than predicate: column > value.

func (*PredicateBuilder) In 

func (pb *PredicateBuilder) In(column string, values ...any) (*Predicate, error)

In creates an IN predicate: column IN (values...).

func (*PredicateBuilder) IsNotNull 

func (pb *PredicateBuilder) IsNotNull(column string) (*Predicate, error)

IsNotNull creates an IS NOT NULL predicate.

func (*PredicateBuilder) IsNull 

func (pb *PredicateBuilder) IsNull(column string) (*Predicate, error)

IsNull creates an IS NULL predicate.

func (*PredicateBuilder) Le 

func (pb *PredicateBuilder) Le(column string, value any) (*Predicate, error)

Le creates a less-or-equal predicate: column <= value.

func (*PredicateBuilder) Like added in v0.4.0 

func (pb *PredicateBuilder) Like(column string, pattern string) (*Predicate, error)

Like creates a SQL LIKE predicate: column LIKE pattern, using the default escape character.

func (*PredicateBuilder) Lt 

func (pb *PredicateBuilder) Lt(column string, value any) (*Predicate, error)

Lt creates a less-than predicate: column < value.

func (*PredicateBuilder) NotBetween added in v0.4.0 

func (pb *PredicateBuilder) NotBetween(column string, low, high any) (*Predicate, error)

NotBetween creates a predicate: column < low OR column > high.

func (*PredicateBuilder) NotEq 

func (pb *PredicateBuilder) NotEq(column string, value any) (*Predicate, error)

NotEq creates a not-equal predicate: column != value.

func (*PredicateBuilder) NotIn 

func (pb *PredicateBuilder) NotIn(column string, values ...any) (*Predicate, error)

NotIn creates a NOT IN predicate: column NOT IN (values...).

func (*PredicateBuilder) StartsWith added in v0.4.0 

func (pb *PredicateBuilder) StartsWith(column string, prefix string) (*Predicate, error)

StartsWith creates a predicate matching rows where the string column starts with prefix.

func (*PredicateBuilder) WithCaseSensitive added in v0.4.0 

func (pb *PredicateBuilder) WithCaseSensitive(caseSensitive bool) *PredicateBuilder

WithCaseSensitive returns a builder that matches column names by ASCII case folding when caseSensitive is false, rejecting a name that folds onto two schema columns as ambiguous. The default is true (exact match); the receiver is left unchanged.

The choice is fixed when each predicate is built, because that is when the core resolves the column. ReadBuilder.WithCaseSensitive is the same switch for projection and has no bearing on a predicate built here.

type ReadBuilder 

type ReadBuilder struct {
	// contains filtered or unexported fields
}

ReadBuilder creates TableScan and TableRead instances.

func (*ReadBuilder) Close 

func (rb *ReadBuilder) Close()

Close releases the read builder resources. Safe to call multiple times.

func (*ReadBuilder) NewRead 

func (rb *ReadBuilder) NewRead() (*TableRead, error)

NewRead creates a TableRead for reading data from splits.

func (*ReadBuilder) NewScan 

func (rb *ReadBuilder) NewScan() (*TableScan, error)

NewScan creates a TableScan for planning which data files to read.

func (*ReadBuilder) WithCaseSensitive added in v0.4.0 

func (rb *ReadBuilder) WithCaseSensitive(caseSensitive bool) error

WithCaseSensitive sets whether the names given to WithProjection must match the schema exactly. The default is true. With false, names are matched by ASCII case folding, and a name that folds onto two different schema columns is rejected as ambiguous. Either way the returned records carry the schema's own spelling, not the requested one.

This does not affect predicates. A predicate resolves its column when it is built, so its case sensitivity comes from the builder that produced it — see PredicateBuilder.WithCaseSensitive — and is unaffected by this setting. Call order relative to WithProjection does not matter: projection names are resolved in NewRead.

func (*ReadBuilder) WithFilter 

func (rb *ReadBuilder) WithFilter(p *Predicate) error

WithFilter sets a filter predicate for scan planning and read-side pruning.

The predicate is used in two phases:

  • Scan planning: prunes partitions, buckets, and data files based on file-level statistics (min/max). This is conservative — files whose statistics are inconclusive are kept.
  • Read-side: applies row-level filtering via Parquet native row filters for supported leaf predicates (Eq, NotEq, Lt, Le, Gt, Ge, IsNull, IsNotNull, In, NotIn).

Row-level filtering is exact for most common types (Bool, Int, Long, Float, Double, String, Date, Decimal, Binary). However, the following cases are NOT filtered at the row level and may return non-matching rows:

  • Compound predicates (And/Or/Not) — not yet implemented for row-level filtering.
  • Time, Timestamp, and LocalZonedTimestamp columns (not yet implemented).
  • Schema-evolution: the predicate column does not exist in older data files.
  • Data-evolution mode (data-evolution.enabled = true).

In these cases callers should apply residual filtering on the returned records.

The predicate is consumed (ownership transferred to the read builder); the caller must NOT close it after this call. Passing nil is a no-op.

func (*ReadBuilder) WithLimit added in v0.4.0 

func (rb *ReadBuilder) WithLimit(limit int) error

WithLimit sets a row-count limit hint for scan planning. Planning stops retaining splits once the ones already kept cover the limit, so reading a preview or sample of the first N rows keeps fewer splits. Planning still reads the manifest entries to learn each split's row count, so this bounds how many splits are retained rather than avoiding all planning I/O. It is a hint only: it does not guarantee exactly limit rows are returned, so callers still enforce the final row limit when reading.

A negative limit is rejected with ErrNegativeLimit: it would wrap to a huge value through the unsigned C boundary and stop planning after the first split.

func (*ReadBuilder) WithProjection 

func (rb *ReadBuilder) WithProjection(columns []string) error

WithProjection sets column projection by name. Output order follows the caller-specified order. A name that matches no schema column under any case sensitivity is rejected immediately; case-dependent errors and duplicate names cause NewRead() to fail. An empty list is a valid zero-column projection.

func (*ReadBuilder) WithResources added in v0.4.0 

func (rb *ReadBuilder) WithResources(resources *ResourceContext) error

WithResources shares the context's reservation budget with reads from this builder. The builder retains the context after the caller closes its handle.

type RecordBatchReader 

type RecordBatchReader struct {
	// contains filtered or unexported fields
}

RecordBatchReader iterates over Arrow record batches one at a time via the Arrow C Data Interface (zero-copy). Call NextRecord to advance and Close when done.

func (*RecordBatchReader) Close 

func (r *RecordBatchReader) Close()

Close releases the underlying C record batch readers. Safe to call multiple times and concurrently with NextRecord.

func (*RecordBatchReader) NextRecord 

func (r *RecordBatchReader) NextRecord() (arrow.Record, error)

NextRecord returns the next Arrow record, or io.EOF when iteration is complete. The underlying C batch is imported via the Arrow C Data Interface and released automatically — the caller only needs to call Release on the returned arrow.Record when done. Calls on one reader are serialized; use separate readers for parallel reads.

type ResourceContext added in v0.4.0 

type ResourceContext struct {
	// contains filtered or unexported fields
}

ResourceContext shares a memory reservation budget across readers and writers. Reservations are accounting estimates, not a process memory limit.

func NewResourceContext added in v0.4.0 

func NewResourceContext(memoryLimitBytes uintptr) (*ResourceContext, error)

NewResourceContext creates a shared reservation budget in bytes. A zero limit rejects nonempty reservations.

func (*ResourceContext) Close added in v0.4.0 

func (r *ResourceContext) Close()

Close releases this handle. Builders retain their own resource context clones.

func (*ResourceContext) Metrics added in v0.4.0 

func (r *ResourceContext) Metrics() (ResourceMetrics, error)

Metrics reads the current and peak reservations. The counters are sampled independently.

type ResourceMetrics added in v0.4.0 

type ResourceMetrics struct {
	ReservedMemoryBytes     uintptr
	PeakReservedMemoryBytes uintptr
}

ResourceMetrics reports current and peak reserved bytes.

type Snapshot added in v0.4.0 

type Snapshot struct {
	ID               int64      `json:"id"`
	CommitKind       CommitKind `json:"commitKind"`
	TimeMillis       int64      `json:"timeMillis"`
	TotalRecordCount *int64     `json:"totalRecordCount,omitempty"`
	DeltaRecordCount *int64     `json:"deltaRecordCount,omitempty"`
}

Snapshot describes a Paimon table snapshot.

type Table 

type Table struct {
	// contains filtered or unexported fields
}

Table represents a paimon table.

func (*Table) Close 

func (t *Table) Close()

Close releases the table resources. Safe to call multiple times.

func (*Table) LatestSnapshot added in v0.4.0 

func (t *Table) LatestSnapshot() (*Snapshot, error)

LatestSnapshot returns the latest snapshot, or nil when the table is empty.

func (*Table) NewBlobReader added in v0.4.0 

func (t *Table) NewBlobReader() (*BlobReader, error)

NewBlobReader creates a descriptor reader using this table's FileIO.

func (*Table) NewPostponeFixedBucketWriteBuilder added in v0.4.0 

func (t *Table) NewPostponeFixedBucketWriteBuilder() (*PostponeFixedBucketWriteBuilder, error)

NewPostponeFixedBucketWriteBuilder creates an explicitly selected fixed-bucket builder for a postpone table.

func (*Table) NewPostponeFixedBucketWriteBuilderWithCommitUser added in v0.4.0 

func (t *Table) NewPostponeFixedBucketWriteBuilderWithCommitUser(
	commitUser string,
) (*PostponeFixedBucketWriteBuilder, error)

NewPostponeFixedBucketWriteBuilderWithCommitUser creates a fixed-bucket builder with a stable commit identity.

func (*Table) NewReadBuilder 

func (t *Table) NewReadBuilder() (*ReadBuilder, error)

NewReadBuilder creates a ReadBuilder for this table.

func (*Table) NewReadBuilderWithOptions added in v0.4.0 

func (t *Table) NewReadBuilderWithOptions(options map[string]string) (*ReadBuilder, error)

NewReadBuilderWithOptions creates a ReadBuilder with per-read options.

func (*Table) NewWriteBuilder added in v0.4.0 

func (t *Table) NewWriteBuilder() (*WriteBuilder, error)

NewWriteBuilder creates a write builder with a generated commit identity.

func (*Table) NewWriteBuilderWithCommitUser added in v0.4.0 

func (t *Table) NewWriteBuilderWithCommitUser(commitUser string) (*WriteBuilder, error)

NewWriteBuilderWithCommitUser creates a write builder with a stable commit identity for merging messages across writers and for identifier retries.

func (*Table) PredicateBuilder 

func (t *Table) PredicateBuilder() *PredicateBuilder

PredicateBuilder returns a builder for creating filter predicates on this table.

type TableCommit added in v0.4.0 

type TableCommit struct {
	// contains filtered or unexported fields
}

TableCommit persists or aborts prepared commit messages.

func (*TableCommit) Abort added in v0.4.0 

func (tc *TableCommit) Abort(messages *CommitMessages) error

Abort deletes files for an explicitly abandoned, known-uncommitted write. Never call when the commit outcome is unknown.

func (*TableCommit) Close added in v0.4.0 

func (tc *TableCommit) Close()

Close releases the committer resources. Safe to call multiple times.

func (*TableCommit) Commit added in v0.4.0 

func (tc *TableCommit) Commit(messages *CommitMessages) error

Commit persists messages using the builder's append or overwrite mode.

func (*TableCommit) CommitWithIdentifier added in v0.4.0 

func (tc *TableCommit) CommitWithIdentifier(messages *CommitMessages, commitIdentifier int64) error

CommitWithIdentifier commits with a monotonically increasing identifier.

func (*TableCommit) FilterAndCommitWithIdentifier added in v0.4.0 

func (tc *TableCommit) FilterAndCommitWithIdentifier(
	messages *CommitMessages,
	commitIdentifier int64,
) error

FilterAndCommitWithIdentifier skips messages already committed under commitIdentifier, making retries idempotent. Identifier commits always filter in overwrite mode.

func (*TableCommit) TruncateTable added in v0.4.0 

func (tc *TableCommit) TruncateTable() error

TruncateTable removes all table data.

func (*TableCommit) TruncateTableWithIdentifier added in v0.4.0 

func (tc *TableCommit) TruncateTableWithIdentifier(commitIdentifier int64) error

TruncateTableWithIdentifier truncates with a stable commit identifier.

type TableRead 

type TableRead struct {
	// contains filtered or unexported fields
}

TableRead reads data from splits produced by a TableScan.

func (*TableRead) Close 

func (tr *TableRead) Close()

Close releases the table read resources. Safe to call multiple times.

func (*TableRead) NewRecordBatchReader 

func (tr *TableRead) NewRecordBatchReader(splits []DataSplit) (*RecordBatchReader, error)

NewRecordBatchReader creates a RecordBatchReader that iterates over Arrow record batches for the given data splits. The splits can be non-contiguous and in any order. All splits must originate from the same Plan.

type TableScan 

type TableScan struct {
	// contains filtered or unexported fields
}

TableScan scans a table and produces a Plan containing data splits.

func (*TableScan) Close 

func (ts *TableScan) Close()

Close releases the table scan resources. Safe to call multiple times.

func (*TableScan) Plan 

func (ts *TableScan) Plan() (*Plan, error)

Plan executes the scan and returns a Plan containing data splits to read.

type TableWrite added in v0.4.0 

type TableWrite struct {
	// contains filtered or unexported fields
}

TableWrite accumulates Arrow record batches until PrepareCommit is called.

func (*TableWrite) Close added in v0.4.0 

func (tw *TableWrite) Close()

Close releases the writer resources. Unprepared data is discarded. Safe to call multiple times.

func (*TableWrite) PrepareCommit added in v0.4.0 

func (tw *TableWrite) PrepareCommit() (*CommitMessages, error)

PrepareCommit returns pending writes as commit messages; the writer can be reused.

func (*TableWrite) WriteArrowBatch added in v0.4.0 

func (tw *TableWrite) WriteArrowBatch(record arrow.Record) error

WriteArrowBatch writes one record batch whose schema must match the table schema. The record remains owned by the caller.

type Time 

type Time int32

Time represents a time-of-day value as milliseconds since midnight. Usage: pb.Eq("t", paimon.Time(3600000))

type Timestamp 

type Timestamp struct {
	Millis int64
	Nanos  int32
}

Timestamp represents a timestamp without timezone (millis + sub-millis nanos). Usage: pb.Eq("ts", paimon.Timestamp{Millis: 1700000000000, Nanos: 0})

type WriteBuilder added in v0.4.0 

type WriteBuilder struct {
	// contains filtered or unexported fields
}

WriteBuilder creates writers and committers that share one commit identity.

func (*WriteBuilder) Close added in v0.4.0 

func (wb *WriteBuilder) Close()

Close releases the write builder resources. Safe to call multiple times.

func (*WriteBuilder) NewCommit added in v0.4.0 

func (wb *WriteBuilder) NewCommit() (*TableCommit, error)

NewCommit creates a committer that shares this builder's identity and mode.

func (*WriteBuilder) NewWrite added in v0.4.0 

func (wb *WriteBuilder) NewWrite() (*TableWrite, error)

NewWrite creates a writer that accumulates Arrow record batches.

func (*WriteBuilder) WithOverwrite added in v0.4.0 

func (wb *WriteBuilder) WithOverwrite() error

WithOverwrite enables overwrite mode for this builder's writers and committers.

func (*WriteBuilder) WithResources added in v0.4.0 

func (wb *WriteBuilder) WithResources(resources *ResourceContext) error

WithResources shares the context's reservation budget with writers from this builder. The builder retains the context after the caller closes its handle.

Source Files

View all Source files

Directories

Path Synopsis
Package predicate provides convenience functions for combining Paimon filter predicates.
 Package predicate provides convenience functions for combining Paimon filter predicates.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.