io

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 28, 2026 License: Apache-2.0 Imports: 18 Imported by: 18

Documentation

Overview

Package io provides an interface for IO implementations along with a registry for registering IO implementations for different URI schemes.

Subpackages of this package provide implementations for cloud storage providers which will register themselves if imported. For instance, adding the following import:

import _ "github.com/apache/iceberg-go/io/gocloud"

Will register S3, GCS and Azure, and link all three cloud SDKs. Importing io/gocloud/s3, io/gocloud/gcs or io/gocloud/azure instead registers only that backend. The local filesystem (file:// and empty scheme) is always registered.

Index

Constants

View Source
const (
	S3Region          = "s3.region"
	S3SessionToken    = "s3.session-token"
	S3SecretAccessKey = "s3.secret-access-key"
	S3AccessKeyID     = "s3.access-key-id"
	S3EndpointURL     = "s3.endpoint"
	S3ProxyURI        = "s3.proxy-uri"
	// S3ConnectTimeout accepts seconds as a number, such as "60" or "60.0",
	// or a Go duration string, such as "5s".
	S3ConnectTimeout         = "s3.connect-timeout"
	S3SignerURI              = "s3.signer.uri"
	S3RemoteSigningEnabled   = "s3.remote-signing-enabled"
	S3ForceVirtualAddressing = "s3.force-virtual-addressing"
	S3ClientRegion           = "client.region"
	S3CompatMode             = "s3.compat-mode"
)

Constants for S3 configuration options

View Source
const (
	GCSEndpoint       = "gcs.endpoint"
	GCSServiceHost    = "gcs.service.host"
	GCSKeyPath        = "gcs.keypath"
	GCSJSONKey        = "gcs.jsonkey"
	GCSCredType       = "gcs.credtype"
	GCSUseJSONAPI     = "gcs.usejsonapi"
	GCSOAuthToken     = "gcs.oauth2.token"
	GCSOAuthExpiresAt = "gcs.oauth2.token-expires-at"
	GCSNoAuth         = "gcs.no-auth"
)

Constants for GCS configuration options

View Source
const (
	ADLSSasTokenPrefix         = "adls.sas-token."
	ADLSConnectionStringPrefix = "adls.connection-string."
	ADLSSharedKeyAccountName   = "adls.auth.shared-key.account.name"
	ADLSSharedKeyAccountKey    = "adls.auth.shared-key.account.key"
	ADLSEndpoint               = "adls.endpoint"
	ADLSProtocol               = "adls.protocol"

	// ADLSManagedIdentityEnabled uses ManagedIdentityCredential directly when set
	// to "true", rather than the broader DefaultAzureCredential chain, which already
	// tries managed identity among its other sources. Pair it with ADLSClientID to
	// select a user-assigned managed identity; on its own it uses the system-assigned
	// managed identity.
	//
	// It is evaluated only after shared-key credentials, a SAS-token property for
	// the target hostname, and a connection-string property for the target account
	// name, all of which take precedence over it; if any of them is configured,
	// this flag has no effect.
	//
	// This property is specific to this implementation. Neither the Java nor the
	// PyIceberg implementation defines an equivalent property, so catalog properties
	// that set it will not carry over to them.
	ADLSManagedIdentityEnabled = "adls.auth.managed-identity.enabled"

	// ADLSClientID selects a user-assigned managed identity by client ID. It only
	// takes effect when ADLSManagedIdentityEnabled is "true"; on its own it is ignored.
	//
	// PyIceberg recognizes the same "adls.client-id" key but uses it for service
	// principal authentication, alongside "adls.client-secret" and "adls.tenant-id",
	// which it requires to be set together; see
	// https://py.iceberg.apache.org/configuration/#azure-data-lake. This
	// implementation defines neither of those, so the key is only ever read as a
	// user-assigned managed identity ID here. Java's AzureProperties defines no
	// "adls.client-id" at all.
	ADLSClientID = "adls.client-id"
)

Constants for Azure configuration options

Variables

View Source
var (
	ErrIOSchemeNotFound = errors.New("io scheme not registered")
)

Functions

func GetRegisteredSchemes added in v0.5.0

func GetRegisteredSchemes() []string

GetRegisteredSchemes returns the list of registered scheme names.

func LoadFSFunc added in v0.4.0

func LoadFSFunc(props map[string]string, location string) func(ctx context.Context) (IO, error)

LoadFSFunc is a helper function to create IO Func factory for later usage

func Register added in v0.5.0

func Register(scheme string, factory SchemeFactory)

Register adds a new scheme factory to the registry. If the scheme is already registered, it will panic.

func Unregister added in v0.5.0

func Unregister(scheme string)

Unregister removes the requested scheme factory from the registry.

Types

type BulkRemovableIO added in v0.6.0

type BulkRemovableIO interface {
	IO

	// DeleteFiles deletes all named files. Implementations should make a
	// best-effort attempt to delete as many files as possible, returning
	// the list of successfully deleted paths and a joined error
	// (via [errors.Join]) for any individual failures.
	//
	// Missing files are not considered errors — if a path does not exist,
	// it should be treated as a successful deletion.
	//
	// An empty paths slice is a no-op and must not error.
	DeleteFiles(ctx context.Context, paths []string) (deleted []string, err error)
}

BulkRemovableIO is an optional interface for IO implementations that support deleting multiple files in a single batch operation. Cloud object stores (S3 DeleteObjects, GCS batch, Azure batch) can implement this for significantly better throughput than single-file Remove.

type File

type File interface {
	fs.File
	io.ReadSeekCloser
	io.ReaderAt
}

A File provides access to a single file. The File interface is the minimum implementation required for Iceberg to interact with a file. Directory files should also implement

type FileWriter added in v0.2.0

type FileWriter interface {
	io.WriteCloser
	io.ReaderFrom
}

A FileWriter represents an open writable file.

type IO

type IO interface {
	// Open opens the named file.
	//
	// When Open returns an error, it should be of type *PathError
	// with the Op field set to "open", the Path field set to name,
	// and the Err field describing the problem.
	//
	// Open should reject attempts to open names that do not satisfy
	// fs.ValidPath(name), returning a *PathError with Err set to
	// ErrInvalid or ErrNotExist.
	Open(name string) (File, error)

	// Remove removes the named file or (empty) directory.
	//
	// If there is an error, it will be of type *PathError.
	// Implementations must be safe for concurrent use by multiple goroutines.
	Remove(name string) error
}

IO is an interface to a hierarchical file system.

The IO interface is the minimum implementation required for a file system to utilize an iceberg table. A file system may implement additional interfaces, such as ReadFileIO, to provide additional or optimized functionality.

func FS

func FS(fsys fs.FS) IO

FS wraps an io/fs.FS as an IO interface.

func FSPreProcName

func FSPreProcName(fsys fs.FS, fn func(string) string) IO

FSPreProcName wraps an io/fs.FS like FS, only if fn is non-nil then it is called to preprocess any filenames before they are passed to the underlying fsys.

func LoadFS

func LoadFS(ctx context.Context, props map[string]string, location string) (IO, error)

LoadFS takes a map of properties and an optional URI location and attempts to infer an IO object from it using the registered scheme factories.

The scheme is extracted from the location URI and used to look up the appropriate factory from the registry. The local filesystem (file:// or empty scheme) is registered by default.

Additional schemes can be registered by importing subpackages. Import the backend for each cloud in use, or io/gocloud for all of them:

import _ "github.com/apache/iceberg-go/io/gocloud/s3"

type ListableIO added in v0.6.0

type ListableIO interface {
	IO

	// WalkDir walks the file tree rooted at root, calling fn for each
	// file or directory in the tree, including root. Paths passed to fn
	// are full URIs consistent with the root's scheme and authority.
	WalkDir(root string, fn fs.WalkDirFunc) error
}

ListableIO is the interface implemented by a file system that supports directory walking. This replaces the need for reflect-based access to internal file system structures (e.g., blob storage buckets).

The root parameter is a full URI (e.g., "s3://bucket/path" or "/local/path") as used throughout the iceberg-go codebase. The fn callback receives full URI paths consistent with the root's scheme and authority.

type LocalFS

type LocalFS struct{}

LocalFS is an implementation of IO that implements interaction with the local file system.

func (LocalFS) Create added in v0.2.0

func (LocalFS) Create(name string) (FileWriter, error)

func (LocalFS) Mkdir added in v0.7.0

func (LocalFS) Mkdir(name string) error

func (LocalFS) MkdirAll added in v0.7.0

func (LocalFS) MkdirAll(name string) error

func (LocalFS) Open

func (LocalFS) Open(name string) (File, error)

func (LocalFS) ReadDir added in v0.7.0

func (LocalFS) ReadDir(name string) ([]fs.DirEntry, error)

func (LocalFS) ReadFile added in v0.7.0

func (LocalFS) ReadFile(name string) ([]byte, error)

func (LocalFS) Remove

func (LocalFS) Remove(name string) error

func (LocalFS) RemoveAll added in v0.7.0

func (LocalFS) RemoveAll(name string) error

func (LocalFS) Rename added in v0.7.0

func (LocalFS) Rename(oldpath, newpath string) error

func (LocalFS) RenameNoReplace added in v0.7.0

func (LocalFS) RenameNoReplace(oldpath, newpath string) error

func (LocalFS) Stat added in v0.7.0

func (LocalFS) Stat(name string) (fs.FileInfo, error)

func (LocalFS) WalkDir added in v0.6.0

func (LocalFS) WalkDir(root string, fn fs.WalkDirFunc) error

func (LocalFS) WriteFile added in v0.2.0

func (LocalFS) WriteFile(name string, content []byte) error

type MemFS added in v0.6.0

type MemFS struct {
	// contains filtered or unexported fields
}

MemFS is a simple in-memory IO implementation used for testing and lightweight workloads. Files are stored per named bucket and persist for the lifetime of the process. Use the "mem://bucket/path" URI scheme to access files.

func NewMemFS added in v0.6.0

func NewMemFS() *MemFS

NewMemFS creates an initialized MemFS ready for use.

func (*MemFS) Create added in v0.6.0

func (m *MemFS) Create(name string) (FileWriter, error)

func (*MemFS) Open added in v0.6.0

func (m *MemFS) Open(name string) (File, error)

func (*MemFS) Remove added in v0.6.0

func (m *MemFS) Remove(name string) error

func (*MemFS) WalkDir added in v0.6.0

func (m *MemFS) WalkDir(root string, fn fs.WalkDirFunc) error

func (*MemFS) WriteFile added in v0.6.0

func (m *MemFS) WriteFile(name string, content []byte) error

type MkdirAllIO added in v0.7.0

type MkdirAllIO interface {
	IO

	// MkdirAll should not raise an error if the directory already exists,
	// and should create any parent directories in the path if they
	// do not already exist.
	MkdirAll(path string) error
}

MkdirAllIO is an extension of IO interface that includes the MkdirAll method for creating a directory path recursively

type ReadDirFile

type ReadDirFile interface {
	File

	// ReadDir read the contents of the directory and returns a slice
	// of up to n DirEntry values in directory order. Subsequent calls
	// on the same file will yield further DirEntry values.
	//
	// If n > 0, ReadDir returns at most n DirEntry structures. In this
	// case, if ReadDir returns an empty slice, it will return a non-nil
	// error explaining why.
	//
	// At the end of a directory, the error is io.EOF. (ReadDir must return
	// io.EOF itself, not an error wrapping io.EOF.)
	//
	// If n <= 0, ReadDir returns all the DirEntry values from the directory
	// in a single slice. In this case, if ReadDir succeeds (reads all the way
	// to the end of the directory), it returns the slice and a nil error.
	// If it encounters an error before the end of the directory, ReadDir
	// returns the DirEntry list read until that point and a non-nil error.
	ReadDir(n int) ([]fs.DirEntry, error)
}

A ReadDirFile is a directory file whose entries can be read with the ReadDir method. Every directory file should implement this interface. (It is permissible for any file to implement this interface, but if so ReadDir should return an error for non-directories.)

type ReadFileIO

type ReadFileIO interface {
	IO

	// ReadFile reads the named file and returns its contents.
	// A successful call returns a nil error, not io.EOF.
	// (Because ReadFile reads the whole file, the expected EOF
	// from the final Read is not treated as an error to be reported.)
	//
	// The caller is permitted to modify the returned byte slice.
	// This method should return a copy of the underlying data.
	ReadFile(name string) ([]byte, error)
}

ReadFileIO is the interface implemented by a file system that provides an optimized implementation of ReadFile.

type RemoveAllIO added in v0.7.0

type RemoveAllIO interface {
	IO

	RemoveAll(name string) error
}

RemoveAllIO is an extension of IO interface that includes the RemoveAll method for removing a file path and any children recursively

type RenameIO added in v0.7.0

type RenameIO interface {
	IO

	Rename(oldpath, newpath string) error
}

RenameIO is an extension of IO interface that includes the Rename method for renaming (moving) one file or object. This is atomic on a local filesystem but is not atomic in other implementations like blob stores

type RenameNoReplaceIO added in v0.7.0

type RenameNoReplaceIO interface {
	IO

	RenameNoReplace(oldpath, newpath string) error
}

RenameNoReplaceIO is an extension of IO interface that includes the RenameNoReplace method for moving files only when the destination path does not already exist. Implementations may require oldpath and newpath to live on the same filesystem. This is atomic on a local filesystem but is not atomic in other implementations like blob stores

type SchemeFactory added in v0.5.0

type SchemeFactory func(ctx context.Context, parsed *url.URL, props map[string]string) (IO, error)

SchemeFactory is a function that creates an IO implementation for a given URI and properties.

type StatIO added in v0.7.0

type StatIO interface {
	IO

	// The Stat method returns a FileInfo describing the named file or directory,
	// or an error satisfying errors.Is(err, fs.ErrNotExist) if the file does not exist
	Stat(path string) (fs.FileInfo, error)
}

StatIO is an extension of IO interface that includes the Stat method for retrieving file or directory information

type WriteFileIO added in v0.2.0

type WriteFileIO interface {
	IO

	// Create attempts to create the named file and return a writer
	// for it.
	Create(name string) (FileWriter, error)

	// WriteFile writes p to the named file.
	WriteFile(name string, p []byte) error
}

WriteFileIO is the interface implemented by a file system that provides an optimized implementation of WriteFile

Directories

Path Synopsis
Package gocloud registers every gocloud.dev-backed FileIO implementation and therefore links the AWS, Google Cloud and Azure SDKs.
Package gocloud registers every gocloud.dev-backed FileIO implementation and therefore links the AWS, Google Cloud and Azure SDKs.
azure
Package azure provides the FileIO backend for Azure Data Lake Storage and Blob Storage.
Package azure provides the FileIO backend for Azure Data Lake Storage and Blob Storage.
blobfs
Package blobfs implements the iceberg-go/io FileIO interfaces on top of a gocloud.dev blob bucket.
Package blobfs implements the iceberg-go/io FileIO interfaces on top of a gocloud.dev blob bucket.
gcs
Package gcs provides the FileIO backend for Google Cloud Storage.
Package gcs provides the FileIO backend for Google Cloud Storage.
s3
Package s3 provides the FileIO backend for S3 and S3-compatible object stores.
Package s3 provides the FileIO backend for S3 and S3-compatible object stores.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL