Documentation
¶
Overview ¶
Package io provides an interface for IO implementations along with a registry for registering IO implementations for different URI schemes.
Subpackages of this package provide implementations for cloud storage providers which will register themselves if imported. For instance, adding the following import:
import _ "github.com/apache/iceberg-go/io/gocloud"
Will register S3, GCS and Azure, and link all three cloud SDKs. Importing io/gocloud/s3, io/gocloud/gcs or io/gocloud/azure instead registers only that backend. The local filesystem (file:// and empty scheme) is always registered.
Index ¶
- Constants
- Variables
- func GetRegisteredSchemes() []string
- func LoadFSFunc(props map[string]string, location string) func(ctx context.Context) (IO, error)
- func Register(scheme string, factory SchemeFactory)
- func Unregister(scheme string)
- type BulkRemovableIO
- type File
- type FileWriter
- type IO
- type ListableIO
- type LocalFS
- func (LocalFS) Create(name string) (FileWriter, error)
- func (LocalFS) Mkdir(name string) error
- func (LocalFS) MkdirAll(name string) error
- func (LocalFS) Open(name string) (File, error)
- func (LocalFS) ReadDir(name string) ([]fs.DirEntry, error)
- func (LocalFS) ReadFile(name string) ([]byte, error)
- func (LocalFS) Remove(name string) error
- func (LocalFS) RemoveAll(name string) error
- func (LocalFS) Rename(oldpath, newpath string) error
- func (LocalFS) RenameNoReplace(oldpath, newpath string) error
- func (LocalFS) Stat(name string) (fs.FileInfo, error)
- func (LocalFS) WalkDir(root string, fn fs.WalkDirFunc) error
- func (LocalFS) WriteFile(name string, content []byte) error
- type MemFS
- type MkdirAllIO
- type ReadDirFile
- type ReadFileIO
- type RemoveAllIO
- type RenameIO
- type RenameNoReplaceIO
- type SchemeFactory
- type StatIO
- type WriteFileIO
Constants ¶
const ( S3Region = "s3.region" S3SessionToken = "s3.session-token" S3SecretAccessKey = "s3.secret-access-key" S3AccessKeyID = "s3.access-key-id" S3EndpointURL = "s3.endpoint" S3ProxyURI = "s3.proxy-uri" // S3ConnectTimeout accepts seconds as a number, such as "60" or "60.0", // or a Go duration string, such as "5s". S3ConnectTimeout = "s3.connect-timeout" S3SignerURI = "s3.signer.uri" S3RemoteSigningEnabled = "s3.remote-signing-enabled" S3ForceVirtualAddressing = "s3.force-virtual-addressing" S3ClientRegion = "client.region" S3CompatMode = "s3.compat-mode" )
Constants for S3 configuration options
const ( GCSEndpoint = "gcs.endpoint" GCSServiceHost = "gcs.service.host" GCSKeyPath = "gcs.keypath" GCSJSONKey = "gcs.jsonkey" GCSCredType = "gcs.credtype" GCSUseJSONAPI = "gcs.usejsonapi" GCSOAuthToken = "gcs.oauth2.token" GCSOAuthExpiresAt = "gcs.oauth2.token-expires-at" GCSNoAuth = "gcs.no-auth" )
Constants for GCS configuration options
const ( ADLSSasTokenPrefix = "adls.sas-token." ADLSConnectionStringPrefix = "adls.connection-string." ADLSEndpoint = "adls.endpoint" ADLSProtocol = "adls.protocol" // ADLSManagedIdentityEnabled uses ManagedIdentityCredential directly when set // to "true", rather than the broader DefaultAzureCredential chain, which already // tries managed identity among its other sources. Pair it with ADLSClientID to // select a user-assigned managed identity; on its own it uses the system-assigned // managed identity. // // It is evaluated only after shared-key credentials, a SAS-token property for // the target hostname, and a connection-string property for the target account // name, all of which take precedence over it; if any of them is configured, // this flag has no effect. // // This property is specific to this implementation. Neither the Java nor the // PyIceberg implementation defines an equivalent property, so catalog properties // that set it will not carry over to them. ADLSManagedIdentityEnabled = "adls.auth.managed-identity.enabled" // ADLSClientID selects a user-assigned managed identity by client ID. It only // takes effect when ADLSManagedIdentityEnabled is "true"; on its own it is ignored. // // PyIceberg recognizes the same "adls.client-id" key but uses it for service // principal authentication, alongside "adls.client-secret" and "adls.tenant-id", // which it requires to be set together; see // https://py.iceberg.apache.org/configuration/#azure-data-lake. This // implementation defines neither of those, so the key is only ever read as a // user-assigned managed identity ID here. Java's AzureProperties defines no // "adls.client-id" at all. ADLSClientID = "adls.client-id" )
Constants for Azure configuration options
Variables ¶
var (
ErrIOSchemeNotFound = errors.New("io scheme not registered")
)
Functions ¶
func GetRegisteredSchemes ¶ added in v0.5.0
func GetRegisteredSchemes() []string
GetRegisteredSchemes returns the list of registered scheme names.
func LoadFSFunc ¶ added in v0.4.0
LoadFSFunc is a helper function to create IO Func factory for later usage
func Register ¶ added in v0.5.0
func Register(scheme string, factory SchemeFactory)
Register adds a new scheme factory to the registry. If the scheme is already registered, it will panic.
func Unregister ¶ added in v0.5.0
func Unregister(scheme string)
Unregister removes the requested scheme factory from the registry.
Types ¶
type BulkRemovableIO ¶ added in v0.6.0
type BulkRemovableIO interface {
IO
// DeleteFiles deletes all named files. Implementations should make a
// best-effort attempt to delete as many files as possible, returning
// the list of successfully deleted paths and a joined error
// (via [errors.Join]) for any individual failures.
//
// Missing files are not considered errors — if a path does not exist,
// it should be treated as a successful deletion.
//
// An empty paths slice is a no-op and must not error.
DeleteFiles(ctx context.Context, paths []string) (deleted []string, err error)
}
BulkRemovableIO is an optional interface for IO implementations that support deleting multiple files in a single batch operation. Cloud object stores (S3 DeleteObjects, GCS batch, Azure batch) can implement this for significantly better throughput than single-file Remove.
type File ¶
A File provides access to a single file. The File interface is the minimum implementation required for Iceberg to interact with a file. Directory files should also implement
type FileWriter ¶ added in v0.2.0
type FileWriter interface {
io.WriteCloser
io.ReaderFrom
}
A FileWriter represents an open writable file.
type IO ¶
type IO interface {
// Open opens the named file.
//
// When Open returns an error, it should be of type *PathError
// with the Op field set to "open", the Path field set to name,
// and the Err field describing the problem.
//
// Open should reject attempts to open names that do not satisfy
// fs.ValidPath(name), returning a *PathError with Err set to
// ErrInvalid or ErrNotExist.
Open(name string) (File, error)
// Remove removes the named file or (empty) directory.
//
// If there is an error, it will be of type *PathError.
// Implementations must be safe for concurrent use by multiple goroutines.
Remove(name string) error
}
IO is an interface to a hierarchical file system.
The IO interface is the minimum implementation required for a file system to utilize an iceberg table. A file system may implement additional interfaces, such as ReadFileIO, to provide additional or optimized functionality.
func FSPreProcName ¶
FSPreProcName wraps an io/fs.FS like FS, only if fn is non-nil then it is called to preprocess any filenames before they are passed to the underlying fsys.
func LoadFS ¶
LoadFS takes a map of properties and an optional URI location and attempts to infer an IO object from it using the registered scheme factories.
The scheme is extracted from the location URI and used to look up the appropriate factory from the registry. The local filesystem (file:// or empty scheme) is registered by default.
Additional schemes can be registered by importing subpackages. Import the backend for each cloud in use, or io/gocloud for all of them:
import _ "github.com/apache/iceberg-go/io/gocloud/s3"
type ListableIO ¶ added in v0.6.0
type ListableIO interface {
IO
// WalkDir walks the file tree rooted at root, calling fn for each
// file or directory in the tree, including root. Paths passed to fn
// are full URIs consistent with the root's scheme and authority.
WalkDir(root string, fn fs.WalkDirFunc) error
}
ListableIO is the interface implemented by a file system that supports directory walking. This replaces the need for reflect-based access to internal file system structures (e.g., blob storage buckets).
The root parameter is a full URI (e.g., "s3://bucket/path" or "/local/path") as used throughout the iceberg-go codebase. The fn callback receives full URI paths consistent with the root's scheme and authority.
type LocalFS ¶
type LocalFS struct{}
LocalFS is an implementation of IO that implements interaction with the local file system.
func (LocalFS) RenameNoReplace ¶ added in v0.7.0
type MemFS ¶ added in v0.6.0
type MemFS struct {
// contains filtered or unexported fields
}
MemFS is a simple in-memory IO implementation used for testing and lightweight workloads. Files are stored per named bucket and persist for the lifetime of the process. Use the "mem://bucket/path" URI scheme to access files.
func NewMemFS ¶ added in v0.6.0
func NewMemFS() *MemFS
NewMemFS creates an initialized MemFS ready for use.
type MkdirAllIO ¶ added in v0.7.0
type MkdirAllIO interface {
IO
// MkdirAll should not raise an error if the directory already exists,
// and should create any parent directories in the path if they
// do not already exist.
MkdirAll(path string) error
}
MkdirAllIO is an extension of IO interface that includes the MkdirAll method for creating a directory path recursively
type ReadDirFile ¶
type ReadDirFile interface {
File
// ReadDir read the contents of the directory and returns a slice
// of up to n DirEntry values in directory order. Subsequent calls
// on the same file will yield further DirEntry values.
//
// If n > 0, ReadDir returns at most n DirEntry structures. In this
// case, if ReadDir returns an empty slice, it will return a non-nil
// error explaining why.
//
// At the end of a directory, the error is io.EOF. (ReadDir must return
// io.EOF itself, not an error wrapping io.EOF.)
//
// If n <= 0, ReadDir returns all the DirEntry values from the directory
// in a single slice. In this case, if ReadDir succeeds (reads all the way
// to the end of the directory), it returns the slice and a nil error.
// If it encounters an error before the end of the directory, ReadDir
// returns the DirEntry list read until that point and a non-nil error.
ReadDir(n int) ([]fs.DirEntry, error)
}
A ReadDirFile is a directory file whose entries can be read with the ReadDir method. Every directory file should implement this interface. (It is permissible for any file to implement this interface, but if so ReadDir should return an error for non-directories.)
type ReadFileIO ¶
type ReadFileIO interface {
IO
// ReadFile reads the named file and returns its contents.
// A successful call returns a nil error, not io.EOF.
// (Because ReadFile reads the whole file, the expected EOF
// from the final Read is not treated as an error to be reported.)
//
// The caller is permitted to modify the returned byte slice.
// This method should return a copy of the underlying data.
ReadFile(name string) ([]byte, error)
}
ReadFileIO is the interface implemented by a file system that provides an optimized implementation of ReadFile.
type RemoveAllIO ¶ added in v0.7.0
RemoveAllIO is an extension of IO interface that includes the RemoveAll method for removing a file path and any children recursively
type RenameIO ¶ added in v0.7.0
RenameIO is an extension of IO interface that includes the Rename method for renaming (moving) one file or object. This is atomic on a local filesystem but is not atomic in other implementations like blob stores
type RenameNoReplaceIO ¶ added in v0.7.0
RenameNoReplaceIO is an extension of IO interface that includes the RenameNoReplace method for moving files only when the destination path does not already exist. Implementations may require oldpath and newpath to live on the same filesystem. This is atomic on a local filesystem but is not atomic in other implementations like blob stores
type SchemeFactory ¶ added in v0.5.0
SchemeFactory is a function that creates an IO implementation for a given URI and properties.
type StatIO ¶ added in v0.7.0
type StatIO interface {
IO
// The Stat method returns a FileInfo describing the named file or directory,
// or an error satisfying errors.Is(err, fs.ErrNotExist) if the file does not exist
Stat(path string) (fs.FileInfo, error)
}
StatIO is an extension of IO interface that includes the Stat method for retrieving file or directory information
type WriteFileIO ¶ added in v0.2.0
type WriteFileIO interface {
IO
// Create attempts to create the named file and return a writer
// for it.
Create(name string) (FileWriter, error)
// WriteFile writes p to the named file.
WriteFile(name string, p []byte) error
}
WriteFileIO is the interface implemented by a file system that provides an optimized implementation of WriteFile
Directories
¶
| Path | Synopsis |
|---|---|
|
Package gocloud registers every gocloud.dev-backed FileIO implementation and therefore links the AWS, Google Cloud and Azure SDKs.
|
Package gocloud registers every gocloud.dev-backed FileIO implementation and therefore links the AWS, Google Cloud and Azure SDKs. |
|
azure
Package azure provides the FileIO backend for Azure Data Lake Storage and Blob Storage.
|
Package azure provides the FileIO backend for Azure Data Lake Storage and Blob Storage. |
|
blobfs
Package blobfs implements the iceberg-go/io FileIO interfaces on top of a gocloud.dev blob bucket.
|
Package blobfs implements the iceberg-go/io FileIO interfaces on top of a gocloud.dev blob bucket. |
|
gcs
Package gcs provides the FileIO backend for Google Cloud Storage.
|
Package gcs provides the FileIO backend for Google Cloud Storage. |
|
s3
Package s3 provides the FileIO backend for S3 and S3-compatible object stores.
|
Package s3 provides the FileIO backend for S3 and S3-compatible object stores. |