Documentation
¶
Index ¶
- type Bucket
- func (b *Bucket) Close() error
- func (b *Bucket) Download(ctx context.Context, opts *DownloadOptions) (res drivers.FileIterator, resErr error)
- func (b *Bucket) ListObjects(ctx context.Context, path, delimiter string, pageSize uint32, pageToken string) ([]drivers.ObjectStoreEntry, string, error)
- func (b *Bucket) ListObjectsForGlob(ctx context.Context, glob string) ([]drivers.ObjectStoreEntry, error)
- func (b *Bucket) Underlying() *blob.Bucket
- type DownloadOptions
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Bucket ¶
type Bucket struct {
// contains filtered or unexported fields
}
Bucket wraps a blob.Bucket with functionality for implementing the drivers.ObjectStore interface. NOTE: It currently only supports listing objects, but eventually we should refactor NewIterator to a member function of this struct.
func NewBucket ¶
NewBucket wraps a *blob.Bucket. It takes ownership of the bucket and will close it when Close is called.
func (*Bucket) Download ¶
func (b *Bucket) Download(ctx context.Context, opts *DownloadOptions) (res drivers.FileIterator, resErr error)
Download returns an iterator for downloading objects matching a glob pattern. The downloaded objects will be stored in a temporary directory with the same file hierarchy the bucket, enabling parsing of hive partitioning on the downloaded files. The client should call Close() once done to release all resources. Calling Close() on the iterator will also close the bucket.
func (*Bucket) ListObjects ¶
func (*Bucket) ListObjectsForGlob ¶ added in v0.78.0
func (b *Bucket) ListObjectsForGlob(ctx context.Context, glob string) ([]drivers.ObjectStoreEntry, error)
ListObjectsForGlob lists objects in the bucket that match the given glob pattern. The glob pattern should be a valid path *without* scheme or bucket name. E.g. to list gs://my-bucket/path/to/files/*, the glob pattern should be "path/to/files/*".
func (*Bucket) Underlying ¶
Underlying returns the underlying *blob.Bucket.
type DownloadOptions ¶
type DownloadOptions struct {
// Glob is the pattern to match files in the bucket
Glob string
// Format is the format of the files (e.g. "csv", "json", "parquet")
Format string
// TempDir where temporary files should be stored
TempDir string
// KeepFilesUntilClose if true, files will not be deleted until Close() is called.
KeepFilesUntilClose bool
// BatchSizeBytes is the size of the batch to download before sending it to the client.
BatchSizeBytes int64
// CloseBucket will close the bucket when the iterator is closed.
CloseBucket bool
}
Options for Download
Source Files