storage

package
v0.12.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 41 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AmazonS3Configured

func AmazonS3Configured(s *schema.AmazonS3Storage) bool

AmazonS3Configured reports whether S3 is enabled with complete or implicit credentials.

func CheckGet

func CheckGet(store Storage, url string) error

CheckGet checks input support without probing unrelated output capabilities.

func CheckPut

func CheckPut(store Storage, url string) error

CheckPut checks output support without probing unrelated input capabilities.

func Download

func Download(ctx context.Context, store Storage, transfers []Transfer, parallelLimit int)

Download downloads a list of transfers from storage, in parallel.

Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.

func FTPConfigured

func FTPConfigured(h *schema.FTPStorage) bool

FTPConfigured reports whether FTP storage is enabled.

func FileSize

func FileSize(path string) int64

FileSize returns the file size in bytes, or return 0 if there's an error calling os.Stat().

func GenericS3Configured

func GenericS3Configured(s *schema.GenericS3Storage) bool

GenericS3Configured reports whether generic S3 has its required settings.

func GoogleConfigured

func GoogleConfigured(g *schema.GoogleCloudStorage) bool

GoogleConfigured reports whether Google storage is enabled.

func HTTPConfigured

func HTTPConfigured(h *schema.HTTPStorage) bool

HTTPConfigured reports whether HTTP storage is enabled.

func LocalConfigured

func LocalConfigured(l *schema.LocalStorage) bool

LocalConfigured reports whether local storage is enabled with allowed directories.

func OpenRootRegularFile

func OpenRootRegularFile(root *os.Root, path string) (*os.File, error)

OpenRootRegularFile opens a confined regular file without blocking on a FIFO.

func SwiftConfigured

func SwiftConfigured(s *schema.SwiftStorage) bool

SwiftConfigured resolves required settings from config and the environment.

func Upload

func Upload(ctx context.Context, store Storage, transfers []Transfer, parallelLimit int)

Upload uploads a list of transfers to storage, in parallel.

Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.

func ValidateS3ReadMounts

func ValidateS3ReadMounts(mounts []S3ReadMount) error

func ValidateS3WriteMounts

func ValidateS3WriteMounts(mounts []S3WriteMount) error

Types

type AmazonS3

type AmazonS3 struct {
	// contains filtered or unexported fields
}

AmazonS3 provides access to an S3 object store.

func NewAmazonS3

func NewAmazonS3(conf *config.AmazonS3Storage) (*AmazonS3, error)

NewAmazonS3 creates an AmazonS3 session instance

func (*AmazonS3) Get

func (s3b *AmazonS3) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from S3 to the host path.

func (*AmazonS3) Join

func (s3b *AmazonS3) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*AmazonS3) List

func (s3b *AmazonS3) List(ctx context.Context, url string) ([]*Object, error)

List returns a list of objects at the given url.

func (*AmazonS3) Put

func (s3b *AmazonS3) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to S3.

func (*AmazonS3) Stat

func (s3b *AmazonS3) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*AmazonS3) UnsupportedOperations

func (s3b *AmazonS3) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type ErrInvalidURL

type ErrInvalidURL struct {
	// contains filtered or unexported fields
}

ErrInvalidURL is returned by SupportsGet / SupportsPut when a url's format is invalid.

func (*ErrInvalidURL) Error

func (e *ErrInvalidURL) Error() string

type ErrUnsupportedProtocol

type ErrUnsupportedProtocol struct {
	// contains filtered or unexported fields
}

ErrUnsupportedProtocol is returned by SupportsGet / SupportsPut when a url's protocol is unsupported by that backend

func (*ErrUnsupportedProtocol) Error

func (e *ErrUnsupportedProtocol) Error() string

type FTP

type FTP struct {
	// contains filtered or unexported fields
}

FTP provides read access to public URLs.

func NewFTP

func NewFTP(conf *config.FTPStorage) (*FTP, error)

NewFTP creates a new FTP instance.

func (*FTP) Get

func (b *FTP) Get(ctx context.Context, url, path string) (*Object, error)

Get copies a file from a given URL to the host path.

func (*FTP) Join

func (b *FTP) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*FTP) List

func (b *FTP) List(ctx context.Context, url string) ([]*Object, error)

List is not supported by FTP storage.

func (*FTP) Put

func (b *FTP) Put(ctx context.Context, url string, hostPath string) (*Object, error)

Put is not supported by FTP storage.

func (*FTP) Stat

func (b *FTP) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*FTP) UnsupportedOperations

func (b *FTP) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Fake

type Fake struct{}

Fake implements a the Storage interface with methods that do nothing and return nil. This is a testing utility.

func (Fake) Get

func (f Fake) Get(ctx context.Context, url, path string) (*Object, error)

Get a single object from storage URL, written to a local file path.

func (Fake) Join

func (f Fake) Join(url, path string) (string, error)

Join a directory URL with a subpath.

func (Fake) List

func (f Fake) List(ctx context.Context, url string) ([]*Object, error)

List a directory. Calling List on a File is an error.

func (Fake) Put

func (f Fake) Put(ctx context.Context, url, path string) (*Object, error)

Put a single object to storage URL, from a local file path. Returns the Object that was created in storage.

func (Fake) Stat

func (f Fake) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (Fake) UnsupportedOperations

func (f Fake) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations are supported by the storage system for the given URL.

type GenericS3

type GenericS3 struct {
	// contains filtered or unexported fields
}

GenericS3 provides access to an S3 object store.

func NewGenericS3

func NewGenericS3(conf *config.GenericS3Storage) (*GenericS3, error)

NewGenericS3 creates a new GenericS3 instance, given an endpoint URL and a set of authentication credentials.

func (*GenericS3) Get

func (s3b *GenericS3) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from S3 to the host path.

func (*GenericS3) Join

func (s3b *GenericS3) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*GenericS3) List

func (s3b *GenericS3) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*GenericS3) Put

func (s3b *GenericS3) Put(ctx context.Context, url, path string) (*Object, error)

Put publishes a local file through an API upload or a configured mounted write. Mounted writes verify remote existence and size; API uploads require an upload response. Directory publication returns aggregate metadata after every file succeeds.

func (*GenericS3) Stat

func (s3b *GenericS3) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*GenericS3) UnsupportedOperations

func (s3b *GenericS3) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type GoogleCloud

type GoogleCloud struct {
	// contains filtered or unexported fields
}

GoogleCloud provides access to an GS object store.

func NewGoogleCloud

func NewGoogleCloud(conf *config.GoogleCloudStorage) (*GoogleCloud, error)

NewGoogleCloud creates an GoogleCloud client instance, give an endpoint URL and a set of authentication credentials.

func (*GoogleCloud) Get

func (gs *GoogleCloud) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from GS to the host path.

func (*GoogleCloud) Join

func (gs *GoogleCloud) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*GoogleCloud) List

func (gs *GoogleCloud) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*GoogleCloud) Put

func (gs *GoogleCloud) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to GS.

func (*GoogleCloud) Stat

func (gs *GoogleCloud) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*GoogleCloud) UnsupportedOperations

func (gs *GoogleCloud) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type HTTP

type HTTP struct {
	// contains filtered or unexported fields
}

HTTP provides access to URLs that need no credentials, such as public or presigned URLs: downloads with GET and uploads with PUT.

func NewHTTP

func NewHTTP(conf *config.HTTPStorage) (*HTTP, error)

NewHTTP creates a new HTTP instance.

func (*HTTP) Get

func (b *HTTP) Get(ctx context.Context, url, path string) (*Object, error)

Get copies a file from a given URL to the host path.

func (*HTTP) Join

func (b *HTTP) Join(url, path string) (string, error)

Join joins the given URL with the given subpath. Each segment of the subpath is escaped, so a file name containing a character such as #, ? or % stays part of the path, and the URL's query, which may carry a token, is kept.

func (*HTTP) List

func (b *HTTP) List(ctx context.Context, url string) ([]*Object, error)

List is not supported by HTTP storage.

func (*HTTP) Put

func (b *HTTP) Put(ctx context.Context, url string, hostPath string) (*Object, error)

Put uploads a file from the host path to the given URL with a PUT request. This is how presigned upload URLs (S3, Google Cloud Storage and others) are used, so the worker needs no storage credentials.

func (*HTTP) Stat

func (b *HTTP) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*HTTP) UnsupportedOperations

func (b *HTTP) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Hostfile

type Hostfile struct {
	// The path relative to the "root" given to WalkFiles().
	Rel string
	// The absolute path of the file on the host.
	Abs string
	// Size in bytes.
	Size int64
	// LastModified time
	LastModified time.Time
}

Hostfile returns information about a file found by WalkFiles.

func WalkFiles

func WalkFiles(root string) ([]Hostfile, error)

WalkFiles recursively walks a directory, returning a list of files.

type Local

type Local struct {
	// contains filtered or unexported fields
}

Local provides access to a local-disk storage system.

func NewLocal

func NewLocal(conf *config.LocalStorage) (*Local, error)

NewLocal returns a Local instance, configured to limit file system access to the given allowed directories.

func (*Local) Get

func (local *Local) Get(ctx context.Context, url, path string) (*Object, error)

Get copies a file from storage into the given hostPath.

func (*Local) Join

func (local *Local) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Local) List

func (local *Local) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*Local) Put

func (local *Local) Put(ctx context.Context, url, path string) (*Object, error)

Put copies a file from the hostPath into storage.

func (*Local) Stat

func (local *Local) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*Local) UnsupportedOperations

func (local *Local) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Mux

type Mux struct {
	Backends []Storage
}

Mux provides a client for accessing multiple storage systems, i.e. for downloading/uploading task files from S3, GS, local disk, etc.

For a given storage url, the storage backend is usually determined by the url prefix, e.g. "s3://my-bucket/file" will access the S3 backend.

func NewMux

func NewMux(conf *config.Config) (*Mux, error)

NewMux returns a new Mux instance with the given additional configuration.

func NewWorkerMux

func NewWorkerMux(conf *config.Config, mounts []S3ReadMount, writes ...S3WriteMount) (*Mux, error)

NewWorkerMux activates only the S3 mounts captured in the submitted plan. NewMux leaves filesystem access disabled for server-side validation.

func (*Mux) AttachLogger

func (mux *Mux) AttachLogger(log *logger.Logger)

AttachLogger will log information (such as retry warnings) to the given logger.

func (*Mux) Get

func (mux *Mux) Get(ctx context.Context, url, path string) (*Object, error)

Get downloads a file from a storage system at the given "url". The file is downloaded to the given local "path".

func (*Mux) Join

func (mux *Mux) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Mux) List

func (mux *Mux) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*Mux) Put

func (mux *Mux) Put(ctx context.Context, url, path string) (*Object, error)

Put uploads a file to a storage system at the given "url". The file is uploaded from the given local "path".

func (*Mux) Stat

func (mux *Mux) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*Mux) UnsupportedOperations

func (mux *Mux) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Object

type Object struct {
	// The storage-specific full URL of the object.
	// e.g. for S3 this might be "s3://my-bucket/dir1/obj.txt"
	URL string

	// The name of the object in the storage system.
	// e.g. for S3 this might be "dir/object.txt"
	Name string

	// ETag is an identifier for a specific version of the object.
	// This is an opaque string. Each system has a different representation:
	// md5, sha1, crc32, etc. This field may be empty, if the system can't provide
	// a unique ID (for example the local filesystem).
	ETag string

	LastModified time.Time

	// Size of the object, in bytes.
	Size int64
}

Object represents metadata about an object in storage.

type Retrier

type Retrier struct {
	Backend    Storage
	NewBackOff func() backoff.BackOff
	// MaxTries counts total attempts; zero still makes one attempt.
	MaxTries    uint
	ShouldRetry func(error) bool
	Notify      backoff.Notify
}

Retrier retries operations on a storage backend using a fresh backoff per call.

func NewSwiftRetrier

func NewSwiftRetrier(conf *config.SwiftStorage) (*Retrier, error)

NewSwiftRetrier returns a Swift storage client that retries operations on error.

func (*Retrier) Get

func (r *Retrier) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from storage to a local path.

func (*Retrier) Join

func (r *Retrier) Join(url, path string) (string, error)

Join joins a directory URL with the given subpath.

func (*Retrier) List

func (r *Retrier) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given URL.

func (*Retrier) Put

func (r *Retrier) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object from a local path to storage.

func (*Retrier) Stat

func (r *Retrier) Stat(ctx context.Context, url string) (*Object, error)

Stat returns metadata about the given URL.

func (*Retrier) UnsupportedOperations

func (r *Retrier) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations are supported for the URL.

type S3ReadMount

type S3ReadMount struct {
	Endpoint string
	Bucket   string
	Prefix   string
	Root     string
}

S3ReadMount records a worker mount resolved for one S3 backend. Credentials remain in worker configuration, outside the submitted plan.

type S3WriteMount

type S3WriteMount struct {
	Endpoint string
	Bucket   string
	Prefix   string
	Root     string
	Timeout  time.Duration
}

S3WriteMount records an explicit destination and its publication deadline.

type Storage

type Storage interface {
	// Stat returns information about the object at the given storage URL.
	Stat(ctx context.Context, url string) (*Object, error)

	// List a directory. Calling List on a File is an error.
	List(ctx context.Context, url string) ([]*Object, error)

	// Get a single object from storage URL, written to a local file path.
	Get(ctx context.Context, url, path string) (*Object, error)

	// Put a single object to storage URL, from a local file path.
	// Returns the Object that was created in storage.
	Put(ctx context.Context, url, path string) (*Object, error)

	// Join a directory URL with a subpath.
	Join(url, path string) (string, error)

	// UnsupportedOperations describes which operations are supported by the storage system
	// for the given URL.
	//
	// A backend normally uses this to match the url prefix (e.g. "s3://"),
	// and some backends (generic s3) might check for the existence of the bucket.
	UnsupportedOperations(url string) UnsupportedOperations
}

Storage provides an interface for a storage backend, providing access to concrete storage systems such as Google Storage, local filesystem, etc.

Storage backends must be safe for concurrent use.

type Swift

type Swift struct {
	// contains filtered or unexported fields
}

Swift provides access to an sw object store.

func NewSwift

func NewSwift(conf *config.SwiftStorage) (*Swift, error)

NewSwift creates an Swift client instance, give an endpoint URL and a set of authentication credentials.

func (*Swift) Get

func (sw *Swift) Get(ctx context.Context, url, path string) (obj *Object, retErr error)

Get copies an object from storage to the host path.

func (*Swift) Join

func (sw *Swift) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Swift) List

func (sw *Swift) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*Swift) Put

func (sw *Swift) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to storage.

func (*Swift) Stat

func (sw *Swift) Stat(ctx context.Context, url string) (*Object, error)

Stat returns metadata about the given url, such as checksum.

func (*Swift) UnsupportedOperations

func (sw *Swift) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Transfer

type Transfer interface {
	URL() string
	Path() string
	Started()
	Finished(obj *Object)
	Failed(err error)
}

Transfer defines the interface of a single storage upload or download request.

Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.

type UnsupportedOperations

type UnsupportedOperations struct {
	Get, Put, List, Stat, Join error
}

UnsupportedOperations describes any operations that are not supported by a storage backend.

func AllSupported

func AllSupported() UnsupportedOperations

AllSupported returns an UnsupportedOperations indicating that all operations are supported.

func AllUnsupported

func AllUnsupported(err error) UnsupportedOperations

AllUnsupported returns an UnsupportedOperations indicating that no operations are supported.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL