Documentation
¶
Index ¶
- Variables
- func FetchFileFromGCSWithZstd(ctx context.Context, client ObjectStorage, gsURL string, localFile *os.File) error
- func FetchFromGCS(ctx context.Context, client ObjectStorage, gsURL string) ([]byte, error)
- func FetchLocalFileFromGCSWithZstd(ctx context.Context, client ObjectStorage, gsURL string, localFilePath string) (err error)
- func NewS3ClientFromEnv(ctx context.Context) (*s3.Client, error)
- func ObjectSize(r io.Reader) (int64, bool)
- func Open(ctx context.Context, client ObjectStorage, gsURL string) (io.ReadCloser, error)
- func SendBytesToGCS(ctx context.Context, client ObjectStorage, gsURL string, content []byte) error
- func SendFileToGCSWithZstd(ctx context.Context, client ObjectStorage, gsURL string, localFile *os.File) error
- func SendLocalFileToGCSWithZstd(ctx context.Context, client ObjectStorage, gsURL string, localFilePath string) (err error)
- func UsesS3() bool
- type ObjectStorage
- type ServiceError
- type SizedReader
Constants ¶
This section is empty.
Variables ¶
var ErrObjectNotFound = errors.New("object not found in storage backend")
ErrObjectNotFound reports the object or its bucket is absent from the storage backend, as opposed to a transient fetch failure.
Functions ¶
func FetchFileFromGCSWithZstd ¶
func FetchFileFromGCSWithZstd(ctx context.Context, client ObjectStorage, gsURL string, localFile *os.File) error
FetchFileFromGCSWithZstd downloads and decompresses into an already-open file. The caller retains ownership of localFile.
func FetchFromGCS ¶
func NewS3ClientFromEnv ¶ added in v0.5.0
NewS3ClientFromEnv builds an S3 client from the standard AWS environment variables. AWS_S3_USE_PATH_STYLE=true selects path-style addressing.
func ObjectSize ¶ added in v0.5.0
ObjectSize returns the size of the object r, a reader GetObject returned, reads, and false if its backend did not report one.
func Open ¶
func Open(ctx context.Context, client ObjectStorage, gsURL string) (io.ReadCloser, error)
Open streams the object at gsURL; the caller must Close the returned reader. Unlike FetchFromGCS it does not buffer the whole object in memory.
func SendBytesToGCS ¶
SendBytesToGCS uploads the given bytes (uncompressed) to gsURL. Intended for small objects such as the snapshot manifest.
func SendFileToGCSWithZstd ¶
func SendFileToGCSWithZstd(ctx context.Context, client ObjectStorage, gsURL string, localFile *os.File) error
SendFileToGCSWithZstd compresses and uploads an already-open file. The caller retains ownership of localFile.
Types ¶
type ObjectStorage ¶
type ObjectStorage interface {
GetObject(ctx context.Context, bucket, object string) (io.ReadCloser, error)
PutObject(ctx context.Context, bucket, object string, reader io.Reader) error
}
func NewFromEnv ¶ added in v0.5.0
func NewFromEnv(ctx context.Context) (ObjectStorage, error)
NewFromEnv builds the ObjectStorage for the backend UsesS3 selects.
func NewGCSClient ¶
func NewGCSClient(ctx context.Context, opts ...option.ClientOption) (ObjectStorage, error)
NewGCSClient returns a GCS-backed ObjectStorage. It builds its own storage.Client from opts (and more from the same opts for the part-upload pool, see uploadClient) rather than accepting one, because it installs a RetryAlways policy (see setRetry) that is only safe for this package's own operations and must not leak onto a client shared with other code.
func NewS3Client ¶
func NewS3Client(client *s3.Client) ObjectStorage
type ServiceError ¶ added in v0.5.0
type ServiceError struct {
// HTTPStatus is the status of the service's response.
HTTPStatus int
// Code is the service's error code, such as "PreconditionFailed", if the
// backend reports one.
Code string
// Message is the service's own description, if the backend reports one
// it knows names no caller, resource or credential.
Message string
}
ServiceError describes a storage service's failed response in terms its backend has judged safe to log: never a URI, header or credential.
func DescribeServiceError ¶ added in v0.5.0
func DescribeServiceError(err error) (ServiceError, bool)
DescribeServiceError describes the storage service response that err carries, if any. Each backend decides what of its errors is safe to describe; an error in err's chain from any other source with an HTTPStatusCode() int method, as the S3 SDK's response errors have, is described by its status alone.
type SizedReader ¶ added in v0.5.0
type SizedReader interface {
io.ReadCloser
// ObjectSize is the whole object's size in bytes, as the backend reported
// it, or a negative number if it is unknown.
ObjectSize() int64
}
SizedReader is implemented by the readers GetObject returns when the backend reported the object's size as it opened it. The GCS and S3 clients in this package return one whenever the response carries the size.
Source Files