Documentation
¶
Index ¶
- Constants
- func CreateAWSAssumeRoleCredentials(client *sts.Client, roleARN string, externalID *string) aws.CredentialsProvider
- func CreateAWSStaticCredentials(awsKey, awsPassword, awsToken string) aws.CredentialsProvider
- func ExtractPrefixHierarchy(fileKey string) []string
- func GetHeadObject(ctx context.Context, r PreSignRequestParams) (*s3.HeadObjectOutput, error)
- func IsKeyNotFoundError(err error) bool
- func MakeKeyNotFoundError(err error) error
- func NewKeyNotFoundError(msg string) error
- func NewKeyNotFoundErrorf(msg string, args ...interface{}) error
- func PreSign(ctx context.Context, r PreSignRequestParams) (string, error)
- func WithSeed(provider aws.CredentialsProvider, cacheKey string, ...) aws.CredentialsProvider
- type Bucket
- func NewGridFSBucket(ctx context.Context, opts GridFSOptions) (Bucket, error)
- func NewGridFSBucketWithClient(ctx context.Context, client *mongo.Client, opts GridFSOptions) (Bucket, error)
- func NewLocalBucket(opts LocalOptions) (Bucket, error)
- func NewLocalTemporaryBucket(opts LocalOptions) (Bucket, error)
- func NewParallelSyncBucket(opts ParallelBucketOptions, b Bucket) (Bucket, error)
- func NewS3Bucket(ctx context.Context, options S3Options) (Bucket, error)
- func NewS3BucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (Bucket, error)
- func NewS3MultiPartBucket(ctx context.Context, options S3Options) (Bucket, error)
- func NewS3MultiPartBucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (Bucket, error)
- type BucketItem
- type BucketIterator
- type CopyOptions
- type FastGetS3Bucket
- type GridFSOptions
- type LifecycleRule
- type LocalOptions
- type ParallelBucketOptions
- type PreSignRequestParams
- type PutCounter
- type S3Options
- type S3Permissions
- type StreamPutCounter
- type StreamPutCounterWithBytes
- type SyncBucket
- type SyncOptions
- type Transition
Constants ¶
const PresignExpireTime = 24 * time.Hour
PresignExpireTime sets the amount of time the link is live before expiring.
Variables ¶
This section is empty.
Functions ¶
func CreateAWSAssumeRoleCredentials ¶
func CreateAWSAssumeRoleCredentials(client *sts.Client, roleARN string, externalID *string) aws.CredentialsProvider
CreateAWSAssumeRoleCredentials creates an AWS CredentialsProvider that assumes the given role ARN using the provided STS client. An optional external ID can be provided to further secure the assume role operation.
func CreateAWSStaticCredentials ¶
func CreateAWSStaticCredentials(awsKey, awsPassword, awsToken string) aws.CredentialsProvider
CreateAWSStaticCredentials is a wrapper for creating static AWS credentials.
func ExtractPrefixHierarchy ¶
ExtractPrefixHierarchy extracts all possible prefixes from a file key in descending order of specificity. For "a/b/c/file.txt", returns ["a/b/c/", "a/b/", "a/", ""]. For "file.txt", returns [""].
func GetHeadObject ¶
func GetHeadObject(ctx context.Context, r PreSignRequestParams) (*s3.HeadObjectOutput, error)
GetHeadObject fetches the metadata of an S3 object. Warning: despite the input parameters, the function doesn't pre-sign the request to get the head object at all.
func IsKeyNotFoundError ¶
IsKeyNotFoundError checks an error object to see if it is a key not found error.
func MakeKeyNotFoundError ¶
MakeKeyNotFoundError constructs a key not found error from an existing error of any type.
func NewKeyNotFoundError ¶
NewKeyNotFoundError creates a new error object to represent a key not found error.
func NewKeyNotFoundErrorf ¶
NewKeyNotFoundErrorf creates a new error object to represent a key not found error with a formatted message.
func PreSign ¶
func PreSign(ctx context.Context, r PreSignRequestParams) (string, error)
PreSign returns a presigned URL that expires in 24 hours.
func WithSeed ¶
func WithSeed(provider aws.CredentialsProvider, cacheKey string, minimumLifetime time.Duration) aws.CredentialsProvider
WithSeed wraps the given CredentialsProvider with a caching layer that uses the given cacheKey to identify cached credentials. The given minimumLifetime should match the expectation of the AWS client using the credentials, e.g. the ExpiryWindow.
Types ¶
type Bucket ¶
type Bucket interface {
// Check validity of the bucket. This is dependent on the underlying
// implementation.
Check(context.Context) error
// Exists returns whether the given key exists in the bucket or not.
Exists(context.Context, string) (bool, error)
// Join concatenates elements with the appropriate path separator of
// the bucket, ignoring empty elements. This is analogous to
// `filepath.Join`.
Join(...string) string
// Produces a Writer and Reader interface to the file named by
// the string.
Writer(context.Context, string) (io.WriteCloser, error)
Reader(context.Context, string) (io.ReadCloser, error)
// Put and Get write simple byte streams (in the form of
// io.Readers) to/from specified keys.
//
// TODO: consider if these, particularly Get are not
// substantively different from Writer/Reader methods, or
// might just be a wrapper.
Put(context.Context, string, io.Reader) error
Get(context.Context, string) (io.ReadCloser, error)
// Upload and Download write files from the local file
// system to the specified key.
Upload(context.Context, string, string) error
Download(context.Context, string, string) error
SyncBucket
// Copy does a special copy operation that does not require downloading
// a file. Note that CopyOptions.DestinationBucket must have the same
// type as the calling bucket object.
Copy(context.Context, CopyOptions) error
// Remove the specified object(s) from the bucket.
// RemoveMany continues on error and returns any accumulated errors.
Remove(context.Context, string) error
RemoveMany(context.Context, ...string) error
// Remove all objects with the given prefix, continuing on error and
// returning any accumulated errors.
// Note that this operation is not atomic.
RemovePrefix(context.Context, string) error
// Remove all objects matching the given regular expression, continuing
// on error and returning any accumulated errors.
// Note that this operation is not atomic.
RemoveMatching(context.Context, string) error
// List returns an iterator over the contents of a bucket with the
// the given prefix. Contents are iterated lexicographically by key
// name.
List(context.Context, string) (BucketIterator, error)
// String returns the bucket name.
String() string
// MoveObjects moves multiple objects from sourceKeys in this bucket to destKeys in another bucket specified by destBucket.
// The lengths of sourceKeys and destKeys must match.
MoveObjects(ctx context.Context, destBucket Bucket, sourceKeys, destKeys []string) error
}
Bucket defines an interface for accessing a remote blob store, like S3. Should be generic enough to be implemented for GCP equivalent.
Other goals of this project are to allow us to have a single interface for interacting with blob storage, and allow us to fully move off of our legacy goamz package and stabalize all blob-storage operations across all projects. There should be no interface dependencies on external packages required to use this library.
See, the following implemenations for previous approaches.
- https://github.com/evergreen-ci/evergreen/blob/master/thirdparty/s3.go
- https://github.com/mongodb/curator/tree/master/sthree
The preferred AWS SDK is here: https://docs.aws.amazon.com/sdk-for-go/api/
In no particular order:
- implementation constructors should make it possible to use custom http.Clients (to aid in pooling.)
- We should probably implement .String methods.
- Do use the grip package for logging.
- get/put should support multipart upload/download?
- we'll want to do retries with back-off (potentially configurable in bucketinfo?)
- we might need to have variants that Put/Get byte slices rather than readers.
- pass contexts to requests for timeouts.
func NewGridFSBucket ¶
func NewGridFSBucket(ctx context.Context, opts GridFSOptions) (Bucket, error)
NewGridFSBucket returns a bucket backed by GridFS with the given options.
func NewGridFSBucketWithClient ¶
func NewGridFSBucketWithClient(ctx context.Context, client *mongo.Client, opts GridFSOptions) (Bucket, error)
NewGridFSBucketWithClient returns a new bucket backed by GridFS with the existing Mongo client and given options.
func NewLocalBucket ¶
func NewLocalBucket(opts LocalOptions) (Bucket, error)
NewLocalBucket returns an implementation of the Bucket interface that stores files in the local file system. Returns an error if the directory doesn't exist.
func NewLocalTemporaryBucket ¶
func NewLocalTemporaryBucket(opts LocalOptions) (Bucket, error)
NewLocalTemporaryBucket returns an "local" bucket implementation that stores resources in the local filesystem in a temporary directory created for this purpose. Returns an error if there were issues creating the temporary directory. This implementation does not provide a mechanism to delete the temporary directory.
func NewParallelSyncBucket ¶
func NewParallelSyncBucket(opts ParallelBucketOptions, b Bucket) (Bucket, error)
NewParallelSyncBucket returns a layered bucket implemenation that supports parallel sync operations.
func NewS3Bucket ¶
NewS3Bucket returns a Bucket implementation backed by S3. This implementation does not support multipart uploads, if you would like to add objects larger than 5 gigabytes see NewS3MultiPartBucket.
func NewS3BucketWithHTTPClient ¶
func NewS3BucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (Bucket, error)
NewS3BucketWithHTTPClient returns a Bucket implementation backed by S3 with an existing HTTP client connection. This implementation does not support multipart uploads, if you would like to add objects larger than 5 gigabytes see NewS3MultiPartBucket.
func NewS3MultiPartBucket ¶
NewS3MultiPartBucket returns a Bucket implementation backed by S3 that supports multipart uploads for large objects.
func NewS3MultiPartBucketWithHTTPClient ¶
func NewS3MultiPartBucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (Bucket, error)
NewS3MultiPartBucketWithHTTPClient returns a Bucket implementation backed by S3 with an existing HTTP client connection that supports multipart uploads for large objects.
type BucketItem ¶
type BucketItem interface {
Bucket() string
Name() string
Hash() string
// Size returns the object size in bytes.
Size() int64
Get(context.Context) (io.ReadCloser, error)
}
BucketItem provides a basic interface for getting an object from a bucket.
type BucketIterator ¶
type BucketIterator interface {
Next(context.Context) bool
Err() error
Item() BucketItem
}
BucketIterator provides a way to interact with the contents of a bucket, as in the output of the List operation.
type CopyOptions ¶
type CopyOptions struct {
SourceKey string
DestinationKey string
DestinationBucket Bucket
IsDestination bool
}
CopyOptions describes the arguments to the Copy method for moving objects between Buckets.
type FastGetS3Bucket ¶
type FastGetS3Bucket interface {
Bucket
// GetToWriter allows the user to pass in an io.WriterAt (which is likely
// going to be a file) that has the contents of the remote key automatically
// copied to it in parallel. GetToWriter is significantly more efficient for
// large files than a Bucket.Get.
GetToWriter(context.Context, string, io.WriterAt) error
}
FastGetS3Bucket is a Bucket but with an additional method, GetToWriter. Only S3 bucket types can support this access pattern.
func NewFastGetS3BucketWithHTTPClient ¶
func NewFastGetS3BucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (FastGetS3Bucket, error)
NewFastGetS3BucketWithHttpClient does the same thing as NewS3BucketWithHTTPClient, but returns the FastGetS3Bucket interface instead.
type GridFSOptions ¶
type GridFSOptions struct {
Name string
Prefix string
Database string
MongoDBURI string
DryRun bool
DeleteOnSync bool
DeleteOnPush bool
DeleteOnPull bool
Verbose bool
}
GridFSOptions support the use and creation of GridFS backed buckets.
type LifecycleRule ¶
type LifecycleRule struct {
// ID is the unique identifier for the rule
ID string
// Prefix is the object key prefix that the rule applies to (extracted from Filter or deprecated Prefix field)
Prefix string
// Status indicates whether the rule is currently active ("Enabled" or "Disabled")
Status string
// ExpirationDays is the number of days after creation when objects expire (nil if not set)
ExpirationDays *int32
// TransitionToIADays is the number of days after creation when objects should transition to STANDARD_IA (nil if not set)
TransitionToIADays *int32
// TransitionToGlacierDays is the number of days after creation when objects transition to GLACIER (nil if not set)
TransitionToGlacierDays *int32
// Transitions contains all transition rules with full details (empty if none)
Transitions []Transition
}
LifecycleRule represents a simplified S3 bucket lifecycle rule with commonly needed fields extracted.
func FindMatchingRule ¶
func FindMatchingRule(rules []LifecycleRule, fileKey string) *LifecycleRule
FindMatchingRule finds the most specific lifecycle rule that matches the given file key. It uses longest-prefix matching: for "a/b/c/file.txt", it tries "a/b/c/", "a/b/", "a/", and "". Only enabled rules are considered. Returns nil if no matching enabled rule is found.
type LocalOptions ¶
type LocalOptions struct {
Path string
Prefix string
// UseSlash sets the prefix separator to the slash ('/') character
// instead of the OS specific separator.
UseSlash bool
DryRun bool
DeleteOnSync bool
DeleteOnPush bool
DeleteOnPull bool
Verbose bool
}
LocalOptions describes the configuration of a local Bucket.
type ParallelBucketOptions ¶
type ParallelBucketOptions struct {
// Workers sets the number of worker threads.
Workers int
// DryRun enables running in a mode that will not execute any
// operations that modify the bucket.
DryRun bool
// DeleteOnSync will delete all objects from the target that do not
// exist in the source after the completion of a sync operation
// (Push/Pull).
DeleteOnSync bool
// DeleteOnPush will delete all objects from the target that do not
// exist in the source after the completion of Push.
DeleteOnPush bool
// DeleteOnPull will delete all objects from the target that do not
// exist in the source after the completion of Pull.
DeleteOnPull bool
}
ParallelBucketOptions support the use and creation of parallel sync buckets.
type PreSignRequestParams ¶
type PreSignRequestParams struct {
Bucket string
FileKey string
Region string
SignatureExpiryWindow time.Duration
// Static credentials specific fields.
AWSKey string
AWSSecret string
AWSSessionToken string
// AssumeRole specific fields.
AWSRoleARN string
ExternalID *string
}
PreSignRequestParams holds all the parameters needed to sign a URL or fetch S3 object metadata.
type PutCounter ¶
type PutCounter interface {
Bucket
UploadWithCount(ctx context.Context, key, path string) (int, error)
}
PutCounter is an optional interface implemented by S3 buckets that returns the number of PUT-equivalent API calls made per upload.
type S3Options ¶
type S3Options struct {
// DryRun enables running in a mode that will not execute any
// operations that modify the bucket.
DryRun bool
// DeleteOnSync will delete all objects from the target that do not
// exist in the destination after the completion of a sync operation
// (Push/Pull).
DeleteOnSync bool
// DeleteOnPush will delete all objects from the target that do not
// exist in the source after the completion of Push.
DeleteOnPush bool
// DeleteOnPull will delete all objects from the target that do not
// exist in the source after the completion of Pull.
DeleteOnPull bool
// Compress enables gzipping of uploaded objects. For downloading, objects
// that are compressed with gzip are automatically decoded.
Compress bool
// ExpectedChecksumSHA256 enables checksum verification of downloads using the
// SHA256 checksum stored in S3. This requires that the object was
// originally uploaded with the checksum information.
ExpectedChecksumSHA256 string
// UploadChecksumSHA256 enables sending the SHA256 checksum of uploads.
UploadChecksumSHA256 bool
// Verbose sets the logging mode to "debug".
Verbose bool
// MaxRetries sets the number of retry attempts for S3 operations.
// By default it defers to the AWS SDK's default.
MaxRetries *int
// Credentials allows the passing in of explicit AWS credentials. These
// will override the default credentials chain. (Optional)
Credentials aws.CredentialsProvider
// credentials chain and the Credentials value (see above). (Optional)
SharedCredentialsFilepath string
// credentials profile from the shared credentials file. (Optional)
SharedCredentialsProfile string
// AssumeRoleARN specifies an IAM role ARN. When not empty, it will be
// used to assume the given role for this session. See
// `https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html` for
// more information. (Optional)
AssumeRoleARN string
// AssumeRoleOptions provide a mechanism to override defaults by
// applying changes to the AssumeRoleProvider struct created with this
// session. This field is ignored if AssumeRoleARN is not set.
// (Optional)
AssumeRoleOptions []func(*stscreds.AssumeRoleOptions)
// Region specifies the AWS region.
Region string
// Name specifies the name of the bucket.
Name string
// Prefix specifies the prefix to use. (Optional)
Prefix string
// Permissions sets the S3 permissions to use for each object. Defaults
// to FULL_CONTROL. See
// `https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html`
// for more information.
Permissions S3Permissions
// ContentType sets the standard MIME type of the object data. Defaults
// to nil. See
//`https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17`
// for more information.
ContentType string
// StorageClass sets the S3 storage class for uploaded objects. Defaults
// to S3's default storage class (STANDARD) if not set. See
// https://aws.amazon.com/s3/storage-classes/ for more information.
StorageClass s3Types.StorageClass
// IfNotExists, when set to true, will avoid overwriting an already-existing
// object at the destination key if it already exists.
IfNotExists bool
// MaxConcurrentCopies specifies the maximum number of concurrent copy
// operations for MoveObjects. If nil or <=0, defaults to 20.
// (Optional)
MaxConcurrentCopies *int
// VersionID pins download operations to a specific S3 object version.
// When set, Get and Reader operations will request this version of the
// object. (Optional)
VersionID string
}
S3Options support the use and creation of S3 backed buckets.
type S3Permissions ¶
type S3Permissions string
S3Permissions is a type that describes the object canned ACL from S3.
const ( S3PermissionsPrivate S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLPrivate)) S3PermissionsPublicRead S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLPublicRead)) S3PermissionsPublicReadWrite S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLPublicReadWrite)) S3PermissionsAuthenticatedRead S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLAuthenticatedRead)) S3PermissionsAWSExecRead S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLAwsExecRead)) S3PermissionsBucketOwnerRead S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLBucketOwnerRead)) S3PermissionsBucketOwnerFullControl S3Permissions = S3Permissions(string(s3Types.ObjectCannedACLBucketOwnerFullControl)) )
Valid S3 permissions.
func (S3Permissions) Validate ¶
func (p S3Permissions) Validate() error
Validate checks that the S3Permissions string is valid.
type StreamPutCounter ¶
type StreamPutCounter interface {
Bucket
PutWithCount(ctx context.Context, key string, r io.Reader) (int, error)
}
StreamPutCounter is an optional interface implemented by S3 buckets that returns the number of PUT-equivalent API calls made when streaming content to S3.
type StreamPutCounterWithBytes ¶
type StreamPutCounterWithBytes interface {
Bucket
PutWithCountAndBytes(ctx context.Context, key string, r io.Reader) (puts int, bytesWritten int64, err error)
}
StreamPutCounterWithBytes extends StreamPutCounter to also return bytes written to S3.
type SyncBucket ¶
type SyncBucket interface {
// Sync methods: these methods are the recursive, efficient
// copy methods of files from S3 to the local file
// system.
Push(context.Context, SyncOptions) error
Pull(context.Context, SyncOptions) error
}
SyncBucket defines an interface to access a remote blob store and synchronize the local file system tree with the remote store.
type SyncOptions ¶
SyncOptions describes the arguments to the sync operations (Push and Pull). Note that exclude is a regular expression.
type Transition ¶
type Transition struct {
// Days is the number of days after creation when the transition occurs (nil if Date is used instead)
Days *int32
// StorageClass is the target storage class (e.g., "STANDARD_IA", "GLACIER", "DEEP_ARCHIVE")
StorageClass string
}
Transition represents a storage class transition with timing information.