pail

package module
v0.0.0-...-896ecbb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 24, 2026 License: Apache-2.0 Imports: 35 Imported by: 34

README

===========================================
``pail`` -- Blob Storage System Abstraction
===========================================

Overview
--------

Pail is a high-level Go interface to blob storage containers like AWS's
S3 and similar services. Pail also provides implementation backed by
local file systems, mostly used for testing.

Documentation
-------------

The core API documentation is in the `godoc
<https://godoc.org/github.com/evergreen-ci/pail/>`_.

Contribute
----------

Open tickets in the `EVG project <http://jira.mongodb.org/browse/EVG>`_, and
feel free to open pull requests here.

Development
-----------

The pail project uses a ``makefile`` to coordinate testing. Use the following
command to build the cedar binary: ::

  make build

The artifact is at ``build/pail``. The makefile provides the following
targets:

``test``
   Runs all tests, sequentially, for all packages.

``test-<package>``
   Runs all tests for a specific package

``RACE_DETECTOR=1 make test-package``
   As with their ``test`` counterpart, these targets run tests with
   the race detector enabled.

``lint``, ``lint-<package>``
   Installs and runs the ``gometaliter`` with appropriate settings to
   lint the project.

Documentation

Index

Constants

View Source
const PresignExpireTime = 24 * time.Hour

PresignExpireTime sets the amount of time the link is live before expiring.

Variables

This section is empty.

Functions

func CreateAWSAssumeRoleCredentials

func CreateAWSAssumeRoleCredentials(client *sts.Client, roleARN string, externalID *string) aws.CredentialsProvider

CreateAWSAssumeRoleCredentials creates an AWS CredentialsProvider that assumes the given role ARN using the provided STS client. An optional external ID can be provided to further secure the assume role operation.

func CreateAWSStaticCredentials

func CreateAWSStaticCredentials(awsKey, awsPassword, awsToken string) aws.CredentialsProvider

CreateAWSStaticCredentials is a wrapper for creating static AWS credentials.

func ExtractPrefixHierarchy

func ExtractPrefixHierarchy(fileKey string) []string

ExtractPrefixHierarchy extracts all possible prefixes from a file key in descending order of specificity. For "a/b/c/file.txt", returns ["a/b/c/", "a/b/", "a/", ""]. For "file.txt", returns [""].

func GetHeadObject

func GetHeadObject(ctx context.Context, r PreSignRequestParams) (*s3.HeadObjectOutput, error)

GetHeadObject fetches the metadata of an S3 object. Warning: despite the input parameters, the function doesn't pre-sign the request to get the head object at all.

func IsKeyNotFoundError

func IsKeyNotFoundError(err error) bool

IsKeyNotFoundError checks an error object to see if it is a key not found error.

func MakeKeyNotFoundError

func MakeKeyNotFoundError(err error) error

MakeKeyNotFoundError constructs a key not found error from an existing error of any type.

func NewKeyNotFoundError

func NewKeyNotFoundError(msg string) error

NewKeyNotFoundError creates a new error object to represent a key not found error.

func NewKeyNotFoundErrorf

func NewKeyNotFoundErrorf(msg string, args ...interface{}) error

NewKeyNotFoundErrorf creates a new error object to represent a key not found error with a formatted message.

func PreSign

func PreSign(ctx context.Context, r PreSignRequestParams) (string, error)

PreSign returns a presigned URL that expires in 24 hours.

func WithSeed

func WithSeed(provider aws.CredentialsProvider, cacheKey string, minimumLifetime time.Duration) aws.CredentialsProvider

WithSeed wraps the given CredentialsProvider with a caching layer that uses the given cacheKey to identify cached credentials. The given minimumLifetime should match the expectation of the AWS client using the credentials, e.g. the ExpiryWindow.

Types

type Bucket

type Bucket interface {
	// Check validity of the bucket. This is dependent on the underlying
	// implementation.
	Check(context.Context) error

	// Exists returns whether the given key exists in the bucket or not.
	Exists(context.Context, string) (bool, error)

	// Join concatenates elements with the appropriate path separator of
	// the bucket, ignoring empty elements. This is analogous to
	// `filepath.Join`.
	Join(...string) string

	// Produces a Writer and Reader interface to the file named by
	// the string.
	Writer(context.Context, string) (io.WriteCloser, error)
	Reader(context.Context, string) (io.ReadCloser, error)

	// Put and Get write simple byte streams (in the form of
	// io.Readers) to/from specified keys.
	//
	// TODO: consider if these, particularly Get are not
	// substantively different from Writer/Reader methods, or
	// might just be a wrapper.
	Put(context.Context, string, io.Reader) error
	Get(context.Context, string) (io.ReadCloser, error)

	// Upload and Download write files from the local file
	// system to the specified key.
	Upload(context.Context, string, string) error
	Download(context.Context, string, string) error

	SyncBucket

	// Copy does a special copy operation that does not require downloading
	// a file. Note that CopyOptions.DestinationBucket must have the same
	// type as the calling bucket object.
	Copy(context.Context, CopyOptions) error

	// Remove the specified object(s) from the bucket.
	// RemoveMany continues on error and returns any accumulated errors.
	Remove(context.Context, string) error
	RemoveMany(context.Context, ...string) error

	// Remove all objects with the given prefix, continuing on error and
	// returning any accumulated errors.
	// Note that this operation is not atomic.
	RemovePrefix(context.Context, string) error

	// Remove all objects matching the given regular expression, continuing
	// on error and returning any accumulated errors.
	// Note that this operation is not atomic.
	RemoveMatching(context.Context, string) error

	// List returns an iterator over the contents of a bucket with the
	// the given prefix. Contents are iterated lexicographically by key
	// name.
	List(context.Context, string) (BucketIterator, error)

	// String returns the bucket name.
	String() string

	// MoveObjects moves multiple objects from sourceKeys in this bucket to destKeys in another bucket specified by destBucket.
	// The lengths of sourceKeys and destKeys must match.
	MoveObjects(ctx context.Context, destBucket Bucket, sourceKeys, destKeys []string) error
}

Bucket defines an interface for accessing a remote blob store, like S3. Should be generic enough to be implemented for GCP equivalent.

Other goals of this project are to allow us to have a single interface for interacting with blob storage, and allow us to fully move off of our legacy goamz package and stabalize all blob-storage operations across all projects. There should be no interface dependencies on external packages required to use this library.

See, the following implemenations for previous approaches.

The preferred AWS SDK is here: https://docs.aws.amazon.com/sdk-for-go/api/

In no particular order:

  • implementation constructors should make it possible to use custom http.Clients (to aid in pooling.)
  • We should probably implement .String methods.
  • Do use the grip package for logging.
  • get/put should support multipart upload/download?
  • we'll want to do retries with back-off (potentially configurable in bucketinfo?)
  • we might need to have variants that Put/Get byte slices rather than readers.
  • pass contexts to requests for timeouts.

func NewGridFSBucket

func NewGridFSBucket(ctx context.Context, opts GridFSOptions) (Bucket, error)

NewGridFSBucket returns a bucket backed by GridFS with the given options.

func NewGridFSBucketWithClient

func NewGridFSBucketWithClient(ctx context.Context, client *mongo.Client, opts GridFSOptions) (Bucket, error)

NewGridFSBucketWithClient returns a new bucket backed by GridFS with the existing Mongo client and given options.

func NewLocalBucket

func NewLocalBucket(opts LocalOptions) (Bucket, error)

NewLocalBucket returns an implementation of the Bucket interface that stores files in the local file system. Returns an error if the directory doesn't exist.

func NewLocalTemporaryBucket

func NewLocalTemporaryBucket(opts LocalOptions) (Bucket, error)

NewLocalTemporaryBucket returns an "local" bucket implementation that stores resources in the local filesystem in a temporary directory created for this purpose. Returns an error if there were issues creating the temporary directory. This implementation does not provide a mechanism to delete the temporary directory.

func NewParallelSyncBucket

func NewParallelSyncBucket(opts ParallelBucketOptions, b Bucket) (Bucket, error)

NewParallelSyncBucket returns a layered bucket implemenation that supports parallel sync operations.

func NewS3Bucket

func NewS3Bucket(ctx context.Context, options S3Options) (Bucket, error)

NewS3Bucket returns a Bucket implementation backed by S3. This implementation does not support multipart uploads, if you would like to add objects larger than 5 gigabytes see NewS3MultiPartBucket.

func NewS3BucketWithHTTPClient

func NewS3BucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (Bucket, error)

NewS3BucketWithHTTPClient returns a Bucket implementation backed by S3 with an existing HTTP client connection. This implementation does not support multipart uploads, if you would like to add objects larger than 5 gigabytes see NewS3MultiPartBucket.

func NewS3MultiPartBucket

func NewS3MultiPartBucket(ctx context.Context, options S3Options) (Bucket, error)

NewS3MultiPartBucket returns a Bucket implementation backed by S3 that supports multipart uploads for large objects.

func NewS3MultiPartBucketWithHTTPClient

func NewS3MultiPartBucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (Bucket, error)

NewS3MultiPartBucketWithHTTPClient returns a Bucket implementation backed by S3 with an existing HTTP client connection that supports multipart uploads for large objects.

type BucketItem

type BucketItem interface {
	Bucket() string
	Name() string
	Hash() string
	// Size returns the object size in bytes.
	Size() int64
	Get(context.Context) (io.ReadCloser, error)
}

BucketItem provides a basic interface for getting an object from a bucket.

type BucketIterator

type BucketIterator interface {
	Next(context.Context) bool
	Err() error
	Item() BucketItem
}

BucketIterator provides a way to interact with the contents of a bucket, as in the output of the List operation.

type CopyOptions

type CopyOptions struct {
	SourceKey         string
	DestinationKey    string
	DestinationBucket Bucket
	IsDestination     bool
}

CopyOptions describes the arguments to the Copy method for moving objects between Buckets.

type FastGetS3Bucket

type FastGetS3Bucket interface {
	Bucket
	// GetToWriter allows the user to pass in an io.WriterAt (which is likely
	// going to be a file) that has the contents of the remote key automatically
	// copied to it in parallel. GetToWriter is significantly more efficient for
	// large files than a Bucket.Get.
	GetToWriter(context.Context, string, io.WriterAt) error
}

FastGetS3Bucket is a Bucket but with an additional method, GetToWriter. Only S3 bucket types can support this access pattern.

func NewFastGetS3BucketWithHTTPClient

func NewFastGetS3BucketWithHTTPClient(ctx context.Context, client *http.Client, options S3Options) (FastGetS3Bucket, error)

NewFastGetS3BucketWithHttpClient does the same thing as NewS3BucketWithHTTPClient, but returns the FastGetS3Bucket interface instead.

type GridFSOptions

type GridFSOptions struct {
	Name         string
	Prefix       string
	Database     string
	MongoDBURI   string
	DryRun       bool
	DeleteOnSync bool
	DeleteOnPush bool
	DeleteOnPull bool
	Verbose      bool
}

GridFSOptions support the use and creation of GridFS backed buckets.

type LifecycleRule

type LifecycleRule struct {
	// ID is the unique identifier for the rule
	ID string

	// Prefix is the object key prefix that the rule applies to (extracted from Filter or deprecated Prefix field)
	Prefix string

	// Status indicates whether the rule is currently active ("Enabled" or "Disabled")
	Status string

	// ExpirationDays is the number of days after creation when objects expire (nil if not set)
	ExpirationDays *int32

	// TransitionToIADays is the number of days after creation when objects should transition to STANDARD_IA (nil if not set)
	TransitionToIADays *int32

	// TransitionToGlacierDays is the number of days after creation when objects transition to GLACIER (nil if not set)
	TransitionToGlacierDays *int32

	// Transitions contains all transition rules with full details (empty if none)
	Transitions []Transition
}

LifecycleRule represents a simplified S3 bucket lifecycle rule with commonly needed fields extracted.

func FindMatchingRule

func FindMatchingRule(rules []LifecycleRule, fileKey string) *LifecycleRule

FindMatchingRule finds the most specific lifecycle rule that matches the given file key. It uses longest-prefix matching: for "a/b/c/file.txt", it tries "a/b/c/", "a/b/", "a/", and "". Only enabled rules are considered. Returns nil if no matching enabled rule is found.

type LocalOptions

type LocalOptions struct {
	Path   string
	Prefix string
	// UseSlash sets the prefix separator to the slash ('/') character
	// instead of the OS specific separator.
	UseSlash     bool
	DryRun       bool
	DeleteOnSync bool
	DeleteOnPush bool
	DeleteOnPull bool
	Verbose      bool
}

LocalOptions describes the configuration of a local Bucket.

type ParallelBucketOptions

type ParallelBucketOptions struct {
	// Workers sets the number of worker threads.
	Workers int
	// DryRun enables running in a mode that will not execute any
	// operations that modify the bucket.
	DryRun bool
	// DeleteOnSync will delete all objects from the target that do not
	// exist in the source after the completion of a sync operation
	// (Push/Pull).
	DeleteOnSync bool
	// DeleteOnPush will delete all objects from the target that do not
	// exist in the source after the completion of Push.
	DeleteOnPush bool
	// DeleteOnPull will delete all objects from the target that do not
	// exist in the source after the completion of Pull.
	DeleteOnPull bool
}

ParallelBucketOptions support the use and creation of parallel sync buckets.

type PreSignRequestParams

type PreSignRequestParams struct {
	Bucket                string
	FileKey               string
	Region                string
	SignatureExpiryWindow time.Duration

	// Static credentials specific fields.
	AWSKey          string
	AWSSecret       string
	AWSSessionToken string

	// AssumeRole specific fields.
	AWSRoleARN string
	ExternalID *string
}

PreSignRequestParams holds all the parameters needed to sign a URL or fetch S3 object metadata.

type PutCounter

type PutCounter interface {
	Bucket
	UploadWithCount(ctx context.Context, key, path string) (int, error)
}

PutCounter is an optional interface implemented by S3 buckets that returns the number of PUT-equivalent API calls made per upload.

type S3Options

type S3Options struct {
	// DryRun enables running in a mode that will not execute any
	// operations that modify the bucket.
	DryRun bool
	// DeleteOnSync will delete all objects from the target that do not
	// exist in the destination after the completion of a sync operation
	// (Push/Pull).
	DeleteOnSync bool
	// DeleteOnPush will delete all objects from the target that do not
	// exist in the source after the completion of Push.
	DeleteOnPush bool
	// DeleteOnPull will delete all objects from the target that do not
	// exist in the source after the completion of Pull.
	DeleteOnPull bool
	// Compress enables gzipping of uploaded objects. For downloading, objects
	// that are compressed with gzip are automatically decoded.
	Compress bool
	// ExpectedChecksumSHA256 enables checksum verification of downloads using the
	// SHA256 checksum stored in S3. This requires that the object was
	// originally uploaded with the checksum information.
	ExpectedChecksumSHA256 string
	// UploadChecksumSHA256 enables sending the SHA256 checksum of uploads.
	UploadChecksumSHA256 bool
	// Verbose sets the logging mode to "debug".
	Verbose bool
	// MaxRetries sets the number of retry attempts for S3 operations.
	// By default it defers to the AWS SDK's default.
	MaxRetries *int
	// Credentials allows the passing in of explicit AWS credentials. These
	// will override the default credentials chain. (Optional)
	Credentials aws.CredentialsProvider
	// SharedCredentialsFilepath, when not empty, will override the default
	// credentials chain and the Credentials value (see above). (Optional)
	SharedCredentialsFilepath string
	// SharedCredentialsProfile, when not empty, will fetch the given
	// credentials profile from the shared credentials file. (Optional)
	SharedCredentialsProfile string
	// AssumeRoleARN specifies an IAM role ARN. When not empty, it will be
	// used to assume the given role for this session. See
	// `https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html` for
	// more information. (Optional)
	AssumeRoleARN string
	// AssumeRoleOptions provide a mechanism to override defaults by
	// applying changes to the AssumeRoleProvider struct created with this
	// session. This field is ignored if AssumeRoleARN is not set.
	// (Optional)
	AssumeRoleOptions []func(*stscreds.AssumeRoleOptions)
	// Region specifies the AWS region.
	Region string
	// Name specifies the name of the bucket.
	Name string
	// Prefix specifies the prefix to use. (Optional)
	Prefix string
	// Permissions sets the S3 permissions to use for each object. Defaults
	// to FULL_CONTROL. See
	// `https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html`
	// for more information.
	Permissions S3Permissions
	// ContentType sets the standard MIME type of the object data. Defaults
	// to nil. See
	//`https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17`
	// for more information.
	ContentType string
	// StorageClass sets the S3 storage class for uploaded objects. Defaults
	// to S3's default storage class (STANDARD) if not set. See
	// https://aws.amazon.com/s3/storage-classes/ for more information.
	StorageClass s3Types.StorageClass
	// IfNotExists, when set to true, will avoid overwriting an already-existing
	// object at the destination key if it already exists.
	IfNotExists bool
	// MaxConcurrentCopies specifies the maximum number of concurrent copy
	// operations for MoveObjects. If nil or <=0, defaults to 20.
	// (Optional)
	MaxConcurrentCopies *int
	// VersionID pins download operations to a specific S3 object version.
	// When set, Get and Reader operations will request this version of the
	// object. (Optional)
	VersionID string
}

S3Options support the use and creation of S3 backed buckets.

type S3Permissions

type S3Permissions string

S3Permissions is a type that describes the object canned ACL from S3.

func (S3Permissions) Validate

func (p S3Permissions) Validate() error

Validate checks that the S3Permissions string is valid.

type StreamPutCounter

type StreamPutCounter interface {
	Bucket
	PutWithCount(ctx context.Context, key string, r io.Reader) (int, error)
}

StreamPutCounter is an optional interface implemented by S3 buckets that returns the number of PUT-equivalent API calls made when streaming content to S3.

type StreamPutCounterWithBytes

type StreamPutCounterWithBytes interface {
	Bucket
	PutWithCountAndBytes(ctx context.Context, key string, r io.Reader) (puts int, bytesWritten int64, err error)
}

StreamPutCounterWithBytes extends StreamPutCounter to also return bytes written to S3.

type SyncBucket

type SyncBucket interface {
	// Sync methods: these methods are the recursive, efficient
	// copy methods of files from S3 to the local file
	// system.
	Push(context.Context, SyncOptions) error
	Pull(context.Context, SyncOptions) error
}

SyncBucket defines an interface to access a remote blob store and synchronize the local file system tree with the remote store.

type SyncOptions

type SyncOptions struct {
	Local   string
	Remote  string
	Exclude string
}

SyncOptions describes the arguments to the sync operations (Push and Pull). Note that exclude is a regular expression.

type Transition

type Transition struct {
	// Days is the number of days after creation when the transition occurs (nil if Date is used instead)
	Days *int32

	// StorageClass is the target storage class (e.g., "STANDARD_IA", "GLACIER", "DEEP_ARCHIVE")
	StorageClass string
}

Transition represents a storage class transition with timing information.

Directories

Path Synopsis
cmd
run-linter command
verify-mod-tidy command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL