s3

package
v7.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 8, 2025 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package s3 implements vfs.FileSystem for AWS S3.

Index

Constants

View Source 
const Scheme = "s3"

Scheme defines the file system type.

Variables

This section is empty.

Functions

func GetClient added in v7.1.0 

func GetClient(opt Options) (*s3.Client, error)

GetClient setup S3 client

func StringToACL added in v7.1.0 

func StringToACL(acl string) types.ObjectCannedACL

StringToACL converts a string to an ObjectCannedACL see https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3/types#ObjectCannedACL

func WithClient 

func WithClient(c Client) options.NewFileSystemOption[FileSystem]

WithClient returns clientOpt implementation of NewFileOption

WithClient is used to explicitly specify a Client to use for the filesystem. The client is used to interact with the S3 service.

func WithOptions 

func WithOptions(o Options) options.NewFileSystemOption[FileSystem]

WithOptions returns optionsOpt implementation of NewFileOption

WithOptions is used to specify options for the filesystem. The options are used to configure the filesystem.

Types

type Client 

type Client interface {
	manager.DownloadAPIClient
	manager.UploadAPIClient
	CopyObject(ctx context.Context, in *s3.CopyObjectInput, opts ...func(*s3.Options)) (*s3.CopyObjectOutput, error)
	DeleteObject(ctx context.Context, in *s3.DeleteObjectInput, opts ...func(*s3.Options)) (*s3.DeleteObjectOutput, error)
	HeadBucket(ctx context.Context, in *s3.HeadBucketInput, opts ...func(*s3.Options)) (*s3.HeadBucketOutput, error)
	HeadObject(ctx context.Context, in *s3.HeadObjectInput, opts ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
	ListObjects(ctx context.Context, in *s3.ListObjectsInput, opts ...func(*s3.Options)) (*s3.ListObjectsOutput, error)
	ListObjectVersions(ctx context.Context, in *s3.ListObjectVersionsInput, opts ...func(*s3.Options)) (*s3.ListObjectVersionsOutput, error)
}

type File 

type File struct {
	// contains filtered or unexported fields
}

File implements vfs.File interface for S3 fs.

func (*File) Close 

func (f *File) Close() error

Close cleans up underlying mechanisms for reading from and writing to the file. Closes and removes the local temp file, and triggers a Write to S3 of anything in the f.writeBuffer if it has been created.

func (*File) CopyToFile 

func (f *File) CopyToFile(file vfs.File) (err error)

CopyToFile puts the contents of File into the targetFile passed. Uses the S3 CopyObject method if the target file is also on S3, otherwise uses io.CopyBuffer.

func (*File) CopyToLocation 

func (f *File) CopyToLocation(location vfs.Location) (vfs.File, error)

CopyToLocation creates a copy of *File, using the file's current name as the new file's name at the given location. If the given location is also s3, the AWS API for copying files will be utilized, otherwise, standard io.Copy will be done to the new file.

func (*File) Delete 

func (f *File) Delete(opts ...options.DeleteOption) error

Delete clears any local temp file, or write buffer from read/writes to the file, then makes a DeleteObject call to s3 for the file. If delete.AllVersions option is provided, DeleteObject call is made to s3 for each version of the file. Returns any error returned by the API.

func (*File) Exists 

func (f *File) Exists() (bool, error)

Exists returns whether (boolean) the object exists on s3, based on a call for the object's HEAD through the s3 API.

func (*File) LastModified 

func (f *File) LastModified() (*time.Time, error)

LastModified returns the LastModified property of a HEAD request to the s3 object.

func (*File) Location 

func (f *File) Location() vfs.Location

Location returns a vfs.Location at the location of the object. IE: if file is at s3://bucket/here/is/the/file.txt the location points to s3://bucket/here/is/the/

func (*File) MoveToFile 

func (f *File) MoveToFile(file vfs.File) error

MoveToFile puts the contents of File into the targetFile passed using File.CopyToFile. If the copy succeeds, the source file is deleted. Any errors from the copy or delete are returned.

func (*File) MoveToLocation 

func (f *File) MoveToLocation(location vfs.Location) (vfs.File, error)

MoveToLocation works by first calling File.CopyToLocation(vfs.Location) then, if that succeeds, it deletes the original file, returning the new file. If the copy process fails the error is returned, and the Delete isn't called. If the call to Delete fails, the error and the file generated by the copy are both returned.

func (*File) Name 

func (f *File) Name() string

Name returns the name portion of the file's key property. IE: "file.txt" of "s3://some/path/to/file.txt

func (*File) Path 

func (f *File) Path() string

Path return the directory portion of the file's key. IE: "path/to" of "s3://some/path/to/file.txt

func (*File) Read 

func (f *File) Read(p []byte) (n int, err error)

Read implements the standard for io.Reader.

func (*File) Seek 

func (f *File) Seek(offset int64, whence int) (int64, error)

Seek implements the standard for io.Seeker.

func (*File) Size 

func (f *File) Size() (uint64, error)

Size returns the ContentLength value from an S3 HEAD request on the file's object.

func (*File) String 

func (f *File) String() string

String implement fmt.Stringer, returning the file's URI as the default string.

func (*File) Touch 

func (f *File) Touch() error

Touch creates a zero-length file on the vfs.File if no File exists. Update File's last modified timestamp. Returns error if unable to touch File.

func (*File) URI 

func (f *File) URI() string

URI returns the File's URI as a string.

func (*File) Write 

func (f *File) Write(data []byte) (int, error)

Write implements the standard for io.Writer. Note that writes are not committed to S3 until Close() is called.

type FileSystem 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem implements vfs.FileSystem for the S3 file system.

func NewFileSystem 

func NewFileSystem(opts ...options.NewFileSystemOption[FileSystem]) *FileSystem

NewFileSystem initializer for FileSystem struct accepts aws-sdk client and returns Filesystem or error.

func (*FileSystem) Client 

func (fs *FileSystem) Client() (Client, error)

Client returns the underlying aws s3 client, creating it, if necessary See Overview for authentication resolution

func (*FileSystem) Name 

func (fs *FileSystem) Name() string

Name returns "AWS S3"

func (*FileSystem) NewFile 

func (fs *FileSystem) NewFile(authorityStr, name string, opts ...options.NewFileOption) (vfs.File, error)

NewFile function returns the s3 implementation of vfs.File.

func (*FileSystem) NewLocation 

func (fs *FileSystem) NewLocation(authorityStr, name string) (vfs.Location, error)

NewLocation function returns the s3 implementation of vfs.Location.

func (*FileSystem) Retry deprecated 

func (fs *FileSystem) Retry() vfs.Retry

Retry will return the default no-op retrier. The S3 client provides its own retryer interface, and is available to override via the s3.FileSystem Options type.

Deprecated: This method is deprecated and will be removed in a future release.

func (*FileSystem) Scheme 

func (fs *FileSystem) Scheme() string

Scheme return "s3" as the initial part of a file URI ie: s3://

func (*FileSystem) WithClient deprecated 

func (fs *FileSystem) WithClient(client interface{}) *FileSystem

WithClient passes in an s3 client and returns the file system (chainable)

Deprecated: This method is deprecated and will be removed in a future release. Use WithClient option: 

fs := s3.NewFileSystem(WithClient(client))

instead of: 

fs := s3.NewFileSystem().WithClient(client)

func (*FileSystem) WithOptions deprecated 

func (fs *FileSystem) WithOptions(opts vfs.Options) *FileSystem

WithOptions sets options for client and returns the file system (chainable)

Deprecated: This method is deprecated and will be removed in a future release. Use WithOptions option: 

fs := s3.NewFileSystem(WithOptions(opts))

instead of: 

fs := s3.NewFileSystem().WithOptions(opts)

type Location 

type Location struct {
	// contains filtered or unexported fields
}

Location implements the vfs.Location interface specific to S3 fs.

func (*Location) Authority 

func (l *Location) Authority() authority.Authority

Authority returns the bucket the location is contained in.

func (*Location) ChangeDir deprecated 

func (l *Location) ChangeDir(relativePath string) error

ChangeDir takes a relative path, and modifies the underlying Location's path. The caller is modified by this so the only return is any error. For this implementation there are no errors.

Deprecated: Use NewLocation instead: 

loc, err := loc.NewLocation("../../")

func (*Location) DeleteFile 

func (l *Location) DeleteFile(fileName string, opts ...options.DeleteOption) error

DeleteFile removes the file at fileName path.

func (*Location) Exists 

func (l *Location) Exists() (bool, error)

Exists returns true if the bucket exists, and the user in the underlying s3.fileSystem.Client() has the appropriate permissions. Will receive false without an error if the bucket simply doesn't exist. Otherwise could receive false and any errors passed back from the API.

func (*Location) FileSystem 

func (l *Location) FileSystem() vfs.FileSystem

FileSystem returns a vfs.FileSystem interface of the location's underlying file system.

func (*Location) List 

func (l *Location) List() ([]string, error)

List calls the s3 API to list all objects in the location's bucket, with a prefix automatically set to the location's path. This will make a call to the s3 API for every 1000 keys to return. If you have many thousands of keys at the given location, this could become quite expensive.

func (*Location) ListByPrefix 

func (l *Location) ListByPrefix(prefix string) ([]string, error)

ListByPrefix calls the s3 API with the location's prefix modified relatively by the prefix arg passed to the function. The resource considerations of List() apply to this function as well.

func (*Location) ListByRegex 

func (l *Location) ListByRegex(regex *regexp.Regexp) ([]string, error)

ListByRegex retrieves the keys of all the files at the location's current path, then filters out all those that don't match the given regex. The resource considerations of List() apply here as well.

func (*Location) NewFile 

func (l *Location) NewFile(relFilePath string, opts ...options.NewFileOption) (vfs.File, error)

NewFile uses the properties of the calling location to generate a vfs.File (backed by an s3.File). The filePath argument is expected to be a relative path to the location's current path.

func (*Location) NewLocation 

func (l *Location) NewLocation(relativePath string) (vfs.Location, error)

NewLocation makes a copy of the underlying Location, then modifies its path by calling ChangeDir with the relativePath argument, returning the resulting location. The only possible errors come from the call to ChangeDir, which, for the s3 implementation doesn't ever result in an error.

func (*Location) Path 

func (l *Location) Path() string

Path returns the prefix the location references in most s3 calls.

func (*Location) String 

func (l *Location) String() string

String implement fmt.Stringer, returning the location's URI as the default string.

func (*Location) URI 

func (l *Location) URI() string

URI returns the Location's URI as a string.

func (*Location) Volume deprecated 

func (l *Location) Volume() string

Volume returns the bucket the location is contained in.

Deprecated: Use Authority instead. 

authStr := loc.Authority().String()

type Options 

type Options struct {
	AccessKeyID                 string                `json:"accessKeyId,omitempty"`
	SecretAccessKey             string                `json:"secretAccessKey,omitempty"`
	SessionToken                string                `json:"sessionToken,omitempty"`
	Region                      string                `json:"region,omitempty"`
	RoleARN                     string                `json:"roleArn,omitempty"`
	Endpoint                    string                `json:"endpoint,omitempty"`
	ACL                         types.ObjectCannedACL `json:"acl,omitempty"`
	ForcePathStyle              bool                  `json:"forcePathStyle,omitempty"`
	DisableServerSideEncryption bool                  `json:"disableServerSideEncryption,omitempty"`
	// this is explicitly re-enables the AWS SDK V2 log output checksum validation.  Default is false. Set to true to
	// re-enable the log output checksum validation.
	AllowLogOutputChecksumValidationSkipped bool `json:"allowLogOutputChecksumValidationSkipped,omitempty"`
	Retry                                   aws.Retryer
	MaxRetries                              int
	FileBufferSize                          int   // Buffer size in bytes used with utils.TouchCopyBuffered
	DownloadPartitionSize                   int64 // Partition size in bytes used to multipart download of large files using manager.Downloader
	UploadPartitionSize                     int64 // Partition size in bytes used to multipart upload of large files using manager.Uploader
}

Options holds s3-specific options. Currently only client options are used.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.