Documentation
¶
Index ¶
- type Reader
- func (r *Reader) GetNumber() int64
- func (r *Reader) GetSize() int64
- func (r *Reader) GetSkipped() []string
- func (r *Reader) GetType() string
- func (r *Reader) ListObjects(ctx context.Context, path string) ([]string, error)
- func (r *Reader) SetObjectsToStream(list []string)
- func (r *Reader) StreamFile(ctx context.Context, filename string, readersCh chan<- models.File, ...)
- func (r *Reader) StreamFiles(ctx context.Context, readersCh chan<- models.File, errorsCh chan<- error, ...)
- type Writer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// Optional parameters.
options.Options
// contains filtered or unexported fields
}
Reader represents S3 storage reader.
func NewReader ¶
func NewReader( ctx context.Context, client *s3.Client, bucketName string, opts ...options.Opt, ) (*Reader, error)
NewReader returns new S3 storage reader. Must be called with WithDir(path string) or WithFile(path string) - mandatory. Can be called with WithValidator(v validator) - optional. For S3 client next parameters must be set:
- o.UsePathStyle = true
- o.BaseEndpoint = &endpoint - if endpoint != ""
- WithHTTPClient using timeouts to prevent socket locks on errors.
func (*Reader) GetNumber ¶
GetNumber returns the number of asb/asbx files/dirs that was initialized.
func (*Reader) GetSkipped ¶ added in v0.8.0
GetSkipped returns a list of file paths that were skipped during the `StreamFlies` with skipPrefix.
func (*Reader) ListObjects ¶
ListObjects list all object in the path.
func (*Reader) SetObjectsToStream ¶
SetObjectsToStream set objects to stream.
func (*Reader) StreamFile ¶
func (r *Reader) StreamFile( ctx context.Context, filename string, readersCh chan<- models.File, errorsCh chan<- error)
StreamFile opens single file from s3 and sends io.Readers to the `readersCh` In case of an error, it is sent to the `errorsCh` channel.
func (*Reader) StreamFiles ¶
func (r *Reader) StreamFiles( ctx context.Context, readersCh chan<- models.File, errorsCh chan<- error, skipPrefixes []string, )
StreamFiles read files form s3 and send io.Readers to `readersCh` communication chan for lazy loading. In case of error, we send error to `errorsCh` channel. If `skipPrefix` is set, it will skip files that start with this prefix and save a skipped list of files.
type Writer ¶
type Writer struct {
// Optional parameters.
options.Options
// contains filtered or unexported fields
}
Writer represents a s3 storage writer.
func NewWriter ¶
func NewWriter( ctx context.Context, client *s3.Client, bucketName string, opts ...options.Opt, ) (*Writer, error)
NewWriter creates a new writer for S3 storage directory/file writes. Must be called with WithDir(path string) or WithFile(path string) - mandatory. Can be called with WithRemoveFiles() - optional. For S3 client next parameters must be set:
- o.UsePathStyle = true
- o.BaseEndpoint = &endpoint - if endpoint != ""
Source Files
Directories