Documentation
¶
Overview ¶
Package folderloader holds the provider independent logic of the "folder mode" of the retrievers (S3, GCS, Azure Blob...): list the files under a prefix, download and parse the ones that changed, and merge all of them into a single flag set. The providers only have to implement the Store interface.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Validate ¶
Validate checks the settings of a retriever supporting the folder mode. single is the path of a single file, an empty single means folder mode, the only mode where prefix, pattern and recursive can be used. itemName is how the provider calls a single file ("item", "object"...), it is only used in the error messages.
Types ¶
type Entry ¶
type Entry struct {
// Key is the full key (path) of the object.
Key string
// Version tells if the content changed (ETag, generation...). It is opaque for the loader,
// an empty Version never matches the cache.
Version string
}
Entry is an object found by Store.List.
type Loader ¶
type Loader struct {
// contains filtered or unexported fields
}
Loader loads and merges the flag files of a folder. A Loader is not safe for concurrent use: the caller must serialize the calls.
func Reuse ¶
Reuse returns l if it is not nil and was built with the same options, so the cache and the last output format are kept (ex: when a retriever is initialized again with the same settings). Otherwise it returns a new Loader on store, which starts clean because the selection of files changed. The store of l is kept as is when l is reused, it must be equivalent to store.
func (*Loader) Load ¶
Load lists the matching files, downloads and parses the ones that changed since the last call and merges all of them into a single flag set, encoded in the format of the files (see OutputFormat) so that the flag manager decodes it exactly like a single file.
func (*Loader) OutputFormat ¶
OutputFormat is the format of the bytes returned by Load: the format of the files found by the last Load ("json" for an empty flag set). Before the first Load it is guessed from the pattern.
type Options ¶
type Options struct {
// Prefix is the folder to load. Leading "/" are trimmed and a trailing "/" is added if missing.
// "" and "/" are the root.
Prefix string
// Pattern is an optional glob (see path.Match) matched against the file name only.
Pattern string
// Recursive also loads the "sub folders" of Prefix.
Recursive bool
// Logger is used for the warnings, it can be nil.
Logger *fflog.FFLogger
// Location is the name of the bucket / container, used in the logs.
Location string
// LocationKey is the log field holding Location (ex: "bucket"), "location" if empty.
LocationKey string
// Provider is the name of the provider used in the logs and the errors (ex: "S3"). If empty,
// the wording is provider-neutral.
Provider string
}
Options are the settings of a Loader.
type Store ¶
type Store interface {
// List returns the objects under prefix, which is already normalized (no leading "/", a
// trailing "/" unless empty). If recursive is false only the direct children of prefix must be
// returned, so the store does the delimiter work. It may return folder markers (keys ending
// with "/"), the loader ignores them.
List(ctx context.Context, prefix string, recursive bool) ([]Entry, error)
// Get downloads an object, and returns the version of what it actually served. It may be ""
// if the provider has none, the Version found by List is used then.
// The version must identify the same revision as the Version of List, so that they can be
// compared. Differences of quoting are tolerated: a weak prefix (W/) and surrounding double
// quotes are ignored.
Get(ctx context.Context, key string) (content []byte, version string, err error)
}
Store is the access to the objects of a provider.