folderloader

package
v1.59.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 18, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package folderloader holds the provider independent logic of the "folder mode" of the retrievers (S3, GCS, Azure Blob...): list the files under a prefix, download and parse the ones that changed, and merge all of them into a single flag set. The providers only have to implement the Store interface.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Validate

func Validate(itemName, single, prefix, pattern string, recursive bool) error

Validate checks the settings of a retriever supporting the folder mode. single is the path of a single file, an empty single means folder mode, the only mode where prefix, pattern and recursive can be used. itemName is how the provider calls a single file ("item", "object"...), it is only used in the error messages.

Types

type Entry

type Entry struct {
	// Key is the full key (path) of the object.
	Key string
	// Version tells if the content changed (ETag, generation...). It is opaque for the loader,
	// an empty Version never matches the cache.
	Version string
}

Entry is an object found by Store.List.

type Loader

type Loader struct {
	// contains filtered or unexported fields
}

Loader loads and merges the flag files of a folder. A Loader is not safe for concurrent use: the caller must serialize the calls.

func New

func New(store Store, opts Options) *Loader

New creates a Loader reading the files from store.

func Reuse

func Reuse(l *Loader, store Store, opts Options) *Loader

Reuse returns l if it is not nil and was built with the same options, so the cache and the last output format are kept (ex: when a retriever is initialized again with the same settings). Otherwise it returns a new Loader on store, which starts clean because the selection of files changed. The store of l is kept as is when l is reused, it must be equivalent to store.

func (*Loader) Load

func (l *Loader) Load(ctx context.Context) ([]byte, error)

Load lists the matching files, downloads and parses the ones that changed since the last call and merges all of them into a single flag set, encoded in the format of the files (see OutputFormat) so that the flag manager decodes it exactly like a single file.

func (*Loader) OutputFormat

func (l *Loader) OutputFormat() string

OutputFormat is the format of the bytes returned by Load: the format of the files found by the last Load ("json" for an empty flag set). Before the first Load it is guessed from the pattern.

func (*Loader) Reset

func (l *Loader) Reset()

Reset drops the cache and the skipped files.

type Options

type Options struct {
	// Prefix is the folder to load. Leading "/" are trimmed and a trailing "/" is added if missing.
	// "" and "/" are the root.
	Prefix string
	// Pattern is an optional glob (see path.Match) matched against the file name only.
	Pattern string
	// Recursive also loads the "sub folders" of Prefix.
	Recursive bool
	// Logger is used for the warnings, it can be nil.
	Logger *fflog.FFLogger
	// Location is the name of the bucket / container, used in the logs.
	Location string
	// LocationKey is the log field holding Location (ex: "bucket"), "location" if empty.
	LocationKey string
	// Provider is the name of the provider used in the logs and the errors (ex: "S3"). If empty,
	// the wording is provider-neutral.
	Provider string
}

Options are the settings of a Loader.

type Store

type Store interface {
	// List returns the objects under prefix, which is already normalized (no leading "/", a
	// trailing "/" unless empty). If recursive is false only the direct children of prefix must be
	// returned, so the store does the delimiter work. It may return folder markers (keys ending
	// with "/"), the loader ignores them.
	List(ctx context.Context, prefix string, recursive bool) ([]Entry, error)
	// Get downloads an object, and returns the version of what it actually served. It may be ""
	// if the provider has none, the Version found by List is used then.
	// The version must identify the same revision as the Version of List, so that they can be
	// compared. Differences of quoting are tolerated: a weak prefix (W/) and surrounding double
	// quotes are ignored.
	Get(ctx context.Context, key string) (content []byte, version string, err error)
}

Store is the access to the objects of a provider.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL