Documentation
¶
Overview ¶
Package stow provides an abstraction on cloud storage capabilities.
Index ¶
- Constants
- Variables
- func ETagContext(ctx context.Context, i Item) (string, error)
- func IsCursorEnd(cursor string) bool
- func IsNotSupported(err error) bool
- func KindByURL(u *url.URL) (string, error)
- func Kinds() []string
- func LastModContext(ctx context.Context, i Item) (time.Time, error)
- func MetadataContext(ctx context.Context, i Item) (map[string]any, error)
- func NotSupported(feature string) error
- func OpenContext(ctx context.Context, i Item) (io.ReadCloser, error)
- func OpenRangeContext(ctx context.Context, i ItemRanger, start, end uint64) (io.ReadCloser, error)
- func Register(kind string, makefn func(Config) (Location, error), ...)
- func RemoveContainerContext(ctx context.Context, l Location, id string) error
- func RemoveItemContext(ctx context.Context, c Container, id string) error
- func TagsContext(ctx context.Context, i Taggable) (map[string]any, error)
- func Validate(kind string, config Config) error
- func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error
- func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error
- type ClientMethod
- type Config
- type ConfigMap
- type Container
- type ContextContainer
- type ContextItem
- type ContextItemRanger
- type ContextLocation
- type ContextTaggable
- type Copier
- type HttpMethod
- type Item
- func ItemByURLContext(ctx context.Context, l Location, url *url.URL) (Item, error)
- func ItemContext(ctx context.Context, c Container, id string) (Item, error)
- func ItemsContext(ctx context.Context, c Container, prefix, cursor string, count int) ([]Item, string, error)
- func PutContext(ctx context.Context, c Container, name string, r io.Reader, size int64, ...) (Item, error)
- func StreamCopy(ctx context.Context, dst Container, src Item, name string) (Item, error)
- type ItemRanger
- type Location
- type PresignRequestParams
- type PresignResponse
- type Taggable
- type WalkContainersFunc
- type WalkFunc
Constants ¶
const FlyteContentMD5 = "flyteContentMD5"
Variables ¶
var ( // ErrNotFound is returned when something could not be found. ErrNotFound = errors.New("not found") // ErrBadCursor is returned by paging methods when the specified // cursor is invalid. ErrBadCursor = errors.New("bad cursor") )
var ( // CursorStart is a string representing a cursor pointing // to the first page of items or containers. CursorStart = "" // NoPrefix is a string representing no prefix. It can be used // in any function that asks for a prefix value, but where one is // not appropriate. NoPrefix = "" )
Functions ¶
func ETagContext ¶ added in v0.5.0
ETagContext gets the ETag of an item like Item.ETag. An item that is not a ContextItem only has the context checked before the call.
func IsCursorEnd ¶
IsCursorEnd checks whether the cursor indicates there are no more items or not.
func IsNotSupported ¶
IsNotSupported gets whether the error is due to a feature not being supported by a specific implementation.
func KindByURL ¶
KindByURL gets the kind represented by the given URL. It consults all registered locations. Error returned if no match is found.
func LastModContext ¶ added in v0.5.0
LastModContext gets the last modified date of an item like Item.LastMod. An item that is not a ContextItem only has the context checked before the call.
func MetadataContext ¶ added in v0.5.0
MetadataContext gets the metadata of an item like Item.Metadata. An item that is not a ContextItem only has the context checked before the call.
func NotSupported ¶
NotSupported gets an error describing the feature as not supported by this implementation.
func OpenContext ¶ added in v0.5.0
OpenContext opens an item like Item.Open. For a ContextItem the context also covers reading from the returned io.ReadCloser. An item that is not a ContextItem only has the context checked before the call.
func OpenRangeContext ¶ added in v0.5.0
func OpenRangeContext(ctx context.Context, i ItemRanger, start, end uint64) (io.ReadCloser, error)
OpenRangeContext opens a part of an item like ItemRanger.OpenRange. For a ContextItemRanger the context also covers reading from the returned io.ReadCloser. An item that is not a ContextItemRanger only has the context checked before the call.
func Register ¶
func Register(kind string, makefn func(Config) (Location, error), kindmatchfn func(*url.URL) bool, validatefn func(Config) error)
Register adds a Location implementation, with two helper functions. makefn should make a Location with the given Config. kindmatchfn should inspect a URL and return whether it represents a Location of this kind or not. Code can call KindByURL to get a kind string for any given URL and all registered implementations will be consulted. Register is usually called in an implementation package's init method.
func RemoveContainerContext ¶ added in v0.5.0
RemoveContainerContext removes a container like Location.RemoveContainer. A location that is not a ContextLocation only has the context checked before the call.
func RemoveItemContext ¶ added in v0.5.0
RemoveItemContext removes an item like Container.RemoveItem. A container that is not a ContextContainer only has the context checked before the call.
func TagsContext ¶ added in v0.5.0
TagsContext gets the tags of an item like Taggable.Tags. An item that is not a ContextTaggable only has the context checked before the call.
func Walk ¶
Walk walks all Items in the Container. Returns the first error returned by the WalkFunc or nil if no errors were returned. The pageSize is the number of Items to get per request.
func WalkContainers ¶
func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error
WalkContainers walks all Containers in the Location. Returns the first error returned by the WalkContainersFunc or nil if no errors were returned. The pageSize is the number of Containers to get per request.
Types ¶
type ClientMethod ¶
type ClientMethod int
ClientMethod defines common client methods across storage providers
const ( ClientMethodGet ClientMethod = iota ClientMethodPut )
func ClientMethodString ¶
func ClientMethodString(s string) (ClientMethod, error)
ClientMethodString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.
func ClientMethodValues ¶
func ClientMethodValues() []ClientMethod
ClientMethodValues returns all values of the enum
func (ClientMethod) IsAClientMethod ¶
func (i ClientMethod) IsAClientMethod() bool
IsAClientMethod returns "true" if the value is listed in the enum definition. "false" otherwise
func (ClientMethod) MarshalJSON ¶
func (i ClientMethod) MarshalJSON() ([]byte, error)
MarshalJSON implements the json.Marshaler interface for ClientMethod
func (ClientMethod) String ¶
func (i ClientMethod) String() string
func (*ClientMethod) UnmarshalJSON ¶
func (i *ClientMethod) UnmarshalJSON(data []byte) error
UnmarshalJSON implements the json.Unmarshaler interface for ClientMethod
type Config ¶
type Config interface {
// Config gets a string configuration value and a
// bool indicating whether the value was present or not.
Config(name string) (string, bool)
// Set sets the configuration name to specified value
Set(name, value string)
}
Config represents key/value configuration.
type ConfigMap ¶
ConfigMap is a map[string]string that implements the Config method.
type Container ¶
type Container interface {
// ID gets a unique string describing this Container.
ID() string
// Name gets a human-readable name describing this Container.
Name() string
// Item gets an item by its ID.
Item(id string) (Item, error)
// Items gets a page of items with the specified
// prefix for this Container.
// The specified cursor is a pointer to the start of
// the items to get. It should be obtained from a previous
// call to this method, or should be CursorStart for the
// first page.
// count is the number of items to return per page.
// The returned cursor can be checked with IsCursorEnd to
// decide if there are any more items or not.
Items(prefix, cursor string, count int) ([]Item, string, error)
// RemoveItem removes the Item with the specified ID.
RemoveItem(id string) error
// Put creates a new Item with the specified name, and contents
// read from the reader.
Put(name string, r io.Reader, size int64, metadata map[string]any) (Item, error)
// PreSignRequest generates a pre-signed url for the given id (key after bucket/container) and a given clientMethod.
PreSignRequest(ctx context.Context, clientMethod ClientMethod, id string, params PresignRequestParams) (response PresignResponse, err error)
}
Container represents a container.
func ContainerContext ¶ added in v0.5.0
ContainerContext gets a container like Location.Container. A location that is not a ContextLocation only has the context checked before the call.
func ContainersContext ¶ added in v0.5.0
func ContainersContext(ctx context.Context, l Location, prefix, cursor string, count int) ([]Container, string, error)
ContainersContext gets a page of containers like Location.Containers. A location that is not a ContextLocation only has the context checked before the call.
type ContextContainer ¶ added in v0.5.0
type ContextContainer interface {
// ItemContext is Item with a context.
ItemContext(ctx context.Context, id string) (Item, error)
// ItemsContext is Items with a context.
ItemsContext(ctx context.Context, prefix, cursor string, count int) ([]Item, string, error)
// RemoveItemContext is RemoveItem with a context.
RemoveItemContext(ctx context.Context, id string) error
// PutContext is Put with a context.
PutContext(ctx context.Context, name string, r io.Reader, size int64, metadata map[string]any) (Item, error)
}
ContextContainer is a Container whose requests take a context.
type ContextItem ¶ added in v0.5.0
type ContextItem interface {
// OpenContext is Open with a context. The context also covers reading
// from the returned io.ReadCloser, so it must not be cancelled before
// the content is read.
OpenContext(ctx context.Context) (io.ReadCloser, error)
// ETagContext is ETag with a context.
ETagContext(ctx context.Context) (string, error)
// LastModContext is LastMod with a context.
LastModContext(ctx context.Context) (time.Time, error)
// MetadataContext is Metadata with a context.
MetadataContext(ctx context.Context) (map[string]any, error)
}
ContextItem is an Item whose requests take a context.
type ContextItemRanger ¶ added in v0.5.0
type ContextItemRanger interface {
// OpenRangeContext is OpenRange with a context. The context also covers
// reading from the returned io.ReadCloser.
OpenRangeContext(ctx context.Context, start, end uint64) (io.ReadCloser, error)
}
ContextItemRanger is an ItemRanger whose requests take a context.
type ContextLocation ¶ added in v0.5.0
type ContextLocation interface {
// CreateContainerContext is CreateContainer with a context.
CreateContainerContext(ctx context.Context, name string) (Container, error)
// ContainersContext is Containers with a context.
ContainersContext(ctx context.Context, prefix, cursor string, count int) ([]Container, string, error)
// ContainerContext is Container with a context.
ContainerContext(ctx context.Context, id string) (Container, error)
// RemoveContainerContext is RemoveContainer with a context.
RemoveContainerContext(ctx context.Context, id string) error
// ItemByURLContext is ItemByURL with a context.
ItemByURLContext(ctx context.Context, url *url.URL) (Item, error)
}
ContextLocation is a Location whose requests take a context.
type ContextTaggable ¶ added in v0.5.0
type ContextTaggable interface {
// TagsContext is Tags with a context.
TagsContext(ctx context.Context) (map[string]any, error)
}
ContextTaggable is a Taggable whose requests take a context.
type Copier ¶ added in v0.4.0
type Copier interface {
// Copy copies src into the container as an item called name, replacing
// any existing item, and returns the new item. When src comes from the
// same kind of location and the backend supports it, the copy is made
// on the server side and the content never passes through the caller.
// Otherwise the content is streamed from src to the container.
Copy(ctx context.Context, src Item, name string) (Item, error)
}
Copier is implemented by containers that can copy an item into themselves.
type HttpMethod ¶
type HttpMethod = string
HttpMethod defines an alias type for string to represent http Methods. These are defined in RFC 7231 section 4.3.
type Item ¶
type Item interface {
// ID gets a unique string describing this Item.
ID() string
// Name gets a human-readable name describing this Item.
Name() string
// URL gets a URL for this item.
// For example:
// local: file:///path/to/something
// azure: azure://host:port/api/something
// s3: s3://host:post/etc
URL() *url.URL
// Size gets the size of the Item's contents in bytes.
Size() (int64, error)
// Open opens the Item for reading.
// Calling code must close the io.ReadCloser.
Open() (io.ReadCloser, error)
// ETag is a string that is different when the Item is
// different, and the same when the item is the same.
// Usually this is the last modified datetime.
ETag() (string, error)
// LastMod returns the last modified date of the file.
LastMod() (time.Time, error)
// Metadata gets a map of key/values that belong
// to this Item.
Metadata() (map[string]any, error)
}
Item represents an item inside a Container. Such as a file.
func ItemByURLContext ¶ added in v0.5.0
ItemByURLContext gets an item like Location.ItemByURL. A location that is not a ContextLocation only has the context checked before the call.
func ItemContext ¶ added in v0.5.0
ItemContext gets an item like Container.Item. A container that is not a ContextContainer only has the context checked before the call.
func ItemsContext ¶ added in v0.5.0
func ItemsContext(ctx context.Context, c Container, prefix, cursor string, count int) ([]Item, string, error)
ItemsContext gets a page of items like Container.Items. A container that is not a ContextContainer only has the context checked before the call.
func PutContext ¶ added in v0.5.0
func PutContext(ctx context.Context, c Container, name string, r io.Reader, size int64, metadata map[string]any) (Item, error)
PutContext creates an item like Container.Put. A container that is not a ContextContainer only has the context checked before the call.
func StreamCopy ¶ added in v0.4.0
StreamCopy copies src into dst as an item called name by streaming the content through the caller. It is what Copier implementations fall back to when the backend cannot copy src on the server side.
type ItemRanger ¶
type ItemRanger interface {
// OpenRange opens the item for reading starting at byte start and ending
// at byte end.
OpenRange(start, end uint64) (io.ReadCloser, error)
}
ItemRanger represents an item that can be partially downloaded.
type Location ¶
type Location interface {
io.Closer
// CreateContainer creates a new Container with the
// specified name.
CreateContainer(name string) (Container, error)
// Containers gets a page of containers
// with the specified prefix from this Location.
// The specified cursor is a pointer to the start of
// the containers to get. It it obtained from a previous
// call to this method, or should be CursorStart for the
// first page.
// count is the number of items to return per page.
// The returned cursor can be checked with IsCursorEnd to
// decide if there are any more items or not.
Containers(prefix string, cursor string, count int) ([]Container, string, error)
// Container gets the Container with the specified
// identifier.
Container(id string) (Container, error)
// RemoveContainer removes the container with the specified ID.
RemoveContainer(id string) error
// ItemByURL gets an Item at this location with the
// specified URL.
ItemByURL(url *url.URL) (Item, error)
}
Location represents a storage location.
type PresignRequestParams ¶
type PresignResponse ¶ added in v0.3.9
type Taggable ¶
type Taggable interface {
// Tags returns a list of tags that belong to a given Item
Tags() (map[string]any, error)
}
Taggable represents a taggable Item
type WalkContainersFunc ¶
WalkContainersFunc is a function called for each Container visited by WalkContainers. If there was a problem, the incoming error will describe the problem and the function can decide how to handle that error. If an error is returned, processing stops.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package azure provides an abstraction for the Microsoft Azure Storage service.
|
Package azure provides an abstraction for the Microsoft Azure Storage service. |
|
Package google provides an abstraction of Google Cloud Storage.
|
Package google provides an abstraction of Google Cloud Storage. |
|
Package local provides an abstraction of a general filesystem.
|
Package local provides an abstraction of a general filesystem. |
|
Package oracle provides an absraction of the Oracle Storage Cloud Service.
|
Package oracle provides an absraction of the Oracle Storage Cloud Service. |
|
Package s3 provides an abstraction of Amazon S3 (Simple Storage Service).
|
Package s3 provides an abstraction of Amazon S3 (Simple Storage Service). |
|
Package swift provides an absraction of the Openstack Swift storage technology.
|
Package swift provides an absraction of the Openstack Swift storage technology. |