README
¶
Simpleblob
Simpleblob is a Go module that simplifies the storage of arbitrary data by key from Go code. It ships with the following backends:
s3: S3 bucket storagefs: File storage (one file per blob)memory: Memory storage (for tests)
The interface implemented by the backends is:
type Interface interface {
List(ctx context.Context, prefix string) (BlobList, error)
Load(ctx context.Context, name string) ([]byte, error)
Store(ctx context.Context, name string, data []byte) error
}
We plan to extend this with a Delete before 1.0.
To instantiate a backend, _-import all the backends that you want to register, and call:
func GetBackend(ctx context.Context, typeName string, options map[string]any) (Interface, error)
An example can be found in example_test.go.
Every backend accepts a map[string]any with options and performs its own validation on the options. If you use a YAML, TOML and JSON, you could structure it like this:
type Storage struct {
Type string `yaml:"type"`
Options map[string]any `yaml:"options"` // backend specific
}
For Go 1.17, replace any by interface{}.
Limitations
The interface currently does not support streaming of large blobs. In the future we may provide this by implementing fs.FS in the backend for reading, and a similar interface for writing new blobs.
API Stability
We support the last two stable Go versions, currently 1.17 and 1.18.
From a API consumer point of view, we do not plan any backward incompatible changes before a v1.0.
If you want to implement a storage backend, be aware that we will probably add a Delete method before v1.0.
Any future extensions most likely be added with optional interface, similar to the fs.FS design. Utility functions that return a compatible implementation will be used for backends that do not implement the interface, if possible.
Documentation
¶
Overview ¶
Example ¶
package main
import (
"context"
"fmt"
"github.com/PowerDNS/simpleblob"
// Register the memory backend plugin
_ "github.com/PowerDNS/simpleblob/backends/memory"
)
func main() {
// Do not forget the:
// import _ "github.com/PowerDNS/simpleblob/backends/memory"
ctx := context.Background()
storage, err := simpleblob.GetBackend(ctx, "memory", map[string]interface{}{})
check(err)
err = storage.Store(ctx, "example.txt", []byte("hello"))
check(err)
data, err := storage.Load(ctx, "example.txt")
check(err)
fmt.Println("data:", string(data))
list, err := storage.List(ctx, "")
check(err)
for _, entry := range list {
fmt.Println("list:", entry.Name, entry.Size)
}
}
func check(err error) {
if err != nil {
panic(err)
}
}
Output: data: hello list: example.txt 5
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RegisterBackend ¶
RegisterBackend registers a new backend.
Types ¶
type BlobList ¶
type BlobList []Blob
BlobList is a slice of Blob structs
func (BlobList) WithPrefix ¶
WithPrefix filters the BlobList to returns only Blob structs where the name starts with the given prefix.
type InitFunc ¶
type InitFunc func(ctx context.Context, p InitParams) (Interface, error)
InitFunc is the type for the backend constructor function used to register backend.
type InitParams ¶
type InitParams struct {
OptionMap OptionMap // map of key-value options for this backend
}
InitParams contains the parameters for the InitFunc. This allows us to pass extra values in the future, without breaking existing backends that do not expect these.
func (InitParams) OptionsThroughYAML ¶
func (ip InitParams) OptionsThroughYAML(dest interface{}) error
OptionsThroughYAML performs a YAML roundtrip for the OptionMap to load them into a struct with yaml tags. dest: pointer to destination struct
type Interface ¶
type Interface interface {
List(ctx context.Context, prefix string) (BlobList, error)
Load(ctx context.Context, name string) ([]byte, error)
Store(ctx context.Context, name string, data []byte) error
}
Interface defines the interface storage plugins need to implement. This Interface MUST NOT be extended or changed without bumping the major version number, because it would break any backends in external repos. If you want to provide additional features, you can define additional optional interfaces that a backend can implement.
func GetBackend ¶
func GetBackend(ctx context.Context, typeName string, options map[string]interface{}) (Interface, error)
GetBackend creates a new backend instance of given typeName. This type must have been previously registered with RegisterBackend. A context is passed in case a backend needs to do checks that require remote calls. The options map contains backend dependant key-value options. Some backends take no options, others require some specific options.
Source Files
Directories