README
¶
Google Cloud Storage Stow Implementation
Location = Google Cloud Storage
Container = Bucket
Item = File
How to access underlying service types
Use a type conversion to extract the underlying Location, Container, or Item implementations. Then use the Google-specific getters to access the internal Google Cloud Storage Service, Bucket, and Object values.
import (
"log"
"github.com/graymeta/stow"
stowgs "github.com/graymeta/stow/google"
)
stowLoc, err := stow.Dial(stowgs.Kind, stow.ConfigMap{
stowgs.ConfigJSON: "<json config>",
stowgs.ConfigProjectId: "<project id>",
})
if err != nil {
log.Fatal(err)
}
stowBucket, err = stowLoc.Container("mybucket")
if err != nil {
log.Fatal(err)
}
if gsBucket, ok := stowBucket.(*stowgs.Container); ok {
if gsLoc, ok := stowLoc.(*stowgs.Location); ok {
googleService := gsLoc.Service()
googleBucket, err := gsBucket.Bucket()
// < Send platform-specific commands here >
}
}
By default, Stow uses https://www.googleapis.com/auth/devstorage.read_write scope. Different scopes can be used by passing a comma separated list of scopes, like below:
stowLoc, err := stow.Dial(stowgs.Kind, stow.ConfigMap{
stowgs.ConfigJSON: "<json config>",
stowgs.ConfigProjectId: "<project id>",
stowgs.ConfigScopes: "<scope_1>,<scope_2>",
})
Configuration... You need to create a project in google, and then create a service account in google tied to that project. You will need to download a .json file with the configuration for the service account. To run the test suite, the service account will need edit privileges inside the project.
To run the test suite, set the GOOGLE_CREDENTIALS_FILE environment variable to point to the location of the .json file containing the service account credentials and set GOOGLE_PROJECT_ID to the project ID, otherwise the test suite will not be run.
Concerns:
- Google's storage plaform is more eventually consistent than other platforms. Sometimes, the tests appear to be flaky because of this. One example is when deleting files from a bucket, then immediately deleting the bucket...sometimes the bucket delete will fail saying that the bucket isn't empty simply because the file delete messages haven't propagated through Google's infrastructure. We may need to add some delay into the test suite to account for this.
Documentation
¶
Overview ¶
Package google provides an abstraction of Google Cloud Storage. In this package, a Google Cloud Storage Bucket is represented by a Stow Container and a Google Cloud Storage Object is represented by a Stow Item. Note that directories may exist within a Bucket.
Usage and Credentials ¶
A path to the JSON file representing configuration information for the service account is needed, as well as the Project ID that it is tied to.
stow.Dial requires both a string value of the particular Stow Location Kind ("google") and a stow.Config instance. The stow.Config instance requires two entries with the specific key value attributes:
- a key of google.ConfigJSON with a value of the path of the JSON configuration file - a key of google.ConfigProjectID with a value of the Project ID
Location ¶
There are google.location methods which allow the retrieval of a Google Cloud Storage Object (Container or Containers). An Object can also be retrieved based on the its URL (ItemByURL).
Additional google.location methods provide capabilities to create and remove Google Cloud Storage Buckets (CreateContainer or RemoveContainer).
Container ¶
Methods of stow.container allow the retrieval of a Google Bucket's:
- name(ID or Name) - object or complete list of objects (Item or Items, respectively)
Additional methods of google.container allow Stow to:
- remove an Object (RemoveItem) - update or create an Object (Put)
Item ¶
Methods of google.Item allow the retrieval of a Google Cloud Storage Object's: - name (ID or name) - URL - size in bytes - Object specific metadata (information stored within the Google Cloud Service) - last modified date - Etag - content
Index ¶
- Constants
- type Container
- func (c *Container) Bucket() *storage.BucketHandle
- func (c *Container) Copy(ctx context.Context, src stow.Item, name string) (stow.Item, error)
- func (c *Container) ID() string
- func (c *Container) Item(id string) (stow.Item, error)
- func (c *Container) ItemContext(ctx context.Context, id string) (stow.Item, error)
- func (c *Container) Items(prefix string, cursor string, count int) ([]stow.Item, string, error)
- func (c *Container) ItemsContext(ctx context.Context, prefix string, cursor string, count int) ([]stow.Item, string, error)
- func (c *Container) Name() string
- func (c *Container) PreSignRequest(_ context.Context, clientMethod stow.ClientMethod, id string, ...) (response stow.PresignResponse, err error)
- func (c *Container) Put(name string, r io.Reader, size int64, metadata map[string]any) (stow.Item, error)
- func (c *Container) PutContext(ctx context.Context, name string, r io.Reader, size int64, ...) (stow.Item, error)
- func (c *Container) RemoveItem(id string) error
- func (c *Container) RemoveItemContext(ctx context.Context, id string) error
- type Item
- func (i *Item) ETag() (string, error)
- func (i *Item) ETagContext(_ context.Context) (string, error)
- func (i *Item) ID() string
- func (i *Item) LastMod() (time.Time, error)
- func (i *Item) LastModContext(_ context.Context) (time.Time, error)
- func (i *Item) Metadata() (map[string]any, error)
- func (i *Item) MetadataContext(_ context.Context) (map[string]any, error)
- func (i *Item) Name() string
- func (i *Item) Open() (io.ReadCloser, error)
- func (i *Item) OpenContext(ctx context.Context) (io.ReadCloser, error)
- func (i *Item) OpenRange(start, end uint64) (io.ReadCloser, error)
- func (i *Item) OpenRangeContext(ctx context.Context, start, end uint64) (io.ReadCloser, error)
- func (i *Item) Size() (int64, error)
- func (i *Item) StorageObject() *storage.ObjectAttrs
- func (i *Item) URL() *url.URL
- type Location
- func (l *Location) Close() error
- func (l *Location) Container(id string) (stow.Container, error)
- func (l *Location) ContainerContext(ctx context.Context, id string) (stow.Container, error)
- func (l *Location) Containers(prefix string, cursor string, count int) ([]stow.Container, string, error)
- func (l *Location) ContainersContext(ctx context.Context, prefix string, cursor string, count int) ([]stow.Container, string, error)
- func (l *Location) CreateContainer(containerName string) (stow.Container, error)
- func (l *Location) CreateContainerContext(ctx context.Context, containerName string) (stow.Container, error)
- func (l *Location) ItemByURL(url *url.URL) (stow.Item, error)
- func (l *Location) ItemByURLContext(ctx context.Context, url *url.URL) (stow.Item, error)
- func (l *Location) RemoveContainer(id string) error
- func (l *Location) RemoveContainerContext(ctx context.Context, id string) error
- func (l *Location) Service() *storage.Client
Constants ¶
const ( // The service account json blob ConfigJSON = "json" ConfigProjectId = "project_id" ConfigScopes = "scopes" )
const Kind = "google"
Kind represents the name of the location/storage type.
const Protocol = "gs"
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Container ¶
type Container struct {
// contains filtered or unexported fields
}
func (*Container) Bucket ¶
func (c *Container) Bucket() *storage.BucketHandle
Bucket returns the google bucket attributes
func (*Container) Copy ¶ added in v0.4.0
Copy copies src into the container on the Google Cloud Storage side. The metadata of the source object is kept. An item from another kind of location is streamed instead.
The copy is one request made with the credentials of this container. An item of another location may be readable only with its own credentials, so when Google Cloud Storage refuses the copy or does not find the source, the item is streamed through its own client instead.
func (*Container) Item ¶
Item returns a stow.Item instance of a container based on the name of the container
func (*Container) ItemContext ¶ added in v0.5.0
ItemContext is Item with a context.
func (*Container) Items ¶
Items retrieves a list of items that are prepended with the prefix argument. The 'cursor' variable facilitates pagination.
func (*Container) ItemsContext ¶ added in v0.5.0
func (c *Container) ItemsContext(ctx context.Context, prefix string, cursor string, count int) ([]stow.Item, string, error)
ItemsContext is Items with a context.
func (*Container) PreSignRequest ¶
func (c *Container) PreSignRequest(_ context.Context, clientMethod stow.ClientMethod, id string, params stow.PresignRequestParams) (response stow.PresignResponse, err error)
func (*Container) Put ¶
func (c *Container) Put(name string, r io.Reader, size int64, metadata map[string]any) (stow.Item, error)
Put sends a request to upload content to the container. The arguments received are the name of the item, a reader representing the content, and the size of the file.
func (*Container) PutContext ¶ added in v0.5.0
func (c *Container) PutContext(ctx context.Context, name string, r io.Reader, size int64, metadata map[string]any) (stow.Item, error)
PutContext is Put with a context.
func (*Container) RemoveItem ¶
RemoveItem will delete a google storage Object
type Item ¶
type Item struct {
// contains filtered or unexported fields
}
func (*Item) ETagContext ¶ added in v0.5.0
ETagContext is ETag with a context.
func (*Item) LastModContext ¶ added in v0.5.0
LastModContext is LastMod with a context.
func (*Item) MetadataContext ¶ added in v0.5.0
MetadataContext is Metadata with a context.
func (*Item) Open ¶
func (i *Item) Open() (io.ReadCloser, error)
Open returns an io.ReadCloser to the object. Useful for downloading/streaming the object.
func (*Item) OpenContext ¶ added in v0.5.0
OpenContext is Open with a context.
func (*Item) OpenRange ¶
func (i *Item) OpenRange(start, end uint64) (io.ReadCloser, error)
OpenRange returns an io.Reader to the object for a specific byte range
func (*Item) OpenRangeContext ¶ added in v0.5.0
OpenRangeContext is OpenRange with a context.
func (*Item) StorageObject ¶
func (i *Item) StorageObject() *storage.ObjectAttrs
Object returns the Google Storage Object
type Location ¶
type Location struct {
// contains filtered or unexported fields
}
A Location contains a client + the configurations used to create the client.
func (*Location) Close ¶
Close simply satisfies the Location interface. There's nothing that needs to be done in order to satisfy the interface.
func (*Location) Container ¶
Container retrieves a stow.Container based on its name which must be exact.
func (*Location) ContainerContext ¶ added in v0.5.0
ContainerContext is Container with a context.
func (*Location) Containers ¶
func (l *Location) Containers(prefix string, cursor string, count int) ([]stow.Container, string, error)
Containers returns a slice of the Container interface, a cursor, and an error.
func (*Location) ContainersContext ¶ added in v0.5.0
func (l *Location) ContainersContext(ctx context.Context, prefix string, cursor string, count int) ([]stow.Container, string, error)
ContainersContext is Containers with a context.
func (*Location) CreateContainer ¶
CreateContainer creates a new container, in this case a bucket.
func (*Location) CreateContainerContext ¶ added in v0.5.0
func (l *Location) CreateContainerContext(ctx context.Context, containerName string) (stow.Container, error)
CreateContainerContext is CreateContainer with a context.
func (*Location) ItemByURL ¶
ItemByURL retrieves a stow.Item by parsing the URL, in this case an item is an object.
func (*Location) ItemByURLContext ¶ added in v0.5.0
ItemByURLContext is ItemByURL with a context.
func (*Location) RemoveContainer ¶
RemoveContainer removes a container simply by name.
func (*Location) RemoveContainerContext ¶ added in v0.5.0
RemoveContainerContext is RemoveContainer with a context.
Source Files