Documentation
¶
Overview ¶
Package gcs is the Google Cloud Storage blob store (decision 0027). Blobs live at <prefix>sha256/<ab>/<hex>.
GCS objects cannot be appended to, and the Go SDK cannot resume a resumable-upload session from persisted state, so an upload session is built from compose instead:
- Bytes accumulate in a pending buffer of at most PartSize. A blob that never fills one part is written in a single request at commit.
- Each full part becomes its own object under <prefix>_uploads/<id>/, written with a must-not-exist precondition. Part names are never reused across parts, so a retry of the same part either lands or finds its own earlier copy.
- GCS composes at most 32 sources per request, so before a 33rd source the session composes what it has into a fold object and continues from that. There is no limit on how many components a composite object has.
- At commit the content key is known; the session composes everything straight onto it with a must-not-exist precondition and DeleteSourceObjects, so the bytes are never copied. A 412 there means the blob is already stored, which the contract treats as success.
- Composite objects carry a CRC32C but no MD5, so the session tracks the CRC32C of everything appended and checks the result against it; the blob.Hasher state still provides SHA-256, SHA-1 and MD5.
The session (part objects, their generations, sizes and checksums) is persisted through pkg/blob/session after every append.
Index ¶
- Constants
- Variables
- type API
- type Attrs
- type ClientConfig
- type Options
- type SignedURLOptions
- type Source
- type Store
- func (s *Store) BeginUpload(ctx context.Context, opts ...blob.UploadOption) (blob.Upload, error)
- func (s *Store) Delete(ctx context.Context, sha string) error
- func (s *Store) Exists(ctx context.Context, sha string) (bool, error)
- func (s *Store) ObjectKey(sha string) string
- func (s *Store) Open(ctx context.Context, sha string) (io.ReadSeekCloser, blob.Info, error)
- func (s *Store) PartSize() int64
- func (s *Store) ResumeUpload(ctx context.Context, id string) (blob.Upload, error)
- func (s *Store) SignURL(_ context.Context, sha string, o blob.SignOptions) (string, error)
- func (s *Store) Signs(via blob.Via) bool
- func (s *Store) Stat(ctx context.Context, sha string) (blob.Info, error)
- func (s *Store) Walk(ctx context.Context, fn func(blob.Info) error) error
Constants ¶
const ( // DefaultPartSize matches the SDK's own chunk size and its parallel // upload's part size: large enough that a 10 GiB blob costs about 660 // Class A operations, small enough to hold in a session row. DefaultPartSize = 16 << 20 )
Variables ¶
var ErrConfig = errors.New("blob/gcs: invalid configuration")
ErrConfig reports an unusable store configuration.
var ErrPrecondition = errors.New("blob/gcs: precondition failed")
ErrPrecondition reports a write or compose onto a name that exists.
var ErrSeek = errors.New("blob/gcs: invalid seek")
ErrSeek reports an invalid Seek.
Functions ¶
This section is empty.
Types ¶
type API ¶
type API interface {
// Write stores data as a new object, checked against crc on the server.
// It returns ErrPrecondition when the name already exists.
Write(ctx context.Context, name string, data []byte, crc uint32) (Attrs, error)
// Attrs returns an object's attributes, or blob.ErrNotFound.
Attrs(ctx context.Context, name string) (Attrs, error)
// Read opens the object from offset to the end, or blob.ErrNotFound.
Read(ctx context.Context, name string, offset int64) (io.ReadCloser, error)
// Compose concatenates srcs into a new object dst, optionally deleting
// the sources. It returns ErrPrecondition when dst already exists.
Compose(ctx context.Context, dst string, srcs []Source, deleteSources bool) (Attrs, error)
// Delete removes an object, only at generation when it is non-zero. It
// returns blob.ErrNotFound when the object is missing.
Delete(ctx context.Context, name string, generation int64) error
// List calls fn for every object whose name starts with prefix.
List(ctx context.Context, prefix string, fn func(Attrs) error) error
// SignedURL returns a V4 signed GET URL for name.
SignedURL(name string, o SignedURLOptions) (string, error)
}
API is the subset of Cloud Storage the store uses, bound to one bucket, so tests can substitute a fake that models GCS and injects failures. NewAPI returns the implementation over the official SDK.
type ClientConfig ¶
type ClientConfig struct {
// Endpoint overrides the Cloud Storage JSON API endpoint, for emulators,
// e.g. "http://localhost:4443". "/storage/v1/" is appended when missing.
Endpoint string
// CredentialsFile is a service-account JSON key. Empty means Application
// Default Credentials, which covers GKE Workload Identity.
CredentialsFile string
// GoogleAccessID and PrivateKey (PEM) sign URLs locally. Without them
// signing finds the service account from the credentials and calls IAM
// signBlob, which needs iam.serviceAccounts.signBlob.
GoogleAccessID string
PrivateKey []byte
// WithoutAuthentication sends no credentials, for emulators.
WithoutAuthentication bool
}
ClientConfig mirrors the blob.gcs section of the application configuration.
func (ClientConfig) CanSign ¶
func (cfg ClientConfig) CanSign() bool
CanSign reports whether a client built from cfg can sign URLs, decided from the configuration and credential files without a request, so a store knows its routes when it is built (decision 0029). The SDK signs locally with a key, or through IAM signBlob as a service account it can name: from GoogleAccessID, a service-account key, impersonated or federated credentials that name one, or the metadata server on Google Cloud. User credentials from "gcloud auth application-default login" name no account and cannot sign without GoogleAccessID.
func (ClientConfig) SignsOverNetwork ¶
func (cfg ClientConfig) SignsOverNetwork() bool
SignsOverNetwork reports whether every URL a client built from cfg signs costs an IAM signBlob request, so a store should cache what it signs (decision 0029).
type Options ¶
type Options struct {
// Prefix is prepended to every object name; a trailing slash is added
// if missing.
Prefix string
// PartSize is the part size; zero means DefaultPartSize. It must be a
// multiple of 256 KiB and at least 1 MiB.
PartSize int64
// Sessions holds upload sessions between requests; session.InUploads
// with a backend name such as "gcs:<store name>" keeps them durable.
Sessions session.Store
// Presign makes the store sign storage URLs (blob.Signer). Set it from
// ClientConfig.CanSign: signing needs a key, or credentials that name a
// service account and may call IAM signBlob.
Presign bool
// SignsOverNetwork marks every signature as an IAM signBlob request, so
// the store caches the URLs it signs (decision 0029). Set it from
// ClientConfig.SignsOverNetwork.
SignsOverNetwork bool
// CDN signs CDN URLs when set, such as a cdn.CloudCDN in front of the
// bucket (decision 0029). It must reach every key under Prefix.
CDN cdn.Signer
}
Options configures New.
type SignedURLOptions ¶
SignedURLOptions describe a signed GET URL: when it expires and the response overrides it carries, empty when not wanted.
type Source ¶
Source names one compose source at a pinned generation, so a source replaced since it was recorded fails the compose instead of being used.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a blob.Store on a GCS bucket.
func (*Store) BeginUpload ¶
BeginUpload starts a session.
func (*Store) ObjectKey ¶
ObjectKey is the object name a blob is stored under, prefix included, for signers that address the bucket directly, such as a CDN. sha must be a valid digest (blob.ValidateSHA256).
func (*Store) ResumeUpload ¶
ResumeUpload reopens a session.
func (*Store) SignURL ¶
SignURL returns a time-limited GET URL, carrying o's filename and media type as response overrides. A store that signs over the network issues URLs on windows and reuses one per window (see sign).
func (*Store) Signs ¶
Signs reports whether the store signs URLs for via: storage URLs when Options.Presign was set, CDN URLs when it has a CDN signer.