Help us shape the future of Go! Take the 2026 Go Developer Survey

gcs

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package gcs is the Google Cloud Storage blob store (decision 0027). Blobs live at <prefix>sha256/<ab>/<hex>.

GCS objects cannot be appended to, and the Go SDK cannot resume a resumable-upload session from persisted state, so an upload session is built from compose instead:

  • Bytes accumulate in a pending buffer of at most PartSize. A blob that never fills one part is written in a single request at commit.
  • Each full part becomes its own object under <prefix>_uploads/<id>/, written with a must-not-exist precondition. Part names are never reused across parts, so a retry of the same part either lands or finds its own earlier copy.
  • GCS composes at most 32 sources per request, so before a 33rd source the session composes what it has into a fold object and continues from that. There is no limit on how many components a composite object has.
  • At commit the content key is known; the session composes everything straight onto it with a must-not-exist precondition and DeleteSourceObjects, so the bytes are never copied. A 412 there means the blob is already stored, which the contract treats as success.
  • Composite objects carry a CRC32C but no MD5, so the session tracks the CRC32C of everything appended and checks the result against it; the blob.Hasher state still provides SHA-256, SHA-1 and MD5.

The session (part objects, their generations, sizes and checksums) is persisted through pkg/blob/session after every append.

Index

Constants

View Source
const (
	// DefaultPartSize matches the SDK's own chunk size and its parallel
	// upload's part size: large enough that a 10 GiB blob costs about 660
	// Class A operations, small enough to hold in a session row.
	DefaultPartSize = 16 << 20
)

Variables

View Source
var ErrConfig = errors.New("blob/gcs: invalid configuration")

ErrConfig reports an unusable store configuration.

View Source
var ErrPrecondition = errors.New("blob/gcs: precondition failed")

ErrPrecondition reports a write or compose onto a name that exists.

View Source
var ErrSeek = errors.New("blob/gcs: invalid seek")

ErrSeek reports an invalid Seek.

Functions

This section is empty.

Types

type API

type API interface {
	// Write stores data as a new object, checked against crc on the server.
	// It returns ErrPrecondition when the name already exists.
	Write(ctx context.Context, name string, data []byte, crc uint32) (Attrs, error)
	// Attrs returns an object's attributes, or blob.ErrNotFound.
	Attrs(ctx context.Context, name string) (Attrs, error)
	// Read opens the object from offset to the end, or blob.ErrNotFound.
	Read(ctx context.Context, name string, offset int64) (io.ReadCloser, error)
	// Compose concatenates srcs into a new object dst, optionally deleting
	// the sources. It returns ErrPrecondition when dst already exists.
	Compose(ctx context.Context, dst string, srcs []Source, deleteSources bool) (Attrs, error)
	// Delete removes an object, only at generation when it is non-zero. It
	// returns blob.ErrNotFound when the object is missing.
	Delete(ctx context.Context, name string, generation int64) error
	// List calls fn for every object whose name starts with prefix.
	List(ctx context.Context, prefix string, fn func(Attrs) error) error
	// SignedURL returns a V4 signed GET URL for name.
	SignedURL(name string, o SignedURLOptions) (string, error)
}

API is the subset of Cloud Storage the store uses, bound to one bucket, so tests can substitute a fake that models GCS and injects failures. NewAPI returns the implementation over the official SDK.

func NewAPI

func NewAPI(ctx context.Context, bucket string, cfg ClientConfig) (API, error)

NewAPI builds the API over the official Cloud Storage client for bucket, which must exist. It uses the JSON API for reads as well as writes.

type Attrs

type Attrs struct {
	Name       string
	Size       int64
	CRC32C     uint32
	Generation int64
}

Attrs are the object attributes the store uses.

type ClientConfig

type ClientConfig struct {
	// Endpoint overrides the Cloud Storage JSON API endpoint, for emulators,
	// e.g. "http://localhost:4443". "/storage/v1/" is appended when missing.
	Endpoint string
	// CredentialsFile is a service-account JSON key. Empty means Application
	// Default Credentials, which covers GKE Workload Identity.
	CredentialsFile string
	// GoogleAccessID and PrivateKey (PEM) sign URLs locally. Without them
	// signing finds the service account from the credentials and calls IAM
	// signBlob, which needs iam.serviceAccounts.signBlob.
	GoogleAccessID string
	PrivateKey     []byte
	// WithoutAuthentication sends no credentials, for emulators.
	WithoutAuthentication bool
}

ClientConfig mirrors the blob.gcs section of the application configuration.

func (ClientConfig) CanSign

func (cfg ClientConfig) CanSign() bool

CanSign reports whether a client built from cfg can sign URLs, decided from the configuration and credential files without a request, so a store knows its routes when it is built (decision 0029). The SDK signs locally with a key, or through IAM signBlob as a service account it can name: from GoogleAccessID, a service-account key, impersonated or federated credentials that name one, or the metadata server on Google Cloud. User credentials from "gcloud auth application-default login" name no account and cannot sign without GoogleAccessID.

func (ClientConfig) SignsOverNetwork

func (cfg ClientConfig) SignsOverNetwork() bool

SignsOverNetwork reports whether every URL a client built from cfg signs costs an IAM signBlob request, so a store should cache what it signs (decision 0029).

type Options

type Options struct {
	// Prefix is prepended to every object name; a trailing slash is added
	// if missing.
	Prefix string
	// PartSize is the part size; zero means DefaultPartSize. It must be a
	// multiple of 256 KiB and at least 1 MiB.
	PartSize int64
	// Sessions holds upload sessions between requests; session.InUploads
	// with a backend name such as "gcs:<store name>" keeps them durable.
	Sessions session.Store
	// Presign makes the store sign storage URLs (blob.Signer). Set it from
	// ClientConfig.CanSign: signing needs a key, or credentials that name a
	// service account and may call IAM signBlob.
	Presign bool
	// SignsOverNetwork marks every signature as an IAM signBlob request, so
	// the store caches the URLs it signs (decision 0029). Set it from
	// ClientConfig.SignsOverNetwork.
	SignsOverNetwork bool
	// CDN signs CDN URLs when set, such as a cdn.CloudCDN in front of the
	// bucket (decision 0029). It must reach every key under Prefix.
	CDN cdn.Signer
}

Options configures New.

type SignedURLOptions

type SignedURLOptions struct {
	Expires            time.Time
	ContentDisposition string
	ContentType        string
}

SignedURLOptions describe a signed GET URL: when it expires and the response overrides it carries, empty when not wanted.

type Source

type Source struct {
	Name       string
	Generation int64
}

Source names one compose source at a pinned generation, so a source replaced since it was recorded fails the compose instead of being used.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a blob.Store on a GCS bucket.

func New

func New(api API, o Options) (*Store, error)

New returns a store over api.

func (*Store) BeginUpload

func (s *Store) BeginUpload(ctx context.Context, opts ...blob.UploadOption) (blob.Upload, error)

BeginUpload starts a session.

func (*Store) Delete

func (s *Store) Delete(ctx context.Context, sha string) error

Delete removes the blob; a missing blob is already gone.

func (*Store) Exists

func (s *Store) Exists(ctx context.Context, sha string) (bool, error)

Exists reports whether the blob is stored.

func (*Store) ObjectKey

func (s *Store) ObjectKey(sha string) string

ObjectKey is the object name a blob is stored under, prefix included, for signers that address the bucket directly, such as a CDN. sha must be a valid digest (blob.ValidateSHA256).

func (*Store) Open

func (s *Store) Open(ctx context.Context, sha string) (io.ReadSeekCloser, blob.Info, error)

Open returns a reader that fetches ranges on demand.

func (*Store) PartSize

func (s *Store) PartSize() int64

PartSize returns the size of the parts an upload is written in.

func (*Store) ResumeUpload

func (s *Store) ResumeUpload(ctx context.Context, id string) (blob.Upload, error)

ResumeUpload reopens a session.

func (*Store) SignURL

func (s *Store) SignURL(_ context.Context, sha string, o blob.SignOptions) (string, error)

SignURL returns a time-limited GET URL, carrying o's filename and media type as response overrides. A store that signs over the network issues URLs on windows and reuses one per window (see sign).

func (*Store) Signs

func (s *Store) Signs(via blob.Via) bool

Signs reports whether the store signs URLs for via: storage URLs when Options.Presign was set, CDN URLs when it has a CDN signer.

func (*Store) Stat

func (s *Store) Stat(ctx context.Context, sha string) (blob.Info, error)

Stat returns a blob's size and digest.

func (*Store) Walk

func (s *Store) Walk(ctx context.Context, fn func(blob.Info) error) error

Walk lists every blob under the prefix; upload sessions live elsewhere.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL