storage

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 13, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package storage is the Cloud Storage semantics layer: buckets, objects, generations and preconditions.

It speaks no HTTP and no JSON envelopes — handlers translate. Keeping the rules here means the awkward parts (generation allocation, precondition atomicity, delimiter rollup) are testable without a server.

Index

Constants

View Source 
const (
	EventFinalized = "google.storage.object.finalize"
	EventDeleted   = "google.storage.object.delete"
	EventArchived  = "google.storage.object.archive"
	EventUpdated   = "google.storage.object.metadataUpdate"
)

The event types GCS emits to first-generation functions.

These are the gen1 names, not the CloudEvents ones (google.cloud.storage.object.v1.finalized). We serve the v1 API and report GEN_1, so a function receives what a gen1 function receives — the names and the envelope have to agree, or a handler reads undefined.

View Source 
const (
	DefaultLocation     = "US"
	DefaultStorageClass = "STANDARD"
)

Defaults applied when a caller does not say, matching what GCS assumes.

View Source 
const DefaultMaxResults = 1000

DefaultMaxResults matches what GCS returns when a client does not ask.

View Source 
const ServiceName = "storage.googleapis.com"

ServiceName is what gen1 reports in an event's resource.service.

Variables

View Source 
var Prefixes = []string{"/storage/v1/", "/upload/storage/v1/", "/download/storage/v1/"}

Prefixes are the path prefixes the JSON API claims.

Functions

func SourceOf 

func SourceOf(bucket string) string

SourceOf identifies a bucket as the source of an event.

Types

type API 

type API struct {
	// contains filtered or unexported fields
}

API is the Cloud Storage JSON API over a Service.

func NewAPI 

func NewAPI(svc *Service) *API

NewAPI wires the handlers.

func (*API) Close 

func (a *API) Close() error

Close removes any in-progress upload staging.

func (*API) ServeHTTP 

func (a *API) ServeHTTP(w http.ResponseWriter, r *http.Request)

type Binding 

type Binding struct {
	Role    string   `json:"role"`
	Members []string `json:"members"`
}

Binding grants a role to members.

type Bucket 

type Bucket struct {
	Name         string `json:"name"`
	Project      string `json:"project"`
	Location     string `json:"location"`
	StorageClass string `json:"storageClass"`
	// Versioning off is the GCS default: an overwrite discards the previous
	// generation and a delete removes it. On, both are kept.
	Versioning     bool      `json:"versioning"`
	Metageneration int64     `json:"metageneration"`
	Created        time.Time `json:"timeCreated"`
	Updated        time.Time `json:"updated"`
}

Bucket is a bucket's metadata.

type BucketPatch 

type BucketPatch struct {
	StorageClass  string
	Versioning    *bool
	Preconditions Preconditions
}

BucketPatch is a partial update. A nil field is left unchanged.

type ListRequest 

type ListRequest struct {
	Bucket     string
	Prefix     string
	Delimiter  string
	MaxResults int
	PageToken  string
}

ListRequest describes an object listing.

type ListResult 

type ListResult struct {
	Objects []Object
	// Prefixes are the delimiter rollups: the synthetic "directories" GCS
	// reports in prefixes[].
	Prefixes      []string
	NextPageToken string
}

ListResult is one page.

type Object 

type Object struct {
	Bucket         string            `json:"bucket"`
	Name           string            `json:"name"`
	Generation     int64             `json:"generation"`
	Metageneration int64             `json:"metageneration"`
	Size           int64             `json:"size"`
	ContentType    string            `json:"contentType"`
	Metadata       map[string]string `json:"metadata,omitempty"`
	CRC32C         string            `json:"crc32c"`
	MD5            string            `json:"md5Hash"`
	ETag           string            `json:"etag"`
	Blob           string            `json:"blob"`
	Created        time.Time         `json:"timeCreated"`
	Updated        time.Time         `json:"updated"`
}

Object is one generation's metadata. Payload bytes live in the blob tree; Blob is their address.

type Policy 

type Policy struct {
	Kind       string    `json:"kind"`
	ResourceID string    `json:"resourceId"`
	Version    int       `json:"version"`
	Etag       string    `json:"etag"`
	Bindings   []Binding `json:"bindings"`
}

Policy is an IAM policy as the JSON API spells it.

type Preconditions 

type Preconditions struct {
	// IfGenerationMatch of 0 means the object must not exist — what
	// storage.Conditions{DoesNotExist: true} compiles to.
	IfGenerationMatch    *int64
	IfGenerationNotMatch *int64

	IfMetagenerationMatch    *int64
	IfMetagenerationNotMatch *int64
}

Preconditions gate an operation. A nil field is no condition.

type Service 

type Service struct {
	// contains filtered or unexported fields
}

Service holds the metadata store, the blob tree and the clock.

func New 

func New(kv store.Store, blobs *blob.Store, clk clock.Clock, bus *events.Bus) *Service

New wires a service. The blob tree holds payloads; kv holds only metadata. A nil bus publishes nothing.

func (*Service) ComposeObject 

func (s *Service) ComposeObject(ctx context.Context, project string, sources []Source, dst Write) (Object, error)

ComposeObject concatenates sources into one object.

The parts are streamed one after another into the blob tree, so composing objects far larger than memory costs the same fixed buffer as any other write.

func (*Service) CopyObject 

func (s *Service) CopyObject(ctx context.Context, project string, src Source, dst Write) (Object, error)

CopyObject writes an existing object's content under a new name.

No bytes move. Content is addressed by its hash, so the copy points at the same file and only the reference count changes — a copy of a gibibyte costs what a copy of a byte costs.

func (*Service) CreateBucket 

func (s *Service) CreateBucket(ctx context.Context, b Bucket) (Bucket, error)

CreateBucket stores a new bucket, failing if the name is taken.

func (*Service) DeleteBucket 

func (s *Service) DeleteBucket(ctx context.Context, project, name string) error

DeleteBucket removes an empty bucket.

func (*Service) DeleteObject 

func (s *Service) DeleteObject(ctx context.Context, project, bucket, name string, p Preconditions) error

DeleteObject removes the live generation.

func (*Service) GetBucket 

func (s *Service) GetBucket(ctx context.Context, project, name string) (Bucket, error)

GetBucket returns a bucket's metadata.

func (*Service) GetIAMPolicy 

func (s *Service) GetIAMPolicy(ctx context.Context, project, bucket, object string) (Policy, error)

GetIAMPolicy returns the stored policy, or an empty one.

func (*Service) GetObject 

func (s *Service) GetObject(ctx context.Context, project, bucket, name string, generation *int64) (Object, error)

GetObject returns metadata for the live generation, or a specific one.

func (*Service) GetObjectIf 

func (s *Service) GetObjectIf(ctx context.Context, project, bucket, name string, generation *int64, p Preconditions) (Object, error)

GetObjectIf is GetObject with conditions. A NotMatch condition that holds is reported as 304, as GCS does for a read.

func (*Service) ListBuckets 

func (s *Service) ListBuckets(ctx context.Context, project string) ([]Bucket, error)

ListBuckets returns a project's buckets, by name.

func (*Service) ListObjects 

func (s *Service) ListObjects(ctx context.Context, project string, req ListRequest) (ListResult, error)

ListObjects returns one page of a bucket's live objects.

Listing walks live pointers, not generation keys: one entry per visible object, rather than every version it has ever had.

func (*Service) OpenObject 

func (s *Service) OpenObject(ctx context.Context, project, bucket, name string, generation *int64, p Preconditions) (Object, *os.File, error)

OpenObject returns metadata and an open file of the content. The caller closes the file; handing back the file rather than bytes is what keeps a download's memory constant.

func (*Service) ProjectOf 

func (s *Service) ProjectOf(ctx context.Context, bucket string) (string, error)

ProjectOf resolves a bucket name to the project holding it, for the object endpoints, whose URLs name no project.

func (*Service) Reset 

func (s *Service) Reset(ctx context.Context, project string) error

Reset clears state. An empty project clears everything.

Blobs are only reclaimed on a full reset: they are content-addressed and shared, so knowing which are unreferenced after clearing one project would need refcounting that nothing else wants. A scoped reset leaves them, and a full one removes the tree.

func (*Service) SetIAMPolicy 

func (s *Service) SetIAMPolicy(ctx context.Context, project, bucket, object string, p Policy) (Policy, error)

SetIAMPolicy stores a policy and returns it as stored.

func (*Service) UpdateBucket 

func (s *Service) UpdateBucket(ctx context.Context, project, name string, patch BucketPatch) (Bucket, error)

UpdateBucket applies a patch and bumps metageneration.

Location is not patchable: GCS fixes it at creation, and silently accepting a change would be worse than refusing one.

func (*Service) UpdateObject 

func (s *Service) UpdateObject(ctx context.Context, project, bucket, name string, patch Write) (Object, error)

UpdateObject replaces an object's mutable metadata.

Metageneration moves, generation does not: the content is untouched, and a client watching for a content change must not see one.

func (*Service) WriteObject 

func (s *Service) WriteObject(ctx context.Context, project string, w Write, r io.Reader) (Object, error)

WriteObject streams r into the blob tree and publishes it as a new generation.

The payload is stored before the pointer is claimed, so a writer that loses the race leaves an unreferenced blob behind. That is deliberate: the alternative is holding the bytes until the outcome is known, which is exactly the memory behaviour this project exists to avoid. Reclaimed by Reset.

type Source 

type Source struct {
	Bucket     string
	Name       string
	Generation *int64
}

Source names an object to read from.

type Write 

type Write struct {
	Bucket        string
	Name          string
	ContentType   string
	Metadata      map[string]string
	Preconditions Preconditions
}

Write is a request to store an object.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.