store

package
v4.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 9, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// local file store type
	LocalFileStore = "local_file"
	// agent-managed store type (cache v2): the agent manages the blob store
	// itself via BUILDKITE_AGENT_CACHE_STORE_URL
	AgentManaged = "agent_managed"
)

Variables

View Source 
var ErrBlobNotFound = errors.New("blob not found")

ErrBlobNotFound is returned by a Blob's Download when the requested object does not exist in the backing store.

Functions

func IsValidStore 

func IsValidStore(storeType string) bool

Types

type Blob 

type Blob interface {
	// Upload uploads a file to blob storage. retention is how long the blob
	// should be kept.
	Upload(ctx context.Context, filePath, key string, retention time.Duration) (*TransferInfo, error)

	// Download downloads a file from blob storage
	Download(ctx context.Context, key, destPath string) (*TransferInfo, error)
}

Blob interface defines the operations for blob storage

func NewBlobStore 

func NewBlobStore(ctx context.Context, store, bucketURL string) (Blob, error)

type CommandResult 

type CommandResult struct {
	Stdout   string
	Stderr   string
	ExitCode int
}

type FileMetadata 

type FileMetadata struct {
	Key       string `json:"key"`              // Original cache key
	Size      int64  `json:"size"`             // File size in bytes
	ModTime   string `json:"mod_time"`         // Original modification time (RFC3339Nano)
	Mode      string `json:"mode"`             // Original file permissions (octal)
	SHA256    string `json:"sha256,omitempty"` // SHA256 checksum for integrity verification
	CreatedAt string `json:"created_at"`       // Timestamp when cached (RFC3339Nano)
	Version   int    `json:"version"`          // Metadata schema version
}

FileMetadata contains metadata for cached files. Persisted as compact JSON in a sidecar file alongside each data file.

type LocalFileBlob 

type LocalFileBlob struct {
	// contains filtered or unexported fields
}

LocalFileBlob implements the Blob interface for local filesystem storage. Cache artifacts are stored as files with accompanying JSON metadata sidecars. Cache keys map directly to file paths under the configured root directory.

Storage layout:

  • Data files: <root>/<key>
  • Metadata files: <root>/<key>.attrs.json

Features:

  • Atomic writes using temp files + rename
  • Path traversal protection via multi-layer validation
  • SHA256 integrity checksums computed during upload
  • Last-writer-wins semantics for concurrent updates

func NewLocalFileBlob 

func NewLocalFileBlob(ctx context.Context, fileURL string) (*LocalFileBlob, error)

NewLocalFileBlob creates a new local file storage backend from a file:// URL.

Supported URL formats:

  • file:///absolute/path/to/cache
  • file://~/cache (expands to user's home directory)
  • file://~/.buildkitecache

The root directory will be created if it doesn't exist.

Returns an error if:

  • URL scheme is not "file"
  • Path is empty or invalid (e.g., "/", ".")
  • Directory creation fails

func (*LocalFileBlob) Download 

func (b *LocalFileBlob) Download(ctx context.Context, key, destPath string) (*TransferInfo, error)

Download retrieves a cached file identified by key and writes it to destPath.

The download process:

  1. Validates the cache key and destination path
  2. Reads the cached data file from storage
  3. Writes atomically to destination using temp file + fsync + rename
  4. Restores original file metadata (mtime) from sidecar if available (best-effort)
  5. Syncs parent directory for durability (best-effort)

Metadata restoration is best-effort; failures are logged but don't fail the download. Atomic writes ensure partial files are never visible at the destination path.

Returns TransferInfo with bytes transferred, transfer speed, and duration. Returns an error if the cache key doesn't exist or file operations fail.

func (*LocalFileBlob) Upload 

func (b *LocalFileBlob) Upload(ctx context.Context, srcPath, key string, _ time.Duration) (*TransferInfo, error)

Upload copies a file from srcPath to the cache storage identified by key.

The upload process:

  1. Validates the source path and cache key
  2. Computes SHA256 hash during copy for integrity verification
  3. Writes data atomically using temp file + fsync + rename
  4. Writes metadata (size, permissions, checksum, timestamps) atomically to sidecar file
  5. Syncs parent directory for durability (best-effort)

Atomic writes ensure readers never see partial data. On Windows, existing files are removed before rename due to platform limitations, creating last-writer-wins semantics for concurrent uploads to the same key.

Returns TransferInfo with bytes transferred, transfer speed, and duration.

type NscStore 

type NscStore struct {
	// contains filtered or unexported fields
}

NscStore implements the Blob interface for NSC artifact storage which uses the nsc CLI tool https://namespace.so/docs/reference/cli/artifact-download https://namespace.so/docs/reference/cli/artifact-upload

func NewNscStore 

func NewNscStore(bucketURL string) (*NscStore, error)

NewNscStore creates a store backed by the nsc CLI. The namespace is parsed from the nsc://<namespace> cache store URL and is required.

func (*NscStore) Download 

func (n *NscStore) Download(ctx context.Context, key, filePath string) (*TransferInfo, error)

func (*NscStore) RefreshRetention added in v4.0.2 

func (n *NscStore) RefreshRetention(ctx context.Context, key string, retention time.Duration)

RefreshRetention pushes the artifact's expiry out to at least retention from now (falling back to nscDefaultRetention when unset) via `nsc artifact extend --ensure_minimum`. Using --ensure_minimum makes the refresh idempotent, so calling it on every restore keeps a hot cache alive without growing its expiry unbounded.

This is best-effort: any failure is logged and swallowed so a restore never fails because its TTL could not be refreshed. Whether to call this at all (e.g. skipping it on a fallback match) is the restore flow's decision, not this store's — see store.RetentionRefresher.

func (*NscStore) Upload 

func (n *NscStore) Upload(ctx context.Context, filePath, key string, retention time.Duration) (*TransferInfo, error)

type Options 

type Options struct {
	S3Endpoint   string
	Bucket       string
	Region       string
	Prefix       string
	UsePathStyle bool
	Concurrency  int
	PartSizeMB   int
}

Options holds configuration for S3Blob and can be constructed from an S3 URL in a similar way to gocloud.dev Example S3 URLs: 

s3://my-bucket
s3://my-bucket/prefix
s3://my-bucket?region=us-east-1
s3://my-bucket/prefix?region=us-east-1&endpoint=http://localhost:9000&use_path_style=true

func OptionsFromURL 

func OptionsFromURL(s3url string) (*Options, error)

type RetentionRefresher added in v4.0.2 

type RetentionRefresher interface {
	RefreshRetention(ctx context.Context, key string, retention time.Duration)
}

RetentionRefresher is implemented by Blob stores that support extending a blob's effective retention/TTL on access (NscStore, S3Blob). Optional because not every store has a retention concept to refresh (LocalFileBlob does not implement it).

type S3Blob 

type S3Blob struct {
	// contains filtered or unexported fields
}

S3Blob implements the Blob interface using AWS S3

func NewS3Blob 

func NewS3Blob(ctx context.Context, s3url string) (*S3Blob, error)

NewS3Blob creates a new S3Blob instance using an S3 URL and prefix

func (*S3Blob) Download 

func (b *S3Blob) Download(ctx context.Context, key, destPath string) (*TransferInfo, error)

Download downloads a file from S3 using parallel range requests for large files

func (*S3Blob) RefreshRetention added in v4.0.2 

func (b *S3Blob) RefreshRetention(ctx context.Context, key string, _ time.Duration)

RefreshRetention extends an object's effective TTL by performing CopyObject on itself to refresh its LastModified timestamp — S3 has no native touch/extend-TTL API, and lifecycle expiration rules key off LastModified.

Whether to call this at all (e.g. skipping it on a fallback match) is the restore flow's decision, not this store's — see store.RetentionRefresher.

The CopySourceIfUnmodifiedSince precondition aborts the operation with an HTTP status code 412 if the object's LastModified timestamp falls within restoreRefreshMinInterval.

This refresh is best-effort only: any failure (incl. objects exceeding S3's 5GB CopyObject limit) must not cause the overall restore operation to fail.

retention is ignored: the self-copy only refreshes LastModified, and the effective lifetime is whatever window the bucket's lifecycle policy applies from that timestamp — the agent cannot lengthen it per object.

func (*S3Blob) Upload 

func (b *S3Blob) Upload(ctx context.Context, filePath, key string, _ time.Duration) (*TransferInfo, error)

Upload uploads a file to S3 using multipart upload for parallel transfers. retention is ignored: S3 object lifetime is governed by the bucket's lifecycle policy, which the agent does not set per object.

type TransferInfo 

type TransferInfo struct {
	BytesTransferred int64
	TransferSpeed    float64 // in MB/s
	RequestID        string
	Duration         time.Duration
	PartCount        int // number of parts used in multipart transfer (0 if not multipart)
	Concurrency      int // number of concurrent uploads/downloads used
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.