Documentation
¶
Index ¶
- Variables
- type Cache
- type CapacitySnapshot
- type EvictionPolicy
- type Filesystem
- func (f *Filesystem) AssemblePartsStaged(_ context.Context, bucket, key, uploadID string, partNumbers []int) (*StagedObject, []string, error)
- func (f *Filesystem) CapacitySnapshot() CapacitySnapshot
- func (f *Filesystem) ConsumeRefusedWriteBytes() int64
- func (f *Filesystem) CreateBucketDir(_ context.Context, bucket string) error
- func (f *Filesystem) Delete(_ context.Context, bucket, key string) error
- func (f *Filesystem) DeleteBucketDir(_ context.Context, bucket string) error
- func (f *Filesystem) DeleteQuarantined(ctx context.Context, bucket string, contentID int64) error
- func (f *Filesystem) DeleteUpload(ctx context.Context, uploadID string) error
- func (f *Filesystem) Exists(_ context.Context, bucket, key string) bool
- func (f *Filesystem) Get(_ context.Context, bucket, key string) (io.ReadCloser, *ObjectInfo, error)
- func (f *Filesystem) ListUploadIDs(ctx context.Context) ([]string, error)
- func (f *Filesystem) Put(ctx context.Context, bucket, key string, r io.Reader, size int64) (*ObjectInfo, error)
- func (f *Filesystem) PutPartStaged(ctx context.Context, uploadID string, partNumber int, r io.Reader, size int64) (*StagedObject, error)
- func (f *Filesystem) PutStaged(ctx context.Context, bucket, key string, r io.Reader, size int64) (*StagedObject, error)
- func (f *Filesystem) Quarantine(ctx context.Context, bucket string, contentID int64) error
- func (f *Filesystem) RootDir() string
- func (f *Filesystem) UsedBytes() int64
- func (f *Filesystem) WalkContentFiles(ctx context.Context, quarantined bool, visit func(string, int64) error) error
- type ObjectInfo
- type StagedObject
Constants ¶
This section is empty.
Variables ¶
var ErrCacheFull = errors.New("cache: storage capacity exceeded")
ErrCacheFull is returned when the cache cannot hold a write's size on top of committed and in-progress writes.
var ErrInvalidPath = errors.New("cache: invalid path (traversal attempt)")
ErrInvalidPath is returned when a bucket or key would escape the cache root.
var ErrSizeExceeded = errors.New("cache: write exceeds its declared size")
ErrSizeExceeded is returned when a write supplies more bytes than the size it reserved.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache interface {
// Put writes data to the cache under the given bucket/key and returns
// the resulting metadata (path, size, etag, checksum).
// The data is fsync'd before returning to guarantee durability.
// size is the most bytes r may supply; the write holds that much capacity
// before reading r and returns ErrCacheFull without reading when the cache
// cannot hold it, or ErrSizeExceeded when r supplies more.
// Returns ErrInvalidPath if bucket/key would escape the cache root.
Put(ctx context.Context, bucket, key string, r io.Reader, size int64) (*ObjectInfo, error)
// PutStaged writes data to a temp file (fsync'd) without replacing the
// existing cache entry. Returns a StagedObject whose Commit method
// atomically renames the temp file to the final path. If Commit is not
// called, Rollback removes the temp file. The capacity held for size, as
// in Put, lasts until Commit or Rollback.
// Callers choose whether to commit before or after their DB transaction.
PutStaged(ctx context.Context, bucket, key string, r io.Reader, size int64) (*StagedObject, error)
// Get opens a cached object for reading. Returns os.ErrNotExist if the
// object is not in the cache.
// Returns ErrInvalidPath if bucket/key would escape the cache root.
Get(ctx context.Context, bucket, key string) (io.ReadCloser, *ObjectInfo, error)
// Delete removes an object from the cache.
// Returns ErrInvalidPath if bucket/key would escape the cache root.
Delete(ctx context.Context, bucket, key string) error
// Exists reports whether an object is present in the cache.
Exists(ctx context.Context, bucket, key string) bool
// UsedBytes returns the total bytes consumed by committed cached objects.
UsedBytes() int64
CapacitySnapshot() CapacitySnapshot
// ConsumeRefusedWriteBytes returns the largest capacity reservation refused
// since the previous call and clears the record. The capacity task owns it.
ConsumeRefusedWriteBytes() int64
// CreateBucketDir ensures the directory for a bucket exists.
// Returns ErrInvalidPath if bucket would escape the cache root.
CreateBucketDir(ctx context.Context, bucket string) error
// DeleteBucketDir removes the cache directory for a bucket.
// Returns ErrInvalidPath if bucket would escape the cache root.
DeleteBucketDir(ctx context.Context, bucket string) error
// PutPartStaged reserves and writes a part beside its final upload path.
// Commit publishes the part; Rollback releases an unpublished reservation.
PutPartStaged(ctx context.Context, uploadID string, partNumber int, r io.Reader, size int64) (*StagedObject, error)
// AssemblePartsStaged concatenates the specified parts in order into a
// staged file next to bucket/key, plus the ordered list of individual part
// MD5 hex digests (for S3 ETag computation). The caller names the final
// destination through Commit or CommitAs once the assembled checksum has
// resolved a content row. The part files are NOT deleted; call DeleteUpload.
// Capacity for the assembled size is held before assembly starts.
AssemblePartsStaged(ctx context.Context, bucket, key, uploadID string, partNumbers []int) (*StagedObject, []string, error)
// DeleteUpload removes all part files for the given upload ID.
DeleteUpload(ctx context.Context, uploadID string) error
}
Cache defines the interface for local object caching. Implementations must be safe for concurrent use.
type CapacitySnapshot ¶
CapacitySnapshot uses the same committed and reserved bytes as write admission.
func (CapacitySnapshot) OccupiedBytes ¶
func (s CapacitySnapshot) OccupiedBytes() int64
type EvictionPolicy ¶
type EvictionPolicy string
EvictionPolicy controls when remotely durable objects are removed from the local cache.
const ( EvictionPolicyLRU EvictionPolicy = "lru" EvictionPolicyAfterUpload EvictionPolicy = "after_upload" EvictionPolicyNone EvictionPolicy = "none" )
func ParseEvictionPolicy ¶
func ParseEvictionPolicy(value string) (EvictionPolicy, bool)
ParseEvictionPolicy returns the canonical eviction policy.
func (EvictionPolicy) EnqueuesAfterUploadEviction ¶
func (p EvictionPolicy) EnqueuesAfterUploadEviction() bool
EnqueuesAfterUploadEviction reports whether upload finalization must create an eviction task in the same transaction.
func (EvictionPolicy) RehydratesColdReads ¶ added in v0.1.1
func (p EvictionPolicy) RehydratesColdReads() bool
RehydratesColdReads reports whether a complete read from a storage provider restores the object to the local cache. Under after_upload nothing would release such a copy again.
type Filesystem ¶
type Filesystem struct {
// contains filtered or unexported fields
}
Filesystem implements the Cache interface using the local filesystem.
func NewFilesystem ¶
func NewFilesystem(dir string, maxBytes int64) (*Filesystem, error)
NewFilesystem creates a filesystem-backed cache rooted at dir. maxBytes sets the capacity limit (0 = unlimited). On startup it walks existing files to initialise usedBytes and removes stale temp files.
func (*Filesystem) AssemblePartsStaged ¶
func (f *Filesystem) AssemblePartsStaged(_ context.Context, bucket, key, uploadID string, partNumbers []int) (*StagedObject, []string, error)
func (*Filesystem) CapacitySnapshot ¶
func (f *Filesystem) CapacitySnapshot() CapacitySnapshot
func (*Filesystem) ConsumeRefusedWriteBytes ¶
func (f *Filesystem) ConsumeRefusedWriteBytes() int64
func (*Filesystem) CreateBucketDir ¶
func (f *Filesystem) CreateBucketDir(_ context.Context, bucket string) error
func (*Filesystem) Delete ¶
func (f *Filesystem) Delete(_ context.Context, bucket, key string) error
func (*Filesystem) DeleteBucketDir ¶
func (f *Filesystem) DeleteBucketDir(_ context.Context, bucket string) error
func (*Filesystem) DeleteQuarantined ¶ added in v0.1.1
DeleteQuarantined deletes a retained file and releases its capacity charge.
func (*Filesystem) DeleteUpload ¶
func (f *Filesystem) DeleteUpload(ctx context.Context, uploadID string) error
func (*Filesystem) Get ¶
func (f *Filesystem) Get(_ context.Context, bucket, key string) (io.ReadCloser, *ObjectInfo, error)
func (*Filesystem) ListUploadIDs ¶ added in v0.1.1
func (f *Filesystem) ListUploadIDs(ctx context.Context) ([]string, error)
ListUploadIDs lists multipart directories retained on disk.
func (*Filesystem) Put ¶
func (f *Filesystem) Put(ctx context.Context, bucket, key string, r io.Reader, size int64) (*ObjectInfo, error)
func (*Filesystem) PutPartStaged ¶ added in v0.1.1
func (f *Filesystem) PutPartStaged(ctx context.Context, uploadID string, partNumber int, r io.Reader, size int64) (*StagedObject, error)
func (*Filesystem) PutStaged ¶
func (f *Filesystem) PutStaged(ctx context.Context, bucket, key string, r io.Reader, size int64) (*StagedObject, error)
func (*Filesystem) Quarantine ¶ added in v0.1.1
Quarantine moves a content file out of the readable path while retaining its capacity charge.
func (*Filesystem) RootDir ¶
func (f *Filesystem) RootDir() string
RootDir returns the absolute path of the cache root directory.
func (*Filesystem) UsedBytes ¶
func (f *Filesystem) UsedBytes() int64
func (*Filesystem) WalkContentFiles ¶ added in v0.1.1
func (f *Filesystem) WalkContentFiles(ctx context.Context, quarantined bool, visit func(string, int64) error) error
WalkContentFiles visits content-addressed files in bucket cache directories.
type ObjectInfo ¶
ObjectInfo holds metadata about a cached object.
type StagedObject ¶
type StagedObject struct {
Info *ObjectInfo
// contains filtered or unexported fields
}
StagedObject represents a cached file that has been written to a temp location but not yet committed to the final cache path. This enables atomic "write then commit" workflows where a DB transaction runs between write and commit.
func (*StagedObject) Commit ¶
func (s *StagedObject) Commit() error
Commit renames the staged file to the final cache path (atomic). Must be called exactly once. After Commit, Rollback is a no-op.
func (*StagedObject) CommitAs ¶
func (s *StagedObject) CommitAs(bucket, key string) error
CommitAs commits to a different final path than the one the object was staged under. Cache residency is content-addressed, so writers stage the bytes first and only learn the destination once the checksum has produced a content row.
func (*StagedObject) Rollback ¶
func (s *StagedObject) Rollback() error
Rollback removes the staged temp file without affecting the final cache path. Safe to call multiple times and after Commit (no-op if already committed).