Documentation
¶
Overview ¶
Package blob implements a content-addressed filesystem blob store.
Layout on disk:
{root}/{tenant}/{xx}/{sha256hex}
where {xx} is the first two hex characters of the SHA-256 digest (git-style prefix sharding). This keeps directory entry counts manageable at scale.
Blobs are written atomically: content is first written to a temp file in the same directory, then renamed into place. A partial write therefore never produces a visible corrupt blob.
Deduplication is structural: two clients storing identical content produce the same SHA-256 key, and the second write is a no-op after the existence check.
The key-alias index maps caller-supplied names to SHA-256 digests. Each alias is a small text file under {root}/{tenant}/.keys/{key} containing the hex SHA. This keeps the key lookup O(1) filesystem calls with no separate index database.
Index ¶
- Variables
- func ValidateKey(key string) error
- type GCConfig
- type GCReport
- type GCWorker
- type GlobalUsage
- type Meta
- type SHARefSource
- type SampledUsage
- type Store
- func (s *Store) Delete(key string) error
- func (s *Store) Get(key string) (io.ReadCloser, Meta, error)
- func (s *Store) GetBySHA(key, hexSHA string) (io.ReadCloser, Meta, error)
- func (s *Store) Head(key string) (Meta, error)
- func (s *Store) List(prefix string) ([]Meta, error)
- func (s *Store) Put(key string, r io.Reader, contentType string) (sha string, md5hex string, created bool, err error)
- func (s *Store) PutBySHA(key, hexSHA, contentType string) (string, bool, error)
- func (s *Store) PutRaw(r io.Reader, contentType string) (sha string, md5hex string, created bool, err error)
- func (s *Store) Root() string
- func (s *Store) Usage() (Usage, error)
- type Usage
- type UsageSampler
Constants ¶
This section is empty.
Variables ¶
var ErrKeyInvalid = errors.New("blob: invalid key")
ErrKeyInvalid is returned when a caller-supplied key contains disallowed characters or is otherwise malformed.
var ErrNotFound = errors.New("blob: not found")
ErrNotFound is returned when a blob or key does not exist.
var ErrSHAInvalid = errors.New("blob: invalid sha256 digest")
ErrSHAInvalid is returned when a SHA-addressed operation receives a digest that is not exactly 64 lowercase hexadecimal characters. Validating the digest before it reaches the filesystem path prevents a panic on a short digest (hexSHA[:2]) and stops non-hex characters from contributing path components to the on-disk layout (D-004).
var ErrTooLarge = errors.New("blob: content too large")
ErrTooLarge is returned when the content exceeds the configured size limit.
Functions ¶
func ValidateKey ¶
ValidateKey is the exported form of the key validation check, for use by callers that want to pre-validate keys before calling Put.
Types ¶
type GCConfig ¶
type GCConfig struct {
// Interval between GC sweeps. Default: 1 hour.
Interval time.Duration
// GracePeriod is how long a blob must sit in .gc-pending/ before it is
// hard-deleted. Default: 10 minutes.
GracePeriod time.Duration
}
GCConfig holds tunable parameters for the GC worker.
func DefaultGCConfig ¶
func DefaultGCConfig() GCConfig
DefaultGCConfig returns a GCConfig with production-safe defaults.
type GCReport ¶
type GCReport struct {
TenantsScanned int
Quarantined int // blobs moved to .gc-pending this cycle
Deleted int // blobs hard-deleted from .gc-pending this cycle
Errors int
}
GCReport summarises the result of a single GC sweep.
type GCWorker ¶
type GCWorker struct {
// contains filtered or unexported fields
}
GCWorker runs periodic mark-and-sweep GC against a Store. It is safe to create only one GCWorker per Store.
func NewGCWorker ¶
func NewGCWorker(s *Store, cfg GCConfig, extRefs SHARefSource) *GCWorker
NewGCWorker creates a GCWorker. extRefs may be nil when there are no external SHA reference sources.
func (*GCWorker) RunOnce ¶
RunOnce executes a single mark-and-sweep cycle synchronously. Useful for tests and for a one-shot manual trigger from an admin endpoint.
type GlobalUsage ¶
type GlobalUsage struct {
TotalBlobCount int64
TotalKeyCount int64
TotalBytes int64
TenantCount int64
SampledAt time.Time
}
GlobalUsage is an aggregate across stores, suitable for telemetry. It is composed by the blob manager from per-store SampledUsage values.
type Meta ¶
type Meta struct {
Key string // caller-assigned name
SHA256 string // hex-encoded SHA-256 of the content
MD5 string // hex-encoded MD5 of the content (S3 ETag); "" if unknown
Size int64 // bytes
ContentType string // preserved from the original store call; may be empty
StoredAt time.Time // when this key was last written
}
Meta holds metadata about a stored blob.
type SHARefSource ¶
SHARefSource is implemented by subsystems that hold SHA references outside the key-alias index (e.g. the timeseries history store). The GC calls CollectLiveSHAs once per sweep cycle and treats every returned SHA as live.
type SampledUsage ¶
type SampledUsage struct {
// BlobCount is the number of distinct blob files on disk (deduplicated).
BlobCount int64
// KeyCount is the number of key aliases.
KeyCount int64
// Bytes is the total size of all blob files in bytes, excluding sidecars.
Bytes int64
// SampledAt is when this entry was last refreshed.
// Zero means no walk has completed yet.
SampledAt time.Time
}
SampledUsage is the cached result of one store's usage walk.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a content-addressed blob store for a single xolu instance. It is safe for concurrent use.
func NewStore ¶
NewStore creates a Store rooted at dir. dir is created if it does not exist. maxSize is the upper bound on accepted blob sizes in bytes; 0 means no limit.
func (*Store) Delete ¶
Delete removes the key alias for key in tenant. The underlying blob file is not removed immediately — blob GC is a separate operation that scans for unreferenced SHAs. Returns ErrNotFound when the key does not exist.
func (*Store) Get ¶
Get retrieves the content and metadata for key in tenant. The caller is responsible for closing the returned ReadCloser. Returns ErrNotFound when the key does not exist.
func (*Store) GetBySHA ¶
GetBySHA retrieves content directly by its SHA-256 hex digest, bypassing the key alias index. Useful when the caller already holds the SHA from a prior Put response. key is used only to populate Meta.Key; pass the SHA itself if no logical key is known.
func (*Store) Head ¶
Head returns metadata for key without reading the blob content. Returns ErrNotFound when the key does not exist.
func (*Store) List ¶
List returns metadata for all keys in tenant whose names start with prefix. An empty prefix returns all keys. Results are in filesystem order (not guaranteed to be sorted); callers should sort if order matters.
func (*Store) Put ¶
func (s *Store) Put(key string, r io.Reader, contentType string) (sha string, md5hex string, created bool, err error)
Put stores content under key for tenant. Returns the SHA-256 hex digest and whether the blob was newly created (false = key existed and was overwritten with possibly different content). contentType is stored alongside the blob for retrieval; it may be empty.
The content is read once, SHA-256 is computed in a single streaming pass, and the result is written atomically. If an identical blob already exists under the computed SHA, no file write is performed.
func (*Store) PutBySHA ¶
PutBySHA stores the blob content under an alias of key, where the blob file is already on disk under hexSHA. This is used when the SHA is computed during a prior Put and the alias needs to be re-targeted. If the blob file does not exist, ErrNotFound is returned.
func (*Store) PutRaw ¶
func (s *Store) PutRaw(r io.Reader, contentType string) (sha string, md5hex string, created bool, err error)
PutRaw stores content and returns the SHA-256 hex digest. No key alias is written. This is the correct path when the SHA itself is the identifier (e.g. the history versioning system, or any purely content-addressed use). Retrieval goes through GetBySHA; the result will not appear in List.
If an identical blob already exists on disk the write is skipped and the existing SHA is returned — the deduplication guarantee still holds.
type Usage ¶
type Usage struct {
// BlobCount is the number of distinct blob files on disk (deduplicated).
// Two keys pointing to the same content count as one blob.
BlobCount int64
// KeyCount is the number of key aliases.
KeyCount int64
// Bytes is the total size of all blob files in bytes, excluding sidecars.
Bytes int64
}
Usage holds disk usage statistics for a single tenant's blob namespace.
type UsageSampler ¶
type UsageSampler struct {
// contains filtered or unexported fields
}
UsageSampler periodically walks one Store and caches its usage figures.
func NewUsageSampler ¶
func NewUsageSampler(s *Store, interval time.Duration) *UsageSampler
NewUsageSampler creates a UsageSampler for a single-tenant store. Call Start to begin sampling.
func (*UsageSampler) Current ¶
func (u *UsageSampler) Current() SampledUsage
Current returns the most recently cached usage for this store's tenant. SampledAt is zero if no walk has completed yet.
func (*UsageSampler) ForceResample ¶
func (u *UsageSampler) ForceResample()
ForceResample runs a usage walk synchronously and updates the cache before returning. Intended for tests that need a deterministic cache state without waiting for the ticker. Safe to call from any goroutine.
func (*UsageSampler) Start ¶
func (u *UsageSampler) Start()
Start launches the sampler goroutine. An initial walk runs immediately so the cache is warm before the first ticker fires.
func (*UsageSampler) Stop ¶
func (u *UsageSampler) Stop()
Stop signals the sampler to stop and blocks until it has exited.
Source Files