Documentation
¶
Overview ¶
Package bicbackend implements the management and storage of blob info cache data, including boltdb and memory implementations.
Index ¶
- Constants
- type BICContentScope
- type BICLocationReference
- type BICReplacementCandidate
- type BlobInfoCache
- type BoltBlobInfoCache
- func (bdc *BoltBlobInfoCache) CandidateLocations(ctx context.Context, transport string, scope BICContentScope, ...) []BICReplacementCandidate
- func (bdc *BoltBlobInfoCache) Close()
- func (bdc *BoltBlobInfoCache) Open()
- func (bdc *BoltBlobInfoCache) RecordDigestTransformerName(ctx context.Context, anyDigest digest.Digest, transformerName string)
- func (bdc *BoltBlobInfoCache) RecordDigestUntransformedPair(ctx context.Context, anyDigest digest.Digest, untransformed digest.Digest)
- func (bdc *BoltBlobInfoCache) RecordKnownLocation(ctx context.Context, transport string, scope BICContentScope, ...)
- func (bdc *BoltBlobInfoCache) UntransformedDigest(ctx context.Context, anyDigest digest.Digest) digest.Digest
- type CandidateWithTime
- type MemoryBlobInfoCache
- func (mem *MemoryBlobInfoCache) CandidateLocations(ctx context.Context, transport string, scope BICContentScope, ...) []BICReplacementCandidate
- func (mem *MemoryBlobInfoCache) Close()
- func (mem *MemoryBlobInfoCache) Open()
- func (mem *MemoryBlobInfoCache) RecordDigestTransformerName(ctx context.Context, blobDigest digest.Digest, transformerName string)
- func (mem *MemoryBlobInfoCache) RecordDigestUntransformedPair(ctx context.Context, anyDigest digest.Digest, untransformed digest.Digest)
- func (mem *MemoryBlobInfoCache) RecordKnownLocation(ctx context.Context, transport string, scope BICContentScope, ...)
- func (mem *MemoryBlobInfoCache) UntransformedDigest(ctx context.Context, anyDigest digest.Digest) digest.Digest
Constants ¶
const ( // Untransformed is the value we store in a blob info BlobInfoCache to indicate that we know that // the blob in the corresponding location is not transformed. Untransformed = "untransformed" // UnknownTransformer is the value we store in a blob info BlobInfoCache to indicate that we don't // know if the blob in the corresponding location is transformed (and if so, how) or not. UnknownTransformer = "unknown" )
const (
// LayerContent is the content scope for layers (blobs).
LayerContent = "Layer"
)
Content Scope type string declarations for common content categories.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BICContentScope ¶
type BICContentScope struct {
Opaque string
}
BICContentScope contains a description of the type of content being recorded, such as "Layer", "Artifact", etc. This reduces the search space by a broad categorization of content type.
type BICLocationReference ¶
type BICLocationReference struct {
Opaque string
}
BICLocationReference encapsulates transport-dependent representation of a blob location within a BICContentScope. Each transport can store arbitrary data using BlobInfoCache.RecordKnownLocation, and ImageDestination.TryReusingBlob can look it up using BlobInfoCache.CandidateLocations.
type BICReplacementCandidate ¶
type BICReplacementCandidate struct {
Digest digest.Digest
TransformerName string // either the name of a known transformer algorithm, or Untransformed or UnknownTransformer
Location BICLocationReference
}
BICReplacementCandidate is an item returned by BlobInfoCache.CandidateLocations.
func SortReplacementCandidates ¶
func SortReplacementCandidates(cs []CandidateWithTime, primaryDigest, untransformedDigest digest.Digest) []BICReplacementCandidate
SortReplacementCandidates consumes AND DESTROYS an array of possible replacement candidates with their last known existence times, the primary digest the user actually asked for, and the corresponding untransformed digest (if known, possibly equal to the primary digest), and returns an appropriately prioritized and/or trimmed result suitable for a return value from types.BlobInfoCache.CandidateLocations.
type BlobInfoCache ¶
type BlobInfoCache interface {
// UntransformedDigest returns an untransformed digest corresponding to anyDigest.
// May return anyDigest if it is known to be untransformed.
// Returns "" if nothing is known about the digest (it may be transformed or untransformed).
UntransformedDigest(ctx context.Context, anyDigest digest.Digest) digest.Digest
// RecordDigestUntransformedPair records that the untransformed version of anyDigest is untransformed.
RecordDigestUntransformedPair(ctx context.Context, anyDigest digest.Digest, untransformed digest.Digest)
// RecordKnownLocation records that a blob with the specified digest exists within the specified (transport, scope) scope,
// and can be reused given the opaque location data.
RecordKnownLocation(ctx context.Context, transport string, scope BICContentScope, digest digest.Digest, location BICLocationReference)
// RecordDigestTransformerName records a transformer for the blob with the specified digest, or Untransformed
// or UnknownTransformer.
RecordDigestTransformerName(ctx context.Context, anyDigest digest.Digest, transformerName string)
// CandidateLocations returns a sorted list of blobs and their locations
// that could possibly be reused within the specified (transport scope) (if they still
// exist, which is not guaranteed).
//
// If !canSubstitute, the returned candidates will match the submitted digest exactly; if
// canSubstitute, data from previous RecordDigestUntransformedPair calls is used to also look
// up variants of the blob which have the same untransformed digest.
//
// The TransformerName fields in returned data must never be UnknownTransformer.
CandidateLocations(ctx context.Context, transport string, scope BICContentScope, digest digest.Digest, canSubstitute bool) []BICReplacementCandidate
// Open sets up the BlobInfoCache for future accesses, potentially acquiring costly state. Each Open() must be paired with a Close().
// Note that public callers may call the BlobInfoCache operations without Open()/Close().
Open()
// Close destroys state created by Open().
Close()
}
BlobInfoCache records data useful for reusing blobs, or substituting equivalent ones, to avoid unnecessary blob copies. It records two kinds of data:
Sets of corresponding digest vs. untransformed digest ("DiffID") pairs: One of the two digests is known to be untransformed, and a single untransformed digest may correspond to more than one transformed digest. This allows matching transformed layer blobs to existing local untransformed layers (to avoid unnecessary download and untransformation), or untransformed layer blobs to existing remote transformed layers (to avoid unnecessary transformation and upload)/
It is allowed to record an (untransformed digest, the same untransformed digest) correspondence, to express that the digest is known to be untransformed (i.e. that a conversion from schema1 does not have to untransform the blob to compute a DiffID value).
Known blob locations, managed by individual transports: The transports call RecordKnownLocation when encountering a blob that could possibly be reused (typically in GetBlob/PutBlob/TryReusingBlob), recording transport-specific information that allows the transport to reuse the blob in the future; then, TryReusingBlob implementations can call CandidateLocations to look up previously recorded blob locations that could be reused.
Each transport defines its own “scopes” within which blob reuse is possible (e.g. in, the docker/distribution case, blobs can be directly reused within a registry, or mounted across registries within a registry server.)
None of the methods return an error indication: errors when neither reading from, nor writing to, the MemoryBlobInfoCache, should be fatal; users of the MemoryBlobInfoCache should just fall back to copying the blobs the usual way.
NOTE: Code for BlobInfoCache is derived from github.com/containers/image/v5/pkg/blobinfocache/, ported with changes for generic transformers versus compression, as well as removing unneeded support for legacy database versions.
func BoltCache ¶
func BoltCache(cachePath string) BlobInfoCache
BoltCache creates a BoltDB backed blob info cache. Note the BIC isn't opened until read/write operations.
func MemoryCache ¶
func MemoryCache() BlobInfoCache
MemoryCache returns a BlobInfoCache implementation which is in-memory only. This is primarily intended for tests, but would also apply in cases where bulk transfers are occurring and persistent storage for the BIC isn't available.
type BoltBlobInfoCache ¶
type BoltBlobInfoCache struct {
// contains filtered or unexported fields
}
BoltBlobInfoCache is a BlobInfoCache implementation which uses a BoltDB file at the specified path.
Note that we don’t keep the database open across operations, because that would lock the file and block any other users; instead, we need to open/close it for every single write or lookup.
func (*BoltBlobInfoCache) CandidateLocations ¶
func (bdc *BoltBlobInfoCache) CandidateLocations(ctx context.Context, transport string, scope BICContentScope, primaryDigest digest.Digest, canSubstitute bool) []BICReplacementCandidate
CandidateLocations returns a sorted, number of blobs and their locations that could possibly be reused within the specified (transport scope) (if they still exist, which is not guaranteed).
If !canSubstitute, the returned candidates will match the submitted digest exactly; if canSubstitute, data from previous RecordDigestUntransformedPair calls is used to also look up variants of the blob which have the same untransformed digest.
func (*BoltBlobInfoCache) Close ¶
func (bdc *BoltBlobInfoCache) Close()
Close destroys state created by Open().
func (*BoltBlobInfoCache) Open ¶
func (bdc *BoltBlobInfoCache) Open()
Open sets up the MemoryBlobInfoCache for future accesses, potentially acquiring costly state. Each Open() must be paired with a Close(). Note that public callers may call the BlobInfoCache operations without Open()/Close().
func (*BoltBlobInfoCache) RecordDigestTransformerName ¶
func (bdc *BoltBlobInfoCache) RecordDigestTransformerName(ctx context.Context, anyDigest digest.Digest, transformerName string)
RecordDigestTransformerName records that the blob with digest anyDigest was transformed with the specified transformer, or is Untransformed.
func (*BoltBlobInfoCache) RecordDigestUntransformedPair ¶
func (bdc *BoltBlobInfoCache) RecordDigestUntransformedPair(ctx context.Context, anyDigest digest.Digest, untransformed digest.Digest)
RecordDigestUntransformedPair records that the untransformed version of anyDigest is untransformed. It’s allowed for anyDigest == untransformed. WARNING: Only call this for LOCALLY VERIFIED data; don’t record a digest pair just because some remote author claims so (e.g. because a manifest/config pair exists); otherwise the MemoryBlobInfoCache could be poisoned and allow substituting unexpected blobs. (Eventually, the DiffIDs in image config could detect the substitution, but that may be too late, and not all image formats contain that data.)
func (*BoltBlobInfoCache) RecordKnownLocation ¶
func (bdc *BoltBlobInfoCache) RecordKnownLocation(ctx context.Context, transport string, scope BICContentScope, blobDigest digest.Digest, location BICLocationReference)
RecordKnownLocation records that a blob with the specified digest exists within the specified (transport, scope) scope, and can be reused given the opaque location data.
func (*BoltBlobInfoCache) UntransformedDigest ¶
func (bdc *BoltBlobInfoCache) UntransformedDigest(ctx context.Context, anyDigest digest.Digest) digest.Digest
UntransformedDigest returns an untransformed digest corresponding to anyDigest. May return anyDigest if it is known to be untransformed. Returns "" if nothing is known about the digest (it may be transformed or untransformed).
type CandidateWithTime ¶
type CandidateWithTime struct {
Candidate BICReplacementCandidate // The replacement candidate
LastSeen time.Time // Time the candidate was last known to exist (either read or written)
}
CandidateWithTime is the input to BICReplacementCandidate prioritization.
type MemoryBlobInfoCache ¶
type MemoryBlobInfoCache struct {
// contains filtered or unexported fields
}
MemoryBlobInfoCache implements an in-memory-only BlobInfoCache.
func (*MemoryBlobInfoCache) CandidateLocations ¶
func (mem *MemoryBlobInfoCache) CandidateLocations(ctx context.Context, transport string, scope BICContentScope, primaryDigest digest.Digest, canSubstitute bool) []BICReplacementCandidate
CandidateLocations returns a prioritized, limited, number of blobs and their locations that could possibly be reused within the specified (transport scope) (if they still exist, which is not guaranteed).
If !canSubstitute, the returned candidates will match the submitted digest exactly; if canSubstitute, data from previous RecordDigestUntransformedPair calls is used to also look up variants of the blob which have the same untransformed digest.
func (*MemoryBlobInfoCache) Close ¶
func (mem *MemoryBlobInfoCache) Close()
Close does nothing for memory BIC.
func (*MemoryBlobInfoCache) Open ¶
func (mem *MemoryBlobInfoCache) Open()
Open does nothing for memory BIC.
func (*MemoryBlobInfoCache) RecordDigestTransformerName ¶
func (mem *MemoryBlobInfoCache) RecordDigestTransformerName(ctx context.Context, blobDigest digest.Digest, transformerName string)
RecordDigestTransformerName records that the blob with the specified digest is either transformed with the specified algorithm, or untransformed, or that we no longer know.
func (*MemoryBlobInfoCache) RecordDigestUntransformedPair ¶
func (mem *MemoryBlobInfoCache) RecordDigestUntransformedPair(ctx context.Context, anyDigest digest.Digest, untransformed digest.Digest)
RecordDigestUntransformedPair records that the untransformed version of anyDigest is untransformed. It’s allowed for anyDigest == untransformed.
func (*MemoryBlobInfoCache) RecordKnownLocation ¶
func (mem *MemoryBlobInfoCache) RecordKnownLocation(ctx context.Context, transport string, scope BICContentScope, blobDigest digest.Digest, location BICLocationReference)
RecordKnownLocation records that a blob with the specified digest exists within the specified (transport, scope) scope, and can be reused given the opaque location data.
func (*MemoryBlobInfoCache) UntransformedDigest ¶
func (mem *MemoryBlobInfoCache) UntransformedDigest(ctx context.Context, anyDigest digest.Digest) digest.Digest
UntransformedDigest returns an untransformed digest corresponding to anyDigest. May return anyDigest if it is known to be untransformed. Returns "" if nothing is known about the digest (it may be transformed or untransformed).
Source Files