snapshotfs

package
v0.23.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 8, 2026 License: Apache-2.0 Imports: 32 Imported by: 10

Documentation

Overview

Package snapshotfs implements virtual filesystem on top of snapshots in repo.Repository.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AllSourcesEntry 

func AllSourcesEntry(rep repo.Repository) fs.Directory

AllSourcesEntry returns fs.Directory that contains the list of all snapshot sources found in the repository.

func AutoDetectEntryFromObjectID added in v0.7.2 

func AutoDetectEntryFromObjectID(ctx context.Context, rep repo.Repository, oid object.ID, maybeName string) fs.Entry

AutoDetectEntryFromObjectID returns fs.Entry (either file or directory) for the provided object ID. It uses heuristics to determine whether object ID is possibly a directory and treats it as such.

func CalculateStorageStats added in v0.10.6 

func CalculateStorageStats(ctx context.Context, rep repo.Repository, manifests []*snapshot.Manifest, callback func(m *snapshot.Manifest) error) error

CalculateStorageStats calculates the storage statistics for a given list of snapshots, by determining the count and size of unique contents and objects for each snapshot in the slice as well as running total so far for all the previous snapshots.

For each snapshot the provided callback is invoked as soon as the results are available and iteration continues until the callback returns a non-nil error.

func DirectoryEntry 

func DirectoryEntry(rep repo.Repository, objectID object.ID, dirSummary *fs.DirectorySummary) fs.Directory

DirectoryEntry returns fs.Directory based on repository object with the specified ID. The existence or validity of the directory object is not validated until its contents are read.

func EntryFromDirEntry added in v0.4.0 

func EntryFromDirEntry(r repo.Repository, md *snapshot.DirEntry) fs.Entry

EntryFromDirEntry returns a filesystem entry based on the directory entry.

func FilesystemDirectoryFromIDWithPath added in v0.7.2 

func FilesystemDirectoryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Directory, error)

FilesystemDirectoryFromIDWithPath returns a filesystem directory entry for the provided object ID, which can be a snapshot manifest ID or an object ID with path.

func FilesystemEntryFromIDWithPath added in v0.7.2 

func FilesystemEntryFromIDWithPath(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (fs.Entry, error)

FilesystemEntryFromIDWithPath returns a filesystem entry for the provided object ID, which can be a snapshot manifest ID or an object ID with path. If multiple snapshots match and they don't agree on root object attributes and consistentAttributes==true the function fails, otherwise it returns the latest of the snapshots.

func FindSnapshotByRootObjectIDOrManifestID added in v0.20.0 

func FindSnapshotByRootObjectIDOrManifestID(ctx context.Context, rep repo.Repository, rootID string, consistentAttributes bool) (*snapshot.Manifest, error)

FindSnapshotByRootObjectIDOrManifestID returns the list of matching snapshots for a given rootID. which can be either snapshot manifest ID (which matches 0 or 1 snapshots) or the root object ID (which can match arbitrary number of snapshots). If multiple snapshots match and they don't agree on root object attributes and consistentAttributes==true the function fails, otherwise it returns the latest of the snapshots.

func GetEntryFromPlaceholder added in v0.9.0 

func GetEntryFromPlaceholder(ctx context.Context, r repo.Repository, defp snapshot.HasDirEntryOrNil) (fs.Entry, error)

GetEntryFromPlaceholder returns a fs.Entry for shallow placeholder defp referencing a real Entry in Repository r.

func GetNestedEntry added in v0.7.2 

func GetNestedEntry(ctx context.Context, startingDir fs.Entry, pathElements []string) (fs.Entry, error)

GetNestedEntry returns nested entry with a given name path.

func IsDirectoryID added in v0.7.2 

func IsDirectoryID(oid object.ID) bool

IsDirectoryID determines whether given object ID represents a directory.

func ParseObjectIDWithPath added in v0.7.2 

func ParseObjectIDWithPath(ctx context.Context, rep repo.Repository, objectIDWithPath string) (object.ID, error)

ParseObjectIDWithPath interprets the given ID string (which could be an object ID optionally followed by nested path specification) and returns corresponding object.ID.

func RewriteFail added in v0.11.0 

func RewriteFail(ctx context.Context, parentPath string, entry *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteFail is a callback that fails the entire rewrite process when a directory is unreadable.

func RewriteKeep added in v0.11.0 

func RewriteKeep(ctx context.Context, parentPath string, input *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteKeep is a callback that keeps the unreadable entry.

func RewriteRemove added in v0.11.0 

func RewriteRemove(ctx context.Context, parentPath string, entry *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteRemove is a callback that removes the entire failed entry.

func SnapshotRoot 

func SnapshotRoot(rep repo.Repository, man *snapshot.Manifest) (fs.Entry, error)

SnapshotRoot returns fs.Entry representing the root of a snapshot.

func WriteDirManifest added in v0.20.0 

func WriteDirManifest(ctx context.Context, rep repo.RepositoryWriter, dirRelativePath string, dirManifest *snapshot.DirManifest, metadataComp compression.Name) (object.ID, error)

WriteDirManifest writes a directory manifest to the repository and returns the object ID.

Types

type DirManifestBuilder added in v0.11.0 

type DirManifestBuilder struct {
	// contains filtered or unexported fields
}

DirManifestBuilder builds directory manifests.

func (*DirManifestBuilder) AddEntry added in v0.11.0 

func (b *DirManifestBuilder) AddEntry(de *snapshot.DirEntry)

AddEntry adds a directory entry to the builder.

func (*DirManifestBuilder) AddFailedEntry added in v0.11.0 

func (b *DirManifestBuilder) AddFailedEntry(relPath string, isIgnoredError bool, err error)

AddFailedEntry adds a failed directory entry to the builder and increments either ignored or fatal error count.

func (*DirManifestBuilder) Build added in v0.11.0 

func (b *DirManifestBuilder) Build(dirModTime fs.UTCTimestamp, incompleteReason string) *snapshot.DirManifest

Build builds the directory manifest.

func (*DirManifestBuilder) Clone added in v0.11.0 

func (b *DirManifestBuilder) Clone() *DirManifestBuilder

Clone clones the current state of dirManifestBuilder.

type DirRewriter added in v0.11.0 

type DirRewriter struct {
	// contains filtered or unexported fields
}

DirRewriter rewrites contents of directories by walking the snapshot tree recursively.

func NewDirRewriter added in v0.11.0 

func NewDirRewriter(ctx context.Context, rep repo.RepositoryWriter, opts DirRewriterOptions) (*DirRewriter, error)

NewDirRewriter creates a new directory rewriter.

func (*DirRewriter) Close added in v0.11.0 

func (rw *DirRewriter) Close(ctx context.Context)

Close closes the rewriter.

func (*DirRewriter) RewriteSnapshotManifest added in v0.11.0 

func (rw *DirRewriter) RewriteSnapshotManifest(ctx context.Context, man *snapshot.Manifest, metadataComp compression.Name) (bool, error)

RewriteSnapshotManifest rewrites the directory tree starting at a given manifest.

type DirRewriterOptions added in v0.11.0 

type DirRewriterOptions struct {
	Parallel int

	RewriteEntry RewriteDirEntryCallback

	// when != nil will be invoked to replace directory that can't be read,
	// by default RewriteAsStub()
	OnDirectoryReadFailure RewriteFailedEntryCallback
}

DirRewriterOptions provides options for directory rewriter.

type EntryCallback added in v0.10.6 

type EntryCallback func(ctx context.Context, entry fs.Entry, oid object.ID, entryPath string) error

EntryCallback is invoked when walking the tree of snapshots.

type RewriteDirEntryCallback added in v0.11.0 

type RewriteDirEntryCallback func(ctx context.Context, parentPath string, input *snapshot.DirEntry) (*snapshot.DirEntry, error)

RewriteDirEntryCallback returns a replacement for the provided directory entry in the provided path. nil indicates that the entry should be removed.

type RewriteFailedEntryCallback added in v0.11.0 

type RewriteFailedEntryCallback func(ctx context.Context, parentPath string, input *snapshot.DirEntry, err error) (*snapshot.DirEntry, error)

RewriteFailedEntryCallback is invoked rewriting a file or directory.

func RewriteAsStub added in v0.11.0 

func RewriteAsStub(rep repo.RepositoryWriter) RewriteFailedEntryCallback

RewriteAsStub returns a callback that replaces the invalid entry with a stub that describes the error.

type TreeWalker 

type TreeWalker struct {
	// contains filtered or unexported fields
}

TreeWalker processes snapshot filesystem trees by invoking the provided callback once for each object found in the tree.

func NewTreeWalker 

func NewTreeWalker(ctx context.Context, options TreeWalkerOptions) (*TreeWalker, error)

NewTreeWalker creates new tree walker.

func (*TreeWalker) Close added in v0.10.6 

func (w *TreeWalker) Close(ctx context.Context)

Close closes the tree walker.

func (*TreeWalker) Err added in v0.10.6 

func (w *TreeWalker) Err() error

Err returns the error encountered when walking the tree.

func (*TreeWalker) GetErrors added in v0.21.0 

func (w *TreeWalker) GetErrors() (errs []error, numErrors int)

GetErrors returns a copy of the list of errors found during the tree walk, as well as the total count of errors encountered.

func (*TreeWalker) Process added in v0.10.6 

func (w *TreeWalker) Process(ctx context.Context, e fs.Entry, entryPath string) error

Process processes the snapshot tree entry.

func (*TreeWalker) ReportError added in v0.10.6 

func (w *TreeWalker) ReportError(ctx context.Context, entryPath string, err error)

ReportError reports the error.

func (*TreeWalker) TooManyErrors added in v0.10.6 

func (w *TreeWalker) TooManyErrors() bool

TooManyErrors reports true if there are too many errors already reported.

type TreeWalkerOptions added in v0.10.6 

type TreeWalkerOptions struct {
	EntryCallback EntryCallback

	Parallelism int
	MaxErrors   int
}

TreeWalkerOptions provides optional fields for TreeWalker.

type UnreadableDirEntryReplacement added in v0.11.0 

type UnreadableDirEntryReplacement struct {
	Info  string             `json:"info"`
	Error string             `json:"error"`
	Entry *snapshot.DirEntry `json:"entry"`
}

UnreadableDirEntryReplacement is serialized as a stub object replacing unreadable file or directory.

type Verifier added in v0.11.0 

type Verifier struct {
	// contains filtered or unexported fields
}

Verifier allows efficient verification of large amounts of filesystem entries in parallel.

func NewVerifier added in v0.11.0 

func NewVerifier(_ context.Context, rep repo.Repository, opts VerifierOptions) *Verifier

NewVerifier creates a verifier.

func (*Verifier) AddToExpectedTotals added in v0.21.0 

func (v *Verifier) AddToExpectedTotals(objs, files, bytes int64)

AddToExpectedTotals adds the provided values to the corresponding stat total values. If the caller can precompute the expected number of entries that will be iterated over, they can add them to the progress output.

func (*Verifier) InParallel added in v0.11.0 

func (v *Verifier) InParallel(ctx context.Context, enqueue func(tw *TreeWalker) error) (VerifierResult, error)

InParallel starts parallel verification and invokes the provided function which can call Process() on in the provided TreeWalker. Errors and stats are accumulated into a VerifierResult and returned, independent of whether the error return is nil, that is, `VerifierResult` will contain useful, partial stats when an error is returned, including a collection of errors found in the verification process.

func (*Verifier) ShowFinalStats added in v0.14.0 

func (v *Verifier) ShowFinalStats(ctx context.Context)

ShowFinalStats logs final verification statistics.

func (*Verifier) ShowStats added in v0.11.0 

func (v *Verifier) ShowStats(ctx context.Context)

ShowStats logs verification statistics.

func (*Verifier) VerifyFile added in v0.11.0 

func (v *Verifier) VerifyFile(ctx context.Context, oid object.ID, entryPath string, size int64) error

VerifyFile verifies a single file object (using content check, blob map check or full read).

type VerifierOptions added in v0.11.0 

type VerifierOptions struct {
	VerifyFilesPercent float64
	FileQueueLength    int
	Parallelism        int
	MaxErrors          int
	BlobMap            map[blob.ID]blob.Metadata
	JSONStats          bool
}

VerifierOptions provides options for the verifier.

type VerifierResult added in v0.21.0 

type VerifierResult struct {
	Stats        VerifierStats `json:"stats"`
	ErrorCount   int           `json:"errorCount"`
	Errors       []error       `json:"-"`
	ErrorStrings []string      `json:"errorStrings,omitempty"`
}

VerifierResult returns results from the verifier.

type VerifierStats added in v0.21.0 

type VerifierStats struct {
	ProcessedObjectCount int64 `json:"processedObjectCount"`
	ProcessedBytes       int64 `json:"processedBytes"`
	ReadFileCount        int64 `json:"readFileCount"`
	ReadBytes            int64 `json:"readBytes"`
	ExpectedTotalObjects int64 `json:"expectedTotalObjectCount"`
	ExpectedTotalFiles   int64 `json:"expectedTotalFileCount"`
	ExpectedTotalBytes   int64 `json:"expectedTotalBytes"`
}

VerifierStats contains stats on the amount of work done and current progress.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.