metadata

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ExpandSharedOuterSources added in v0.3.0 

func ExpandSharedOuterSources(meta *metapb.FileMetadata) error

ExpandSharedOuterSources resolves NestedSegmentSource.SharedOuterSourceIndex references in-place. Sources with a non-zero index inherit Segments, AesKey, AesIv, and (if unset) InnerVolumeSize from meta.SharedOuterSources[index-1]. Slice headers share their underlying array — RAM cost is unchanged from the legacy layout. Safe to call on any FileMetadata; a no-op when SharedOuterSources is empty.

The dedupe written by archive.NewFileMetadataFromContent is the write-side counterpart of this expansion.

func FlatSegments added in v0.3.0 

func FlatSegments(store *metapb.NzbStore) []*metapb.NzbSeg

FlatSegments returns all segments in flat order: files in order, each file's segments in the order they appear (sorted by number at import time).

func KnownHolesFromProto added in v0.3.0 

func KnownHolesFromProto(rows []*metapb.HoleRun) []holes.Run

KnownHolesFromProto rebuilds runs from storage; malformed rows are dropped.

func KnownHolesToProto added in v0.3.0 

func KnownHolesToProto(runs []holes.Run) []*metapb.HoleRun

KnownHolesToProto converts accumulator runs for storage.

Types

type BackupWorker 

type BackupWorker struct {
	// contains filtered or unexported fields
}

func NewBackupWorker 

func NewBackupWorker(configGetter config.ConfigGetter) *BackupWorker

func (*BackupWorker) Start 

func (w *BackupWorker) Start(ctx context.Context) error

func (*BackupWorker) Stop 

func (w *BackupWorker) Stop(ctx context.Context)

type FileMetadataLite added in v0.3.0 

type FileMetadataLite struct {
	FileSize   int64
	ModifiedAt int64
	Status     metapb.FileStatus
}

FileMetadataLite holds the minimal metadata needed for directory listings. This avoids keeping full FileMetadata protos (with SegmentData, Par2Files, etc.) in memory just for Readdir.

type MetadataReader 

type MetadataReader struct {
	// contains filtered or unexported fields
}

MetadataReader provides read operations for the virtual filesystem

func NewMetadataReader 

func NewMetadataReader(service *MetadataService) *MetadataReader

NewMetadataReader creates a new metadata reader

func (*MetadataReader) GetFileMetadata 

func (mr *MetadataReader) GetFileMetadata(virtualPath string) (*metapb.FileMetadata, error)

GetFileMetadata gets metadata for a virtual file

func (*MetadataReader) GetMetadataService 

func (mr *MetadataReader) GetMetadataService() *MetadataService

GetMetadataService returns the underlying metadata service

type MetadataService 

type MetadataService struct {
	// contains filtered or unexported fields
}

MetadataService provides low-level read/write operations for metadata files.

Only a lightweight metadata projection (liteCache) is kept in memory. The full FileMetadata proto — dominated by SegmentData/NestedSources slices holding thousands of message-ID strings — is never cached. Callers that need segments (Open, HealthChecker) re-read from disk each time; the proto then lives only for the duration of the open handle or the health check. This bounds steady-state memory at ~liteCache_entries × 40 bytes instead of the previous unbounded segment retention.

func NewMetadataService 

func NewMetadataService(rootPath string) *MetadataService

NewMetadataService creates a new metadata service

func (*MetadataService) AddKnownHoles added in v0.3.0 

func (ms *MetadataService) AddKnownHoles(virtualPath string, runs []holes.Run) error

AddKnownHoles merges newly confirmed missing-segment runs into a file's persisted hole map, using the same read-modify-write path as status updates. Merging goes through holes.Accumulator, so the write is idempotent and concurrent discoveries collapse into maximal runs.

func (*MetadataService) CleanupEmptyDirectories 

func (ms *MetadataService) CleanupEmptyDirectories(virtualPath string, protected []string) error

CleanupEmptyDirectories recursively removes empty directories under the given virtual path. Uses a bottom-up approach to ensure parent directories are also removed if they become empty. 

func (ms *MetadataService) CleanupOrphanedIDSymlinks(ctx context.Context) (int, error)

CleanupOrphanedIDSymlinks walks the .ids/ directory and removes symlinks whose targets no longer exist. Empty shard directories are cleaned up afterwards. Returns the number of removed symlinks.

func (*MetadataService) CreateDirectory 

func (ms *MetadataService) CreateDirectory(name string) error

func (*MetadataService) CreateFileMetadata 

func (ms *MetadataService) CreateFileMetadata(
	fileSize int64,
	sourceNzbPath string,
	status metapb.FileStatus,
	segmentData []*metapb.SegmentData,
	encryption metapb.Encryption,
	password string,
	salt string,
	aesKey []byte,
	aesIv []byte,
	releaseDate int64,
	par2Files []*metapb.Par2FileReference,
	nzbdavId string,
) *metapb.FileMetadata

CreateFileMetadata creates a new FileMetadata with basic fields

func (*MetadataService) DeleteCorruptedFile added in v0.3.0 

func (ms *MetadataService) DeleteCorruptedFile(ctx context.Context, virtualPath string, deleteSourceNzb bool, physicalPath string, physicalRoot string) error

DeleteCorruptedFile removes a file's metadata (and optionally its source NZB), then removes the physical library file (if any) and cleans up now-empty parent directories in the physical library tree. Metadata-tree cleanup is already handled by DeleteFileMetadataWithSourceNzb; the physical-path removal is error-tolerant since physicalPath is often just a view into the same mount and may already be gone.

func (*MetadataService) DeleteDirectory 

func (ms *MetadataService) DeleteDirectory(virtualPath string) error

DeleteDirectory deletes a metadata directory and all its contents

func (*MetadataService) DeleteFileMetadata 

func (ms *MetadataService) DeleteFileMetadata(virtualPath string) error

DeleteFileMetadata deletes a metadata file

func (*MetadataService) DeleteFileMetadataWithSourceNzb 

func (ms *MetadataService) DeleteFileMetadataWithSourceNzb(ctx context.Context, virtualPath string, deleteSourceNzb bool) error

DeleteFileMetadataWithSourceNzb deletes a metadata file and optionally its source NZB

func (*MetadataService) DirectoryExists 

func (ms *MetadataService) DirectoryExists(virtualPath string) bool

DirectoryExists checks if a metadata directory exists

func (*MetadataService) FileExists 

func (ms *MetadataService) FileExists(virtualPath string) bool

FileExists checks if a metadata file exists for the given virtual path

func (*MetadataService) GetMetadataDirectoryPath 

func (ms *MetadataService) GetMetadataDirectoryPath(virtualPath string) string

GetMetadataDirectoryPath returns the filesystem path for a metadata directory

func (*MetadataService) GetMetadataFilePath 

func (ms *MetadataService) GetMetadataFilePath(virtualPath string) string

GetMetadataFilePath returns the filesystem path for a metadata file

func (*MetadataService) IncStoreRef added in v0.3.0 

func (ms *MetadataService) IncStoreRef(ctx context.Context, storePath string)

IncStoreRef increments the reference count for a v3 store file. No-op when no StoreRefCounter is configured. Refcounting is best-effort: a DB failure is logged but does not fail the caller. A missed increment means a later DecStoreRef may drive the count to 0 prematurely, but store-file deletion is tolerant of the file already being absent.

func (*MetadataService) ListDirectory 

func (ms *MetadataService) ListDirectory(virtualPath string) ([]string, error)

ListDirectory lists all metadata files in a directory

func (*MetadataService) ListDirectoryAll added in v0.3.0 

func (ms *MetadataService) ListDirectoryAll(virtualPath string) (dirs []fs.FileInfo, fileNames []string, err error)

ListDirectoryAll returns both subdirectory fs.FileInfo entries and virtual file names from a single os.ReadDir call. This is used by Readdir to avoid two separate directory reads.

func (*MetadataService) MoveToCorrupted 

func (ms *MetadataService) MoveToCorrupted(ctx context.Context, virtualPath string) error

MoveToCorrupted moves a metadata file to a special corrupted directory for safety

func (*MetadataService) ReadFileMetadata 

func (ms *MetadataService) ReadFileMetadata(virtualPath string) (*metapb.FileMetadata, error)

ReadFileMetadata reads file metadata from disk. The full proto (including SegmentData and NestedSources) is returned to the caller but NOT cached — those slices dominate heap usage and must not be retained beyond the caller's handle. As a side effect, the lightweight projection is cached so subsequent Readdir/Stat calls are fast without a disk read.

func (*MetadataService) ReadFileMetadataLite added in v0.3.0 

func (ms *MetadataService) ReadFileMetadataLite(virtualPath string) (*FileMetadataLite, error)

ReadFileMetadataLite reads only the lightweight fields (size, modtime, status) needed for directory listings. On cache miss it reads at most liteScanBytes from the .meta file and scans the proto wire format for the three lite fields, never instantiating the full FileMetadata proto or its NestedSources/SegmentData slices. Falls back to a full read in the rare case the partial buffer doesn't cover the lite fields.

func (*MetadataService) RenameFileMetadata 

func (ms *MetadataService) RenameFileMetadata(oldVirtualPath, newVirtualPath string) error

RenameFileMetadata atomically renames a metadata file (and its .id sidecar) from oldVirtualPath to newVirtualPath. Uses os.Rename for atomicity on the same filesystem, falling back to read-write-delete for cross-device moves.

func (*MetadataService) SetStoreRefCounter added in v0.3.0 

func (ms *MetadataService) SetStoreRefCounter(c StoreRefCounter)

SetStoreRefCounter wires in a StoreRefCounter so that reference counts on shared NzbStore files are maintained when metadata is deleted or created.

func (*MetadataService) Store added in v0.3.0 

func (ms *MetadataService) Store() *StoreService

Store returns the StoreService used by this MetadataService.

func (*MetadataService) UpdateFileMetadata 

func (ms *MetadataService) UpdateFileMetadata(virtualPath string, updateFunc func(*metapb.FileMetadata)) error

UpdateFileMetadata updates the modified timestamp of metadata

func (*MetadataService) UpdateFileStatus 

func (ms *MetadataService) UpdateFileStatus(virtualPath string, status metapb.FileStatus) error

UpdateFileStatus updates the status of a file in metadata

func (*MetadataService) WriteFileMetadata 

func (ms *MetadataService) WriteFileMetadata(virtualPath string, metadata *metapb.FileMetadata) error

WriteFileMetadata writes file metadata to disk

func (*MetadataService) WriteFileMetadataAuto added in v0.3.0 

func (ms *MetadataService) WriteFileMetadataAuto(ctx context.Context, virtualPath string, metadata *metapb.FileMetadata, index map[string]int64, storeRef string) error

WriteFileMetadataAuto writes v3 store-backed metadata when storeRef is set, falling back to the v1 inline format if the v3 conversion fails (so a store/index problem on one file never blocks the import). With an empty storeRef it writes v1. This is the single entry point import processors should use.

func (*MetadataService) WriteFileMetadataV3 added in v0.3.0 

func (ms *MetadataService) WriteFileMetadataV3(ctx context.Context, virtualPath string, metadata *metapb.FileMetadata, index map[string]int64, storeRef string) error

WriteFileMetadataV3 writes metadata directly in the v3 store-backed format: it converts the inline SegmentData of the main file, PAR2 files, and nested sources into SegmentRefs/SegmentRuns against the release's flat segment index, points the meta at storeRef, and increments the store reference count exactly once.

The conversion runs on a clone so a failure (e.g. a segment id missing from the index) leaves the caller's in-memory meta untouched, letting the caller fall back to a v1 WriteFileMetadata. Freshly-built archive metas still carry the SharedOuterSources dedup, so it is expanded first (mirroring the read path) before each nested source is converted independently and the dedup dissolved.

type StoreRefCounter added in v0.3.0 

type StoreRefCounter interface {
	IncStoreRef(ctx context.Context, storePath string) error
	DecStoreRef(ctx context.Context, storePath string) (int64, error)
}

StoreRefCounter tracks reference counts for shared NzbStore files. Implementations are provided by the database layer; the nil value is always safe to use.

type StoreService added in v0.3.0 

type StoreService struct {
	// contains filtered or unexported fields
}

StoreService reads/writes per-release NzbStore files (zstd proto) and caches decompressed stores keyed by store ref (path).

func NewStoreService added in v0.3.0 

func NewStoreService(rootPath string) *StoreService

NewStoreService creates a StoreService rooted at rootPath with an LRU cache.

func (*StoreService) ReadStore added in v0.3.0 

func (ss *StoreService) ReadStore(ref string) (*metapb.NzbStore, error)

ReadStore reads and decompresses a store, caching the result.

func (*StoreService) RegenerateNZB added in v0.3.0 

func (ss *StoreService) RegenerateNZB(storePath string) ([]byte, error)

RegenerateNZB reads the store at storePath and returns NZB XML bytes. Returns (nil, nil) if the store does not exist.

func (*StoreService) WriteStore added in v0.3.0 

func (ss *StoreService) WriteStore(ref string, store *metapb.NzbStore) error

WriteStore writes zstd(proto) to ref atomically and refreshes the cache.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.