metadata

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 3, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrDirectoryNotEmpty = errors.New("metadata directory not empty")

ErrDirectoryNotEmpty reports that DeleteDirectoryIfEmpty left a directory in place because it still held entries.

View Source 
var ErrMigrationRunning = errors.New("metadata migration already running")

ErrMigrationRunning is returned when a second run is requested.

Functions

func BuildStore 

func BuildStore(n *nzbparser.Nzb) (*metapb.NzbStore, map[string]int64)

BuildStore converts a parsed NZB into a NzbStore (for persistence) plus a message-id → flat-store-index lookup used to emit SegmentRefs. Segments are stored in their natural NzbSegments order (by Number after sort).

func ExpandSharedOuterSources 

func ExpandSharedOuterSources(meta *metapb.FileMetadata) error

ExpandSharedOuterSources resolves NestedSegmentSource.SharedOuterSourceIndex references in-place. Sources with a non-zero index inherit Segments, AesKey, AesIv, and (if unset) InnerVolumeSize from meta.SharedOuterSources[index-1]. Slice headers share their underlying array — RAM cost is unchanged from the legacy layout. Safe to call on any FileMetadata; a no-op when SharedOuterSources is empty.

The dedupe written by archive.NewFileMetadataFromContent is the write-side counterpart of this expansion.

func FlatSegments 

func FlatSegments(store *metapb.NzbStore) []*metapb.NzbSeg

FlatSegments returns all segments in flat order: files in order, each file's segments in the order they appear (sorted by number at import time).

func KnownHolesFromProto 

func KnownHolesFromProto(rows []*metapb.HoleRun) []holes.Run

KnownHolesFromProto rebuilds runs from storage regardless of age or provider set; malformed rows are dropped. Callers deciding what to trust at read time use LiveKnownHoles.

func KnownHolesToProto 

func KnownHolesToProto(runs []holes.Run) []*metapb.HoleRun

KnownHolesToProto converts accumulator runs for storage, stamped now.

func LiveKnownHoles 

func LiveKnownHoles(rows []*metapb.HoleRun, storedFP, fingerprint string, now time.Time) []holes.Run

LiveKnownHoles returns the stored runs still trusted now: stamped within holes.TTL and recorded under fingerprint. Unstamped (legacy) or foreign runs are dropped so the next read through them re-probes the article and, if it is still gone, records it afresh.

Types

type BackupWorker 

type BackupWorker struct {
	// contains filtered or unexported fields
}

func NewBackupWorker 

func NewBackupWorker(configGetter config.ConfigGetter) *BackupWorker

func (*BackupWorker) Start 

func (w *BackupWorker) Start(ctx context.Context) error

func (*BackupWorker) Stop 

func (w *BackupWorker) Stop(ctx context.Context)

type FileMetadataLite 

type FileMetadataLite struct {
	FileSize   int64
	ModifiedAt int64
	Status     metapb.FileStatus
}

FileMetadataLite holds the minimal metadata needed for directory listings. This avoids keeping full FileMetadata protos (with SegmentData, Par2Files, etc.) in memory just for Readdir.

type GroupResult 

type GroupResult struct {
	Key           string   `json:"key"`
	StoreRef      string   `json:"store_ref"`
	Faithful      bool     `json:"faithful"`
	FilesMigrated int      `json:"files_migrated"`
	FilesFailed   int      `json:"files_failed"`
	BytesBefore   int64    `json:"bytes_before"`
	BytesAfter    int64    `json:"bytes_after"`
	Failures      []string `json:"failures,omitempty"`
}

GroupResult reports what happened to one release group.

type LegacyGroup 

type LegacyGroup struct {
	Key   string
	Files []LegacyMeta
}

LegacyGroup is the set of legacy metas belonging to one release. Files are sorted by meta path so a migration run is deterministic.

type LegacyMeta 

type LegacyMeta struct {
	MetaPath    string
	VirtualPath string
	SizeBytes   int64
	Meta        *metapb.FileMetadata
}

LegacyMeta is one v1 (inline-segment) metadata file discovered by a scan.

Meta is deliberately nil after a scan: a legacy meta carries its segments inline, so retaining every one of them across a whole-library walk is the allocation pattern that ReadFileMetadataLite exists to avoid (see the 7.94 GB spike documented at service.go:405). LoadGroupMetas fills it in for one release at a time, just before that release is migrated.

type MetadataReader 

type MetadataReader struct {
	// contains filtered or unexported fields
}

MetadataReader provides read operations for the virtual filesystem

func NewMetadataReader 

func NewMetadataReader(service *MetadataService) *MetadataReader

NewMetadataReader creates a new metadata reader

func (*MetadataReader) GetFileMetadata 

func (mr *MetadataReader) GetFileMetadata(virtualPath string) (*metapb.FileMetadata, error)

GetFileMetadata gets metadata for a virtual file

func (*MetadataReader) GetMetadataService 

func (mr *MetadataReader) GetMetadataService() *MetadataService

GetMetadataService returns the underlying metadata service

type MetadataService 

type MetadataService struct {
	// contains filtered or unexported fields
}

MetadataService provides low-level read/write operations for metadata files.

The full FileMetadata proto is never cached: callers that need segments (Open, HealthChecker) re-read from disk each time, so the proto lives only for the duration of the open handle or the health check. Two caches hold state between calls, and both are bounded:

  • liteCache, a lightweight projection (size, modtime, status) at ~liteCache_entries × 40 bytes.
  • store's decoded NzbStore cache, which v3 metadata resolves segments against. Stores are shared per release and are the one place message-ID strings do survive a call, so that cache is bounded by estimated bytes (defaultStoreCacheBytes) rather than by entry count — bounding it by entry count left steady-state retention unbounded (issue #819).

func NewMetadataService 

func NewMetadataService(rootPath string) *MetadataService

NewMetadataService creates a new metadata service

func (*MetadataService) AddKnownHoles 

func (ms *MetadataService) AddKnownHoles(virtualPath string, runs []holes.Run, fingerprint string) error

AddKnownHoles merges newly confirmed missing-segment runs into a file's persisted hole map under fingerprint, using the same read-modify-write path as status updates. Stored runs recorded under a different provider set are dropped first: they answered a question about other servers. Merging goes through holes.Accumulator, so the write is idempotent and concurrent discoveries collapse into maximal runs; a merged run is stamped now when it touches a new run and otherwise keeps its newest stored stamp.

func (*MetadataService) CleanupEmptyDirectories 

func (ms *MetadataService) CleanupEmptyDirectories(virtualPath string, protected []string) error

CleanupEmptyDirectories recursively removes empty directories under the given virtual path. Uses a bottom-up approach to ensure parent directories are also removed if they become empty. 

func (ms *MetadataService) CleanupOrphanedIDSymlinks(ctx context.Context) (int, error)

CleanupOrphanedIDSymlinks walks the .ids/ directory and removes symlinks whose targets no longer exist. Empty shard directories are cleaned up afterwards. Returns the number of removed symlinks.

func (*MetadataService) CorruptedStats 

func (ms *MetadataService) CorruptedStats(ctx context.Context) (int, int64, error)

CorruptedStats reports how many safety copies corrupted_metadata currently holds and how much disk they occupy.

func (*MetadataService) CreateDirectory 

func (ms *MetadataService) CreateDirectory(name string) error

func (*MetadataService) CreateFileMetadata 

func (ms *MetadataService) CreateFileMetadata(
	fileSize int64,
	sourceNzbPath string,
	status metapb.FileStatus,
	segmentData []*metapb.SegmentData,
	encryption metapb.Encryption,
	password string,
	salt string,
	aesKey []byte,
	aesIv []byte,
	releaseDate int64,
	par2Files []*metapb.Par2FileReference,
	nzbdavId string,
) *metapb.FileMetadata

CreateFileMetadata creates a new FileMetadata with basic fields

func (*MetadataService) DeleteCorruptedFile 

func (ms *MetadataService) DeleteCorruptedFile(ctx context.Context, virtualPath string, physicalPath string, physicalRoot string) error

DeleteCorruptedFile removes a file's metadata, then removes the physical library file (if any) and cleans up now-empty parent directories in the physical library tree. Metadata-tree cleanup is already handled by DeleteFileMetadata; the physical-path removal is error-tolerant since physicalPath is often just a view into the same mount and may already be gone.

func (*MetadataService) DeleteDirectory 

func (ms *MetadataService) DeleteDirectory(virtualPath string) error

DeleteDirectory deletes a metadata directory and all its contents

func (*MetadataService) DeleteDirectoryIfEmpty 

func (ms *MetadataService) DeleteDirectoryIfEmpty(virtualPath string) error

DeleteDirectoryIfEmpty deletes a metadata directory only if it holds no files or subdirectories, so a release folder shared with a concurrent import is never removed from under it. Returns ErrDirectoryNotEmpty when the directory was preserved for that reason.

func (*MetadataService) DeleteFileMetadata 

func (ms *MetadataService) DeleteFileMetadata(ctx context.Context, virtualPath string) error

DeleteFileMetadata deletes a metadata file, its .id sidecar, and — when this was the last metadata file referencing it — the shared .nzbz store behind it.

The store is a per-release artifact shared by every file of a multi-file import, so its lifetime is governed exclusively by the reference count. Nothing here acts on SourceNzbPath: for v3 metadata it aliases StoreRef, and removing it directly destroyed the store out from under sibling files (issue #858).

func (*MetadataService) DirectoryExists 

func (ms *MetadataService) DirectoryExists(virtualPath string) bool

DirectoryExists checks if a metadata directory exists

func (*MetadataService) DirectoryModTime 

func (ms *MetadataService) DirectoryModTime(virtualPath string) time.Time

DirectoryModTime returns the on-disk mtime of the metadata directory backing the given virtual directory, or a zero time.Time if it does not exist or is not a directory (letting callers fall back to a static epoch).

This is a single stat: the kernel already advances a directory's mtime when children are created or removed, so there is no need to scan children. It stays stable across restart health sweeps because UpdateFileStatus skips the write entirely when the status is unchanged, and advances on real imports and deletions, which do mutate directory entries.

func (*MetadataService) FileExists 

func (ms *MetadataService) FileExists(virtualPath string) bool

FileExists checks if a metadata file exists for the given virtual path

func (*MetadataService) GetMetadataDirectoryPath 

func (ms *MetadataService) GetMetadataDirectoryPath(virtualPath string) string

GetMetadataDirectoryPath returns the filesystem path for a metadata directory

func (*MetadataService) GetMetadataFilePath 

func (ms *MetadataService) GetMetadataFilePath(virtualPath string) string

GetMetadataFilePath returns the filesystem path for a metadata file

func (*MetadataService) IncStoreRef 

func (ms *MetadataService) IncStoreRef(ctx context.Context, storePath string)

IncStoreRef increments the reference count for a v3 store file. No-op when no StoreRefCounter is configured. Refcounting is best-effort: a DB failure is logged but does not fail the caller. A missed increment means a later DecStoreRef may drive the count to 0 prematurely, but store-file deletion is tolerant of the file already being absent.

func (*MetadataService) ListDirectory 

func (ms *MetadataService) ListDirectory(virtualPath string) ([]string, error)

ListDirectory lists all metadata files in a directory

func (*MetadataService) ListDirectoryAll 

func (ms *MetadataService) ListDirectoryAll(virtualPath string) (dirs []fs.FileInfo, fileNames []string, err error)

ListDirectoryAll returns both subdirectory fs.FileInfo entries and virtual file names from a single os.ReadDir call. This is used by Readdir to avoid two separate directory reads.

func (*MetadataService) LoadGroupMetas 

func (ms *MetadataService) LoadGroupMetas(g LegacyGroup) (LegacyGroup, error)

LoadGroupMetas returns a copy of g with every LegacyMeta.Meta populated. Files that can no longer be read (deleted or migrated since the scan) are dropped, so a stale scan degrades to a smaller group rather than an error.

func (*MetadataService) MigrateGroup 

func (ms *MetadataService) MigrateGroup(ctx context.Context, g LegacyGroup, storeDir, defaultGroup string) (GroupResult, error)

MigrateGroup converts one release group to the v3 store-backed format.

Ordering is load-bearing: the .nzbz is written and read back before any meta is repointed at it, so a meta never names a store that is missing or corrupt. Each meta is then rewritten atomically by WriteFileMetadataV3, which also increments the store's reference count exactly once, leaving the count equal to the number of migrated metas.

A per-file failure leaves that file as v1 and is recorded; the rest of the group still migrates. If every file fails, the freshly written store is removed so no orphan is left behind.

The group's metas are loaded here rather than at scan time: legacy metas carry their segments inline, so hydrating the whole library at once is the allocation pattern documented at service.go:405.

func (*MetadataService) MoveToCorrupted 

func (ms *MetadataService) MoveToCorrupted(ctx context.Context, virtualPath string) error

MoveToCorrupted moves a metadata file to a special corrupted directory for safety

func (*MetadataService) PruneCorrupted 

func (ms *MetadataService) PruneCorrupted(ctx context.Context, maxAge time.Duration) (int, error)

PruneCorrupted removes safety copies under corrupted_metadata whose .meta is older than maxAge, and reports how many were removed. A maxAge of zero or less prunes every copy. A missing corrupted_metadata directory is not an error.

func (*MetadataService) ReadFileMetadata 

func (ms *MetadataService) ReadFileMetadata(virtualPath string) (*metapb.FileMetadata, error)

ReadFileMetadata reads file metadata from disk. The full proto (including SegmentData and NestedSources) is returned to the caller but NOT cached — those slices dominate heap usage and must not be retained beyond the caller's handle. As a side effect, the lightweight projection is cached so subsequent Readdir/Stat calls are fast without a disk read.

func (*MetadataService) ReadFileMetadataLite 

func (ms *MetadataService) ReadFileMetadataLite(virtualPath string) (*FileMetadataLite, error)

ReadFileMetadataLite reads only the lightweight fields (size, modtime, status) needed for directory listings. On cache miss it reads at most liteScanBytes from the .meta file and scans the proto wire format for the three lite fields, never instantiating the full FileMetadata proto or its NestedSources/SegmentData slices. Falls back to a full read in the rare case the partial buffer doesn't cover the lite fields.

func (*MetadataService) RenameFileMetadata 

func (ms *MetadataService) RenameFileMetadata(oldVirtualPath, newVirtualPath string) error

RenameFileMetadata atomically renames a metadata file (and its .id sidecar) from oldVirtualPath to newVirtualPath. Uses os.Rename for atomicity on the same filesystem, falling back to read-write-delete for cross-device moves.

func (*MetadataService) ScanLegacyMetas 

func (ms *MetadataService) ScanLegacyMetas() ([]LegacyGroup, error)

ScanLegacyMetas walks the metadata root and returns every legacy (non-v3) meta, grouped by release. The group key is source_nzb_path when set, and the meta's parent directory otherwise. Unreadable files are skipped rather than failing the whole scan: a migration should convert what it can.

Each meta is read to obtain its group key and then released, so the walk retains only paths and sizes — peak memory is one meta, not the library. Call LoadGroupMetas to hydrate a single group before migrating it.

func (*MetadataService) SetStoreRefCounter 

func (ms *MetadataService) SetStoreRefCounter(c StoreRefCounter)

SetStoreRefCounter wires in a StoreRefCounter so that reference counts on shared NzbStore files are maintained when metadata is deleted or created.

func (*MetadataService) Store 

func (ms *MetadataService) Store() *StoreService

Store returns the StoreService used by this MetadataService.

func (*MetadataService) UpdateFileMetadata 

func (ms *MetadataService) UpdateFileMetadata(virtualPath string, updateFunc func(*metapb.FileMetadata)) error

UpdateFileMetadata applies updateFunc to existing metadata and writes it back.

IMPORTANT — non-content RMW only: this helper is reserved for mutations that do NOT change the file's logical content (e.g. health status, known-holes). ModifiedAt is always unconditionally restored to its pre-call value, so any attempt to bump it inside updateFunc will be silently discarded.

For paths that legitimately need to set a new ModifiedAt (initial import, content replacement), call WriteFileMetadata or CreateFileMetadata directly with the correct timestamp already set on the metadata struct.

This is not a cheap call: it unmarshals the full proto and rewrites the .meta via temp file + rename, which also bumps the parent directory's mtime. Callers that may be asked to apply a no-op should guard against it first — see UpdateFileStatus.

func (*MetadataService) UpdateFileStatus 

func (ms *MetadataService) UpdateFileStatus(virtualPath string, status metapb.FileStatus) error

UpdateFileStatus updates the status of a file in metadata.

Re-asserting the status a file already has is a no-op and is skipped without touching the disk. This matters at startup: the health worker emits EventTypeFileHealthy for every known-healthy file, and the full read-modify- write path unmarshals the entire proto (megabytes once SegmentData is inline) and rewrites the .meta via temp file + rename for each one. The lite projection carries Status, is served from an LRU, and on a miss reads only liteScanBytes without ever instantiating the full proto — so the unchanged case costs approximately nothing.

Skipping the write also leaves the parent directory's mtime untouched, which is what keeps DirectoryModTime stable across restarts.

func (*MetadataService) WriteFileMetadata 

func (ms *MetadataService) WriteFileMetadata(virtualPath string, metadata *metapb.FileMetadata) error

WriteFileMetadata writes file metadata to disk

func (*MetadataService) WriteFileMetadataAuto 

func (ms *MetadataService) WriteFileMetadataAuto(ctx context.Context, virtualPath string, metadata *metapb.FileMetadata, index map[string]int64, storeRef string) error

WriteFileMetadataAuto writes v3 store-backed metadata when storeRef is set, falling back to the v1 inline format if the v3 conversion fails (so a store/index problem on one file never blocks the import). With an empty storeRef it writes v1. This is the single entry point import processors should use.

func (*MetadataService) WriteFileMetadataV3 

func (ms *MetadataService) WriteFileMetadataV3(ctx context.Context, virtualPath string, metadata *metapb.FileMetadata, index map[string]int64, storeRef string) error

WriteFileMetadataV3 writes metadata directly in the v3 store-backed format: it converts the inline SegmentData of the main file, PAR2 files, and nested sources into SegmentRefs/SegmentRuns against the release's flat segment index, points the meta at storeRef, and increments the store reference count exactly once.

The conversion runs on a clone so a failure (e.g. a segment id missing from the index) leaves the caller's in-memory meta untouched, letting the caller fall back to a v1 WriteFileMetadata. Freshly-built archive metas still carry the SharedOuterSources dedup, so it is expanded first (mirroring the read path) before each nested source is converted independently and the dedup dissolved.

type MigrationProgress 

type MigrationProgress struct {
	TotalGroups     int       `json:"total_groups"`
	ProcessedGroups int       `json:"processed_groups"`
	TotalFiles      int       `json:"total_files"`
	ProcessedFiles  int       `json:"processed_files"`
	CurrentRelease  string    `json:"current_release"`
	StartTime       time.Time `json:"start_time"`
}

MigrationProgress is the live progress of a run.

type MigrationResult 

type MigrationResult struct {
	DryRun            bool          `json:"dry_run"`
	Groups            int           `json:"groups"`
	FaithfulGroups    int           `json:"faithful_groups"`
	SynthesizedGroups int           `json:"synthesized_groups"`
	FilesMigrated     int           `json:"files_migrated"`
	FilesFailed       int           `json:"files_failed"`
	BytesBefore       int64         `json:"bytes_before"`
	BytesAfter        int64         `json:"bytes_after"`
	BytesSaved        int64         `json:"bytes_saved"`
	Failures          []string      `json:"failures,omitempty"`
	Cancelled         bool          `json:"cancelled"`
	Duration          time.Duration `json:"duration"`
	CompletedAt       time.Time     `json:"completed_at"`
}

MigrationResult summarises a completed run (real or dry).

type MigrationStatus 

type MigrationStatus struct {
	IsRunning    bool               `json:"is_running"`
	LegacyFiles  int                `json:"legacy_files"`
	LegacyGroups int                `json:"legacy_groups"`
	Progress     *MigrationProgress `json:"progress,omitempty"`
	LastResult   *MigrationResult   `json:"last_result,omitempty"`
	LastDryRun   *MigrationResult   `json:"last_dry_run,omitempty"`
}

MigrationStatus is what the API reports.

type MigrationWorker 

type MigrationWorker struct {
	// contains filtered or unexported fields
}

MigrationWorker converts legacy inline-segment metadata to the v3 store-backed format. It is manually triggered; nothing runs on startup.

func NewMigrationWorker 

func NewMigrationWorker(ms *MetadataService, configGetter config.ConfigGetter) *MigrationWorker

NewMigrationWorker creates a migration worker. It does not start anything.

func (*MigrationWorker) Cancel 

func (w *MigrationWorker) Cancel()

Cancel stops an in-flight run after the current file.

func (*MigrationWorker) DryRun 

func (w *MigrationWorker) DryRun(ctx context.Context) (*MigrationResult, error)

DryRun performs the real conversion against an isolated temporary metadata root, measures the result, and deletes it. Nothing in the library is touched and no store reference counts move (the temporary service has no counter).

func (*MigrationWorker) GetStatus 

func (w *MigrationWorker) GetStatus() MigrationStatus

GetStatus reports whether a run is active, its progress, and the last results. The legacy counts come from the live progress when running and are otherwise left at zero; callers wanting a fresh count run a dry run.

func (*MigrationWorker) Start 

func (w *MigrationWorker) Start(ctx context.Context) error

Start launches a migration in the background and returns immediately. The run is detached from the caller's context so cancelling an HTTP request does not abort a migration; use Cancel for that.

type StoreRefCounter 

type StoreRefCounter interface {
	IncStoreRef(ctx context.Context, storePath string) error
	// DecStoreRef decrements the count and reports whether the store was tracked
	// at all. A count of 0 is only meaningful as "no references remain" when the
	// second value is true; an untracked store has an unknown, not zero, ref count.
	DecStoreRef(ctx context.Context, storePath string) (int64, bool, error)
}

StoreRefCounter tracks reference counts for shared NzbStore files. Implementations are provided by the database layer; the nil value is always safe to use.

type StoreService 

type StoreService struct {
	// contains filtered or unexported fields
}

StoreService reads/writes per-release NzbStore files (zstd proto) and caches decompressed stores keyed by store ref (path).

The cache is bounded by estimated bytes, not by entry count: stores vary in size by orders of magnitude (a single-file release versus a full season), so an entry-count bound places no bound at all on retained heap. cache is a non-thread-safe simplelru guarded by mu, which lets the eviction callback maintain the byte accounting without re-entering the lock.

func NewStoreService 

func NewStoreService(rootPath string) *StoreService

NewStoreService creates a StoreService rooted at rootPath with an LRU cache bounded by defaultStoreCacheBytes.

func (*StoreService) ReadStore 

func (ss *StoreService) ReadStore(ref string) (*metapb.NzbStore, error)

ReadStore reads and decompresses a store, caching the result.

func (*StoreService) ReadStoreUncached 

func (ss *StoreService) ReadStoreUncached(ref string) (*metapb.NzbStore, error)

ReadStoreUncached reads and decompresses a store straight from disk, ignoring the cache and without populating it.

WriteStore adds the written store to the cache, and ReadStore consults the cache first, so a read-back immediately after a write is served from memory and proves nothing about what actually landed on disk. Post-write integrity checks must use this instead.

func (*StoreService) RegenerateNZB 

func (ss *StoreService) RegenerateNZB(storePath string) ([]byte, error)

RegenerateNZB reads the store at storePath and returns NZB XML bytes. Returns (nil, nil) if the store does not exist.

func (*StoreService) WriteStore 

func (ss *StoreService) WriteStore(ref string, store *metapb.NzbStore) error

WriteStore writes zstd(proto) to ref atomically and refreshes the cache.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.