vault

package
v0.3.17 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 21, 2026 License: GPL-3.0 Imports: 27 Imported by: 0

Documentation

Overview

Package vault keeps every release this server ever resolved, so an indexer is asked for a given release at most once in the server's life.

Payload: zstd-compressed binary records in append-only pack files under <cacheDir>/vault (on the NAS when the cache is shared). Index: small rows in the server database (store.Register("vault", n, ...)); no NZB XML or segment lists in the database. Design: thread-storage/thr_mja5k9vcku/nzb-vault-design.md (sections 1-10).

This file is the contract every 0.3.16 vault worker builds against. Change it only additively and tell the coordinator thread (thr_mja5k9vcku).

Index

Constants

View Source
const IndexerID = "vault"

IndexerID marks releases served from the vault (core.Release.IndexerID).

View Source
const LegacyAliasIndexer = "legacy:nzb"

LegacyAliasIndexer is the alias namespace of migrated pre-vault NZBs.

View Source
const UnreadableDir = ".unreadable"

UnreadableDir is where MigrateDir sets aside legacy files that are not usable NZBs. They are never deleted.

Variables

View Source
var (
	// ErrNotFound: no index row.
	ErrNotFound = errors.New("vault: not found")
	// ErrPayloadMissing: the index row exists but this node cannot read the
	// payload (pack lost, restored without packs, or held only by an
	// unreachable peer). Callers re-grab as if the vault had nothing.
	ErrPayloadMissing = errors.New("vault: payload missing")
)
View Source
var ErrNotImplemented = errors.New("vault: not implemented yet")

ErrNotImplemented is returned by placeholders during the build.

Functions

func Encode

func Encode(r *Record) ([]byte, error)

func ReleaseFor

func ReleaseFor(e Entry) core.Release

ReleaseFor turns an index row into a resolver candidate.

func VerifyRecord

func VerifyRecord(ctx context.Context, r *Record, v Verifier) (bool, error)

VerifyRecord samples the first, middle and last article of the largest file.

Types

type AliasResolver

type AliasResolver interface {
	KeysFor(ctx context.Context, releases []core.Release) (map[string]Key, error)
}

AliasResolver maps candidate releases to their vault keys in one batched query without grabbing NZBs or touching pack files.

type Capper

type Capper interface {
	SetCapBytes(n int64)
}

Capper changes the payload cap at run time (0 = no limit).

type Chooser

type Chooser interface {
	MarkChosen(ctx context.Context, key Key, profileFP string) error
}

Chooser records that a stored release won resolution for a profile (winner-only fingerprint update; called after newznab's Put and on alias hits).

type Entry

type Entry struct {
	CacheID   string // original playable release ID (the parent release ID for a pack)
	Key       Key
	Kind      Kind
	Name      string
	TitleKey  string // normalized title tokens
	TMDB      int
	Year      int
	Season    int
	Episode   int
	Size      int64
	Segments  int
	Source    string // indexer id, "import:sabnzbd", "rosetta", ...
	Private   bool   // from a private indexer: DB-equivalent handling
	Added     time.Time
	LastUsed  time.Time
	Uses      int
	Health    Health
	HealthAt  time.Time
	ProfileFP string // profile fingerprint when it was chosen
	Info      core.ReleaseInfo
}

Entry is one index row (no payload).

type Extent

type Extent struct {
	File   int   `json:"file"`
	Offset int64 `json:"offset"`
	Size   int64 `json:"size"`
}

Extent is a contiguous byte range of one posted file (index into Record.NZB.Files) that belongs to an archive member.

type FramePeer

type FramePeer func(context.Context, string, Key, []byte) ([]byte, error)

FramePeer exchanges sealed frames with a specific live member.

type Health

type Health int

Health of a stored release's articles.

const (
	HealthUnknown Health = iota
	HealthOK
	HealthDead
)

type Key

type Key string

Key is a release's cross-indexer content identity: W116's nzb.ContentID ("nzb:" + hex sha256 over the NZB's sorted, de-duplicated segment message-ids), or TorrentKey for torrent records.

func ContentKey

func ContentKey(doc *nzb.NZB) Key

ContentKey is the release's content identity (nzb.ContentID).

func KeyOf

func KeyOf(r core.Release) Key

KeyOf returns the vault key of a vault release, or "". Pack targets rewrite release IDs (core.TargetReleaseID); the untargeted pack ID is used when set.

func TorrentKey

func TorrentKey(infoHash string) Key

TorrentKey is the key of a torrent record.

type Kind

type Kind string

Kind of record.

const (
	KindUsenet  Kind = "usenet"
	KindTorrent Kind = "torrent"
)

type Layout

type Layout struct {
	Container string         `json:"container"` // raw | rar | 7z | split
	Members   []Member       `json:"members"`
	Volumes   []VolumeLayout `json:"volumes,omitempty"`
	// FirstArticles are decoded first articles of archive volumes, keyed by
	// file index, capped by the vault (archive headers only).
	FirstArticles map[int][]byte `json:"-"`
}

Layout is learned by the stream engine on the first successful open so later opens skip archive discovery entirely.

type Lister

type Lister interface {
	List(ctx context.Context, q string, kind Kind, health *Health, limit int, cursor string) ([]Entry, error)
}

Lister pages the release index in ascending key order, after cursor.

type Member

type Member struct {
	// PrefixHash identifies the first 256 member bytes (or the whole member if shorter).
	PrefixHash string   `json:"prefixHash,omitempty"`
	Name       string   `json:"name"`
	Size       int64    `json:"size"`
	Stored     bool     `json:"stored"`
	Extents    []Extent `json:"extents"`
	Season     int      `json:"season,omitempty"` // 0 when not an episode
	Episode    int      `json:"episode,omitempty"`
}

Member is one playable file inside the release.

type Membership

type Membership interface {
	Kept(ctx context.Context, releases []core.Release) (map[string]bool, error)
}

Membership answers, in one batched query, which releases the vault holds, keyed by core.Release.ID. A vault release matches by key, others by the (IndexerID, ID) alias, and rows without an indexer id by the legacy alias.

type MigrationError

type MigrationError struct {
	Remaining int    // .nzb files still in the old directory
	Failed    int    // files that failed this pass (kept, retried next pass)
	File      string // the first failing file
	Err       error  // its error
}

MigrationError reports a migration pass that left NZBs behind.

func (*MigrationError) Error

func (e *MigrationError) Error() string

func (*MigrationError) Unwrap

func (e *MigrationError) Unwrap() error

type Migrator

type Migrator interface {
	MigrateDir(ctx context.Context, dir string) error
}

Migrator imports a directory of NZBs (Ether's pre-vault NZB cache or an export). Crash-safe and idempotent: import, verify count and bytes, then delete; never delete after a partial import. Files of the old cache are named <hex>.nzb, where hex is newznab's old disk key; each import records the alias (LegacyAliasIndexer, hex) so a release cached before the vault is found without a grab.

type Options

type Options struct {
	Dir      string  // <cacheDir>/vault, or the NAS cache dir's vault/ when shared
	DB       *sql.DB // server database (index only)
	Sealer   Sealer
	NodeID   string // this node (holders); "" on a standalone server
	Shared   bool   // Dir is on the shared NAS (one copy for the cluster)
	CapBytes int64  // payload cap; 0 = no limit (config Cache.LibraryBudgetBytes)
	Logger   *slog.Logger
	Now      func() time.Time
	Events   *events.Bus
}

Options open a vault.

type PutRequest

type PutRequest struct {
	Release core.Release // the resolved release (indexer id, guid, name, title ids)
	NZB     []byte       // Usenet
	Torrent *Torrent     // torrent (debrid)
	Source  string
	Private bool
	// ProfileFP fingerprints the quality profile plus the viewer's audio
	// and language rules at choice time (library-first rule).
	ProfileFP string
}

PutRequest stores a release.

func TorrentRecord

func TorrentRecord(r core.Release, files []TorrentFile, info []byte) PutRequest

TorrentRecord keeps durable torrent identity and file metadata only.

type Record

type Record struct {
	Key     Key
	Kind    Kind
	Name    string
	NZB     *nzb.NZB // KindUsenet
	XML     []byte   // canonical NZB XML regenerated on demand (Usenet)
	Torrent *Torrent // KindTorrent
	Layout  *Layout
	Par2    []byte // retained PAR2 summary (opaque to the vault)
	// Password is the archive password from the NZB head, unsealed for the
	// caller; stored sealed. Never logged, never returned by any API.
	Password string
}

Record is a decoded vault payload.

func Decode

func Decode(payload []byte) (*Record, error)

type ReplicaStore

type ReplicaStore interface {
	Store
	// ReadFrame returns the record's stored compressed frame.
	ReadFrame(ctx context.Context, key Key) ([]byte, error)
	// InstallFrame verifies the frame's CRC and content key, appends it to
	// this node's pack and records this node as a holder, atomically.
	InstallFrame(ctx context.Context, key Key, frame []byte) error
	Holders(ctx context.Context, key Key) ([]string, error)
	DropCache()
	Compact(ctx context.Context) error
}

ReplicaStore is what the cluster layer needs from a node-local vault.

type Sealer

type Sealer interface {
	Seal(name string, plain []byte) (string, error)
	Unseal(name, sealed string) ([]byte, error)
}

Sealer seals secrets at rest (the config sealer).

type SearchCounter

type SearchCounter interface {
	SearchAvoided(ctx context.Context, key Key) error
}

SearchCounter counts indexer searches the library made unnecessary (the resolver's fast path), separately from grabs avoided (Touch).

type Stats

type Stats struct {
	Releases        int
	Bytes           int64
	CapBytes        int64
	GrabsAvoided    int64
	SearchesAvoided int64
}

Stats for the console.

type Store

type Store interface {
	Put(ctx context.Context, r PutRequest) (Key, error)
	Get(ctx context.Context, key Key) (*Record, error)
	ByAlias(ctx context.Context, indexerID, releaseID string) (Key, bool, error)
	Search(ctx context.Context, req core.SearchRequest) ([]Entry, error)
	Entry(ctx context.Context, key Key) (Entry, error)
	SetLayout(ctx context.Context, key Key, l Layout) error
	MarkHealth(ctx context.Context, key Key, h Health) error
	Touch(ctx context.Context, key Key, avoided bool) error // a use; avoided = a grab/search was saved
	Purge(ctx context.Context, key Key) error
	Stats(ctx context.Context) (Stats, error)
}

Store is the vault.

func Open

func Open(ctx context.Context, o Options) (Store, error)

func WithCluster

func WithCluster(ctx context.Context, local ReplicaStore, node clusterbus.Node, peer FramePeer, log *slog.Logger) (Store, func())

type Torrent

type Torrent struct {
	InfoHash string        `json:"infoHash"`
	Magnet   string        `json:"magnet,omitempty"` // trackers only, no secrets
	Files    []TorrentFile `json:"files,omitempty"`
	Info     []byte        `json:"-"` // raw info dictionary when known
}

Torrent is the payload of a torrent (debrid) record. Never store add-on stream URLs: they expire and can embed the viewer's debrid key.

type TorrentFile

type TorrentFile struct {
	Path    string `json:"path"`
	Size    int64  `json:"size"`
	Season  int    `json:"season,omitempty"`
	Episode int    `json:"episode,omitempty"`
}

TorrentFile is one file of a torrent record.

type TrackStore

type TrackStore interface {
	Tracks(ctx context.Context, key Key, season, episode int) (core.MediaInfo, bool, error)
	SetTracks(ctx context.Context, key Key, season, episode int, info core.MediaInfo) error
}

TrackStore persists and reads probed container tracks for vaulted releases.

type Verifier

type Verifier interface {
	VerifyArticles(ctx context.Context, ids []string, groups []string) (bool, error)
}

Verifier proves a release's articles are still retrievable. Bound to W119's (*usenet.Fetcher).VerifyArticles, which distrusts lying STAT.

type VolumeLayout

type VolumeLayout struct {
	File     int   `json:"file"`
	Size     int64 `json:"size"`
	Verified bool  `json:"verified"`
	HeadSize int   `json:"headSize"`
}

VolumeLayout is one archive volume as the engine verified it, so a reopen keeps lazy verification and never trusts a truncated cached head.

Directories

Path Synopsis
Package importer reads owner-selected NZB backups without contacting an indexer.
Package importer reads owner-selected NZB backups without contacting an indexer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL