Documentation
¶
Overview ¶
Package vault keeps every release this server ever resolved, so an indexer is asked for a given release at most once in the server's life.
Payload: zstd-compressed binary records in append-only pack files under <cacheDir>/vault (on the NAS when the cache is shared). Index: small rows in the server database (store.Register("vault", n, ...)); no NZB XML or segment lists in the database. Design: thread-storage/thr_mja5k9vcku/nzb-vault-design.md (sections 1-10).
This file is the contract every 0.3.16 vault worker builds against. Change it only additively and tell the coordinator thread (thr_mja5k9vcku).
Index ¶
- Constants
- Variables
- func Encode(r *Record) ([]byte, error)
- func ReleaseFor(e Entry) core.Release
- func VerifyRecord(ctx context.Context, r *Record, v Verifier) (bool, error)
- type AliasResolver
- type Capper
- type Chooser
- type Entry
- type Extent
- type FramePeer
- type Health
- type Key
- type Kind
- type Layout
- type Lister
- type Member
- type Membership
- type MigrationError
- type Migrator
- type Options
- type PutRequest
- type Record
- type ReplicaStore
- type Sealer
- type SearchCounter
- type Stats
- type Store
- type Torrent
- type TorrentFile
- type TrackStore
- type Verifier
- type VolumeLayout
Constants ¶
const IndexerID = "vault"
IndexerID marks releases served from the vault (core.Release.IndexerID).
const LegacyAliasIndexer = "legacy:nzb"
LegacyAliasIndexer is the alias namespace of migrated pre-vault NZBs.
const UnreadableDir = ".unreadable"
UnreadableDir is where MigrateDir sets aside legacy files that are not usable NZBs. They are never deleted.
Variables ¶
var ( // ErrNotFound: no index row. ErrNotFound = errors.New("vault: not found") // ErrPayloadMissing: the index row exists but this node cannot read the // payload (pack lost, restored without packs, or held only by an // unreachable peer). Callers re-grab as if the vault had nothing. ErrPayloadMissing = errors.New("vault: payload missing") )
var ErrNotImplemented = errors.New("vault: not implemented yet")
ErrNotImplemented is returned by placeholders during the build.
Functions ¶
func ReleaseFor ¶
ReleaseFor turns an index row into a resolver candidate.
Types ¶
type AliasResolver ¶
type AliasResolver interface {
KeysFor(ctx context.Context, releases []core.Release) (map[string]Key, error)
}
AliasResolver maps candidate releases to their vault keys in one batched query without grabbing NZBs or touching pack files.
type Capper ¶
type Capper interface {
SetCapBytes(n int64)
}
Capper changes the payload cap at run time (0 = no limit).
type Chooser ¶
Chooser records that a stored release won resolution for a profile (winner-only fingerprint update; called after newznab's Put and on alias hits).
type Entry ¶
type Entry struct {
CacheID string // original playable release ID (the parent release ID for a pack)
Key Key
Kind Kind
Name string
TitleKey string // normalized title tokens
TMDB int
Year int
Season int
Episode int
Size int64
Segments int
Source string // indexer id, "import:sabnzbd", "rosetta", ...
Private bool // from a private indexer: DB-equivalent handling
Added time.Time
LastUsed time.Time
Uses int
Health Health
HealthAt time.Time
ProfileFP string // profile fingerprint when it was chosen
Info core.ReleaseInfo
}
Entry is one index row (no payload).
type Extent ¶
Extent is a contiguous byte range of one posted file (index into Record.NZB.Files) that belongs to an archive member.
type Key ¶
type Key string
Key is a release's cross-indexer content identity: W116's nzb.ContentID ("nzb:" + hex sha256 over the NZB's sorted, de-duplicated segment message-ids), or TorrentKey for torrent records.
func ContentKey ¶
ContentKey is the release's content identity (nzb.ContentID).
type Layout ¶
type Layout struct {
Container string `json:"container"` // raw | rar | 7z | split
Members []Member `json:"members"`
Volumes []VolumeLayout `json:"volumes,omitempty"`
// FirstArticles are decoded first articles of archive volumes, keyed by
// file index, capped by the vault (archive headers only).
FirstArticles map[int][]byte `json:"-"`
}
Layout is learned by the stream engine on the first successful open so later opens skip archive discovery entirely.
type Lister ¶
type Lister interface {
List(ctx context.Context, q string, kind Kind, health *Health, limit int, cursor string) ([]Entry, error)
}
Lister pages the release index in ascending key order, after cursor.
type Member ¶
type Member struct {
// PrefixHash identifies the first 256 member bytes (or the whole member if shorter).
PrefixHash string `json:"prefixHash,omitempty"`
Name string `json:"name"`
Size int64 `json:"size"`
Stored bool `json:"stored"`
Extents []Extent `json:"extents"`
Season int `json:"season,omitempty"` // 0 when not an episode
Episode int `json:"episode,omitempty"`
}
Member is one playable file inside the release.
type Membership ¶
type Membership interface {
Kept(ctx context.Context, releases []core.Release) (map[string]bool, error)
}
Membership answers, in one batched query, which releases the vault holds, keyed by core.Release.ID. A vault release matches by key, others by the (IndexerID, ID) alias, and rows without an indexer id by the legacy alias.
type MigrationError ¶
type MigrationError struct {
Remaining int // .nzb files still in the old directory
Failed int // files that failed this pass (kept, retried next pass)
File string // the first failing file
Err error // its error
}
MigrationError reports a migration pass that left NZBs behind.
func (*MigrationError) Error ¶
func (e *MigrationError) Error() string
func (*MigrationError) Unwrap ¶
func (e *MigrationError) Unwrap() error
type Migrator ¶
Migrator imports a directory of NZBs (Ether's pre-vault NZB cache or an export). Crash-safe and idempotent: import, verify count and bytes, then delete; never delete after a partial import. Files of the old cache are named <hex>.nzb, where hex is newznab's old disk key; each import records the alias (LegacyAliasIndexer, hex) so a release cached before the vault is found without a grab.
type Options ¶
type Options struct {
Dir string // <cacheDir>/vault, or the NAS cache dir's vault/ when shared
DB *sql.DB // server database (index only)
Sealer Sealer
NodeID string // this node (holders); "" on a standalone server
CapBytes int64 // payload cap; 0 = no limit (config Cache.LibraryBudgetBytes)
Logger *slog.Logger
Now func() time.Time
Events *events.Bus
}
Options open a vault.
type PutRequest ¶
type PutRequest struct {
Release core.Release // the resolved release (indexer id, guid, name, title ids)
NZB []byte // Usenet
Torrent *Torrent // torrent (debrid)
Source string
Private bool
// ProfileFP fingerprints the quality profile plus the viewer's audio
// and language rules at choice time (library-first rule).
ProfileFP string
}
PutRequest stores a release.
func TorrentRecord ¶
func TorrentRecord(r core.Release, files []TorrentFile, info []byte) PutRequest
TorrentRecord keeps durable torrent identity and file metadata only.
type Record ¶
type Record struct {
Key Key
Kind Kind
Name string
NZB *nzb.NZB // KindUsenet
XML []byte // canonical NZB XML regenerated on demand (Usenet)
Torrent *Torrent // KindTorrent
Layout *Layout
Par2 []byte // retained PAR2 summary (opaque to the vault)
// Password is the archive password from the NZB head, unsealed for the
// caller; stored sealed. Never logged, never returned by any API.
Password string
}
Record is a decoded vault payload.
type ReplicaStore ¶
type ReplicaStore interface {
Store
// ReadFrame returns the record's stored compressed frame.
ReadFrame(ctx context.Context, key Key) ([]byte, error)
// InstallFrame verifies the frame's CRC and content key, appends it to
// this node's pack and records this node as a holder, atomically.
InstallFrame(ctx context.Context, key Key, frame []byte) error
Holders(ctx context.Context, key Key) ([]string, error)
DropCache()
Compact(ctx context.Context) error
}
ReplicaStore is what the cluster layer needs from a node-local vault.
type Sealer ¶
type Sealer interface {
Seal(name string, plain []byte) (string, error)
Unseal(name, sealed string) ([]byte, error)
}
Sealer seals secrets at rest (the config sealer).
type SearchCounter ¶
SearchCounter counts indexer searches the library made unnecessary (the resolver's fast path), separately from grabs avoided (Touch).
type Stats ¶
type Stats struct {
Releases int
Bytes int64
CapBytes int64
GrabsAvoided int64
SearchesAvoided int64
}
Stats for the console.
type Store ¶
type Store interface {
Put(ctx context.Context, r PutRequest) (Key, error)
Get(ctx context.Context, key Key) (*Record, error)
ByAlias(ctx context.Context, indexerID, releaseID string) (Key, bool, error)
Search(ctx context.Context, req core.SearchRequest) ([]Entry, error)
Entry(ctx context.Context, key Key) (Entry, error)
SetLayout(ctx context.Context, key Key, l Layout) error
MarkHealth(ctx context.Context, key Key, h Health) error
Touch(ctx context.Context, key Key, avoided bool) error // a use; avoided = a grab/search was saved
Purge(ctx context.Context, key Key) error
Stats(ctx context.Context) (Stats, error)
}
Store is the vault.
func WithCluster ¶
func WithCluster(ctx context.Context, local ReplicaStore, node clusterbus.Node, peer FramePeer, log *slog.Logger) (Store, func())
type Torrent ¶
type Torrent struct {
InfoHash string `json:"infoHash"`
Magnet string `json:"magnet,omitempty"` // trackers only, no secrets
Files []TorrentFile `json:"files,omitempty"`
Info []byte `json:"-"` // raw info dictionary when known
}
Torrent is the payload of a torrent (debrid) record. Never store add-on stream URLs: they expire and can embed the viewer's debrid key.
type TorrentFile ¶
type TorrentFile struct {
Path string `json:"path"`
Size int64 `json:"size"`
Season int `json:"season,omitempty"`
Episode int `json:"episode,omitempty"`
}
TorrentFile is one file of a torrent record.
type TrackStore ¶
type TrackStore interface {
Tracks(ctx context.Context, key Key, season, episode int) (core.MediaInfo, bool, error)
SetTracks(ctx context.Context, key Key, season, episode int, info core.MediaInfo) error
}
TrackStore persists and reads probed container tracks for vaulted releases.