Documentation
¶
Index ¶
- func InsertSegmentGapPlaceholders(n *nzbparser.Nzb) int
- func SanitizeNzbFilenames(n *nzbparser.Nzb)
- type ExtractedFileInfo
- type FirstSegmentData
- type NzbType
- type ParseOptions
- type ParsedFile
- type ParsedNzb
- type Parser
- func (p *Parser) FindDeadArticlesPastEnd(ctx context.Context, files []nzbparser.NzbFile) map[int]int
- func (p *Parser) GetMetadata(nzbXML *nzbparser.Nzb) map[string]string
- func (p *Parser) ParseFile(ctx context.Context, r io.Reader, nzbPath string, ...) (*ParsedNzb, error)
- func (p *Parser) ParseNzb(ctx context.Context, n *nzbparser.Nzb, nzbPath string, ...) (*ParsedNzb, error)
- func (p *Parser) SetSegmentStore(resolve func() SegmentStore)
- func (p *Parser) TrimArticles(ctx context.Context, files []nzbparser.NzbFile, keep map[int]int)
- func (p *Parser) ValidateNzb(parsed *ParsedNzb) error
- func (p *Parser) WarmFirstSegments(ctx context.Context, files []nzbparser.NzbFile)
- type SegmentStore
- type StrmParser
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func InsertSegmentGapPlaceholders ¶
InsertSegmentGapPlaceholders fills holes in each file's segment numbering with placeholder segments (see holes.PlaceholderID). An NZB that omits an article leaves every later byte shifted by one segment; a placeholder of a typical segment's size keeps the offsets true, so archive headers still line up and the missing span is served as a hole instead of corrupting the rest of the file. The gap is known before any provider is asked, so the placeholders are never fetched or STAT-ed.
Gaps are detected from segment numbers (1-based, contiguous when complete) and from the total the subject declares when it exceeds the highest number present — unless the release's subject totals are evidently made up (see subjectTotalsTrusted). Returns the number of placeholders inserted.
func SanitizeNzbFilenames ¶
SanitizeNzbFilenames normalizes poster-controlled filenames in place at the nzbparser boundary so every consumer (the pre-parse fast-fail probe, the parser, persisted metadata, and serve-time volume following) sees a canonical name. Call it immediately after nzbparser.Parse and before any code reads NzbFile.Filename. The raw subject remains available on NzbFile.Subject.
Types ¶
type ExtractedFileInfo ¶
type FirstSegmentData ¶
type FirstSegmentData struct {
File *nzbparser.NzbFile // Reference to the NZB file (for groups, subject, metadata)
Headers nntppool.YEncMeta // yEnc headers (FileName, FileSize, PartSize)
RawBytes []byte // Up to 16KB of raw data for PAR2 detection (may be less if segment is smaller)
MissingFirstSegment bool // True if first segment download failed (article not found, etc.)
IsArticleNotFound bool // True only when 430 Not Found (permanent); false for timeouts/transient
SkippedFirstSegment bool // True when the fetch was intentionally skipped (clean-named multipart file); Headers/RawBytes are empty by design, not by failure
OriginalIndex int // Original position in the parsed NZB file list
// FirstArticleMissingID is set when the first article is gone but the yEnc
// header was recovered from a later one. Headers are valid (PartSize is the
// derived first-part size); RawBytes is empty, so magic-byte and PAR2
// Hash16k matching are unavailable for this file.
FirstArticleMissingID string
}
FirstSegmentData holds cached data from the first segment of an NZB file This avoids redundant fetching when both PAR2 extraction and file parsing need the same data
type ParseOptions ¶
type ParseOptions struct {
// BrokenFileIndexes contains the 0-based positions (in the Nzb.Files slice)
// of files whose sampled segments failed a pre-parse Stat check. Their first
// segments are short-circuited to MissingFirstSegment=true without a Body call.
BrokenFileIndexes map[int]struct{}
// KnownMissingSegmentIDs seeds notFoundIDs so yEnc normalisation and 16KB
// completion never re-issue Stat/Body calls for already-known-missing IDs.
KnownMissingSegmentIDs map[string]struct{}
}
ParseOptions carries pre-parse knowledge into ParseNzb, allowing the processor to skip Body fetches for files whose segments are already known to be missing (identified by a pre-parse Stat check).
type ParsedFile ¶
type ParsedFile struct {
Subject string
Filename string
OriginalFilename string // Recovered posted name before import/display renaming.
Size int64
Segments []*metapb.SegmentData
SegmentRefs []*metapb.SegmentRef // v3: refs into NzbStore (populated when Store != nil)
Groups []string
IsRarArchive bool
Is7zArchive bool
IsPar2Archive bool
Encryption metapb.Encryption // Encryption type (e.g., "rclone"), nil if not encrypted
Password string // Password from NZB meta, nil if not encrypted
Salt string // Salt from NZB meta, nil if not encrypted
ReleaseDate time.Time // Release date from the Usenet post
OriginalIndex int // Original position in the parsed NZB file list
NzbdavID string // Original ID from nzbdav (for backward compatibility)
AesKey []byte // AES encryption key (for nzbdav compatibility)
AesIv []byte // AES initialization vector (for nzbdav compatibility)
// FirstSegmentBytes holds the decoded leading bytes (≤16KB) of this file's first
// segment, captured when the parser fetched it during first-segment analysis. It
// lets the archive analysis phase (UsenetFileSystem) serve a volume's header read,
// which starts at offset 0, from memory instead of re-fetching a segment already
// pulled over the wire this import. Empty when the first segment was skipped/missing
// or for files built outside the parser — those paths fall through to the network.
// Transient (not persisted); valid only for the lifetime of the import.
FirstSegmentBytes []byte
}
ParsedFile represents a file extracted from the NZB
type ParsedNzb ¶
type ParsedNzb struct {
Path string
Filename string
TotalSize int64
Type NzbType
Files []ParsedFile
SegmentsCount int
ExtractedFiles []ExtractedFileInfo
Store *metapb.NzbStore // NzbStore for this release (built at parse time)
SegmentIndex map[string]int64 // message-id → flat store index
// DegradedFiles maps a parsed filename to the message-id of its dead first
// article, for files whose header was recovered from a later article. The
// processor merges it into the repair queue so the hole gets patched.
DegradedFiles map[string]string
// contains filtered or unexported fields
}
ParsedNzb contains the parsed NZB data and extracted metadata
func (*ParsedNzb) GetPassword ¶
GetPassword returns the password for this NZB
func (*ParsedNzb) SetPassword ¶
SetPassword sets the password for this NZB
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser handles NZB file parsing
func NewParser ¶
func NewParser(poolManager pool.Manager, getConfig config.ConfigGetter) *Parser
NewParser creates a new NZB parser
func (*Parser) FindDeadArticlesPastEnd ¶
func (p *Parser) FindDeadArticlesPastEnd(ctx context.Context, files []nzbparser.NzbFile) map[int]int
FindDeadArticlesPastEnd returns, by file index, how many articles to keep for each file with dead articles listed past its end. It only reads files, so it is safe alongside WarmFirstSegments; apply the result with TrimArticles once the warm-up has finished. Each file costs a first-article header fetch, so call it only for a release already known to have a missing article.
func (*Parser) GetMetadata ¶
GetMetadata extracts metadata from the NZB head section
func (*Parser) ParseFile ¶
func (p *Parser) ParseFile(ctx context.Context, r io.Reader, nzbPath string, progressTracker progress.ProgressTracker) (*ParsedNzb, error)
ParseFile parses an NZB file from a reader. progressTracker, if non-nil, receives incremental updates as first segments are fetched (the longest phase). It is safe to pass nil — updates are skipped.
func (*Parser) ParseNzb ¶
func (p *Parser) ParseNzb(ctx context.Context, n *nzbparser.Nzb, nzbPath string, progressTracker progress.ProgressTracker, opts ParseOptions) (*ParsedNzb, error)
ParseNzb processes an already-parsed *nzbparser.Nzb, performing all network operations (first-segment fetches, PAR2 extraction, yEnc normalisation). opts carries knowledge collected before the network phase, e.g. file indexes whose segments are already known to be missing from a pre-parse Stat check.
func (*Parser) SetSegmentStore ¶
func (p *Parser) SetSegmentStore(resolve func() SegmentStore)
SetSegmentStore lets first articles fetched at import be published to the streaming segment store, so the cold open that follows is a cache hit.
func (*Parser) TrimArticles ¶
TrimArticles applies FindDeadArticlesPastEnd's result to files.
func (*Parser) ValidateNzb ¶
ValidateNzb performs basic validation on the parsed NZB
func (*Parser) WarmFirstSegments ¶
WarmFirstSegments fetches the first article of every file that ParseNzb would fetch, into the head cache, and returns when they have all landed or ctx ends. It exists so the fetch can overlap the fast-fail probe instead of following it: on a healthy release the parse pass then finds every head already cached. Errors are not reported; the parse pass diagnoses them.
type SegmentStore ¶
type SegmentStore interface {
Get(messageID string) ([]byte, bool)
Put(messageID string, data []byte) error
}
SegmentStore is where decoded articles are kept for the streaming readers: warm-up writes the articles it fetches, and the PAR2 index read serves from it. Mirrors usenet.SegmentStore.
type StrmParser ¶
type StrmParser struct {
// contains filtered or unexported fields
}
StrmParser handles STRM file parsing containing NXG links
func (*StrmParser) ParseStrmFile ¶
ParseStrmFile parses a STRM file containing an NXG link
func (*StrmParser) ValidateStrmFile ¶
func (p *StrmParser) ValidateStrmFile(parsed *ParsedNzb) error
ValidateStrmFile performs basic validation on the parsed STRM file