Documentation
¶
Index ¶
- Constants
- func NewAdHocClient(ctx context.Context, provider nntppool.Provider, connections, inflight int) (*nntppool.Client, error)
- func RegisterConfigHandlers(ctx context.Context, configManager *config.Manager, poolManager Manager)
- func StatManyTimeout(count, concurrency int, perItem time.Duration) time.Duration
- type HeadroomPolicy
- type ImportAdmission
- type ImportBudget
- func (b *ImportBudget) Acquire(ctx context.Context) (release func(), err error)
- func (b *ImportBudget) Capacity() int
- func (b *ImportBudget) Headroom() int
- func (b *ImportBudget) NotifyStreamChange()
- func (b *ImportBudget) SetCapacity(totalConns int)
- func (b *ImportBudget) SetHeadroom(n int)
- func (b *ImportBudget) SetStreamSource(src StreamActivitySource)
- func (b *ImportBudget) StreamsActive() bool
- type Manager
- type MetricsSnapshot
- type MetricsTracker
- func (mt *MetricsTracker) GetSnapshot() MetricsSnapshot
- func (mt *MetricsTracker) IncArticlesDownloaded()
- func (mt *MetricsTracker) IncArticlesPosted()
- func (mt *MetricsTracker) Reset(ctx context.Context, resetPeak bool, resetTotals bool) error
- func (mt *MetricsTracker) ResetProviderErrors(ctx context.Context) error
- func (mt *MetricsTracker) SetProviderIDs(mapping map[string]string)
- func (mt *MetricsTracker) Start(ctx context.Context)
- func (mt *MetricsTracker) Stop()
- func (mt *MetricsTracker) UpdateDownloadProgress(id string, bytesDownloaded int64)
- type NntpClient
- type ProviderQuotaSnapshot
- type SpeculativeBudget
- type StatsRepository
- type StreamActivitySource
Constants ¶
const MissingRateWarningThreshold = 10.0
MissingRateWarningThreshold is the missing articles per minute rate that triggers a warning.
Variables ¶
This section is empty.
Functions ¶
func NewAdHocClient ¶
func NewAdHocClient(ctx context.Context, provider nntppool.Provider, connections, inflight int) (*nntppool.Client, error)
NewAdHocClient creates a single-provider client for a bounded speed test. The caller owns the returned client and must Close it.
func RegisterConfigHandlers ¶
func RegisterConfigHandlers(ctx context.Context, configManager *config.Manager, poolManager Manager)
RegisterConfigHandlers registers handlers for pool-related configuration changes
func StatManyTimeout ¶
StatManyTimeout scales a per-item Stat timeout into an overall deadline for a StatMany batch of count IDs run at the given concurrency, preserving the same worst-case bound a per-item context.WithTimeout gave when every check ran as its own goroutine: ceil(count/concurrency) waves, each capped at perItem.
Types ¶
type HeadroomPolicy ¶
type HeadroomPolicy struct {
// MaxTotalCostPct is how much aggregate pool throughput may be given up to
// protect playback. Zero — the default — means "free only": stop at the
// knee and never trade throughput for latency, which is the safe default
// for a deployment whose link rate is unknown.
//
// Setting it non-zero buys the larger win. Measured at 400 MB/s: allowing a
// ~9% aggregate cost took stream p50 from 190ms to 146ms and p99 from 245ms
// to 167ms — playback at 95% of its completely-unloaded speed while a full
// import ran.
MaxTotalCostPct int
}
HeadroomPolicy decides when the controller stops climbing.
type ImportAdmission ¶
type ImportAdmission struct {
// contains filtered or unexported fields
}
ImportAdmission is a counting semaphore that gates how many NZB imports may run concurrently end-to-end. A cap of 0 means "unlimited" (the controller is a no-op), which is the default so deployments behave as before until the max_concurrent_imports config knob is set.
Connection-level balancing between imports and streams is handled separately by ImportBudget; this gate only bounds whole-import parallelism (CPU, disk, queue pressure).
func NewImportAdmission ¶
func NewImportAdmission() *ImportAdmission
NewImportAdmission constructs an admission controller with the cap disabled (0 = unlimited). Use SetCap to configure it.
func (*ImportAdmission) Acquire ¶
func (a *ImportAdmission) Acquire(ctx context.Context) (release func(), err error)
Acquire blocks until an admission slot is available or ctx is cancelled. The returned release function MUST be called exactly once when the import is done. When the cap is 0 the call is a fast-path no-op.
func (*ImportAdmission) SetCap ¶
func (a *ImportAdmission) SetCap(cap int)
SetCap updates the cap. Queued waiters are woken if the cap grew. A cap of 0 disables the gate (unlimited).
type ImportBudget ¶
type ImportBudget struct {
// contains filtered or unexported fields
}
ImportBudget bounds the total number of in-flight import segment (body) fetches pool-wide, across all concurrent imports. Its capacity tracks the pool's total connection count and automatically shrinks while streams are active:
effective cap = capacity − min(streamHeadroom × activeStreams, capacity−1)
so imports expand to the full pool when idle, yield headroom to streams under playback, and always keep at least 1 connection so a lone import can make progress. A capacity of 0 disables the budget (no-op), which keeps pool-less paths and test fakes deadlock-free.
func NewImportBudget ¶
func NewImportBudget() *ImportBudget
NewImportBudget constructs a budget with capacity 0 (disabled). Use SetCapacity and SetStreamSource to configure it.
func (*ImportBudget) Acquire ¶
func (b *ImportBudget) Acquire(ctx context.Context) (release func(), err error)
Acquire blocks until a connection token is available or ctx is cancelled. The returned release function MUST be called exactly once when the fetch is done. When the capacity is 0 the call is a fast-path no-op.
func (*ImportBudget) Capacity ¶
func (b *ImportBudget) Capacity() int
Capacity returns the configured total capacity (not the stream-adjusted effective cap). Useful for sizing worker pools.
func (*ImportBudget) Headroom ¶
func (b *ImportBudget) Headroom() int
Headroom returns the current per-stream connection reserve.
func (*ImportBudget) NotifyStreamChange ¶
func (b *ImportBudget) NotifyStreamChange()
NotifyStreamChange should be called when the stream count changes so the budget can wake or hold waiters according to the new effective cap.
func (*ImportBudget) SetCapacity ¶
func (b *ImportBudget) SetCapacity(totalConns int)
SetCapacity updates the total connection capacity (sum of provider connections). Queued waiters are woken if the effective cap grew; on shrink, in-flight fetches drain naturally.
func (*ImportBudget) SetHeadroom ¶
func (b *ImportBudget) SetHeadroom(n int)
SetHeadroom sets the per-stream connection reserve and wakes any waiters the change frees. Called by the adaptive controller.
func (*ImportBudget) SetStreamSource ¶
func (b *ImportBudget) SetStreamSource(src StreamActivitySource)
SetStreamSource wires the activity signal. nil sources are tolerated and pin the effective cap to the full capacity.
func (*ImportBudget) StreamsActive ¶
func (b *ImportBudget) StreamsActive() bool
StreamsActive reports whether the wired stream source currently has at least one active stream. False when no source is wired.
type Manager ¶
type Manager interface {
// GetPool returns the current connection pool or error if not available.
// The returned client exposes the narrow NntpClient surface so tests can
// substitute a fake (see internal/testsupport/fakepool). In production it
// is backed by *nntppool.Client.
GetPool() (NntpClient, error)
// SetProviders creates/recreates the pool with new providers
SetProviders(providers []nntppool.Provider) error
// ClearPool shuts down and removes the current pool
ClearPool() error
// HasPool returns true if a pool is currently available
HasPool() bool
// GetMetrics returns the current pool metrics with calculated speeds
GetMetrics() (MetricsSnapshot, error)
// ResetMetrics resets specific cumulative metrics
ResetMetrics(ctx context.Context, resetPeak bool, resetTotals bool) error
// ResetProviderErrors zeroes all per-provider error counts without
// affecting bytes downloaded, peak speed, or history.
ResetProviderErrors(ctx context.Context) error
// IncArticlesDownloaded increments the count of articles successfully downloaded
IncArticlesDownloaded()
// UpdateDownloadProgress updates the bytes downloaded for a specific stream
UpdateDownloadProgress(id string, bytesDownloaded int64)
// IncArticlesPosted increments the count of articles successfully posted
IncArticlesPosted()
// AddProvider adds a single provider to the running pool.
// If no pool exists, a new one is created with this provider.
AddProvider(provider nntppool.Provider) error
// RemoveProvider removes a provider by its stable configuration ID.
// If the last provider is removed, the pool is closed.
RemoveProvider(name string) error
// ResetProviderQuota resets the download quota counter for a provider,
// clearing its consumed-bytes counter and exceeded flag in-place.
ResetProviderQuota(ctx context.Context, poolName string) error
// SetProviderIDs sets a mapping between pool names and configuration IDs.
SetProviderIDs(mapping map[string]string)
// AcquireImportSlot blocks until an admission slot is available for an
// NZB import to start, or ctx is cancelled. The returned release function
// must be called exactly once when the import has finished (success or
// failure). When the admission cap is unconfigured (0) it is a no-op.
AcquireImportSlot(ctx context.Context) (release func(), err error)
// SetAdmissionCap configures the cap on concurrently running NZB imports.
// A cap of 0 means unlimited.
SetAdmissionCap(cap int)
// AcquireImportConnection blocks until the global import connection
// budget grants a token for one segment (body) fetch, or ctx is
// cancelled. The returned release function must be called exactly once
// when the fetch is done. No-op while the budget capacity is unset (0).
AcquireImportConnection(ctx context.Context) (release func(), err error)
// SetImportConnCapacity sets the import connection budget to the pool's
// total connection count (sum of provider max connections).
SetImportConnCapacity(total int)
// SetStreamHeadroom sets how many connections the import budget holds back
// per active stream. See config.Config.StreamHeadroomConnections.
SetStreamHeadroom(perStream int)
// SpeculativeBudget bounds read-ahead fetches across every stream; nil
// (as returned by test fakes) means unlimited.
SpeculativeBudget() *SpeculativeBudget
// ImportConnCapacity returns the current budget capacity snapshot,
// useful for sizing import worker pools.
ImportConnCapacity() int
// SetStreamSource wires the activity signal so the import connection
// budget can shrink while streams are active.
SetStreamSource(src StreamActivitySource)
// NotifyStreamChange must be called by the stream source whenever its
// active stream count changes, so the budget can re-evaluate.
NotifyStreamChange()
// StatSweepConcurrency returns how many STATs an availability sweep
// (import fast-fail, health check) should keep in flight. While streams
// are active it returns the caller's conservative bound — typically one
// connection's STAT pipeline depth, so sweeps never crowd playback — and
// on an idle pool it opens up to the pool's aggregate STAT pipeline
// capacity (nntppool's StatCapacity), where STAT's one-line replies make
// the extra width nearly free. Falls back to conservative when no pool
// exists or its capacity is below the conservative bound.
StatSweepConcurrency(conservative int) int
}
Manager provides centralized NNTP connection pool management.
func NewManager ¶
func NewManager(ctx context.Context, repo StatsRepository) Manager
NewManager creates a new pool manager
type MetricsSnapshot ¶
type MetricsSnapshot struct {
BytesDownloaded int64 `json:"bytes_downloaded"`
BytesUploaded int64 `json:"bytes_uploaded"`
ArticlesDownloaded int64 `json:"articles_downloaded"`
ArticlesPosted int64 `json:"articles_posted"`
TotalErrors int64 `json:"total_errors"`
ProviderErrors map[string]int64 `json:"provider_errors"`
ProviderBytes map[string]int64 `json:"provider_bytes"`
ProviderBytes24h map[string]int64 `json:"provider_bytes_24h"`
ProviderStartedAt map[string]time.Time `json:"provider_started_at"`
ProviderQuotas map[string]ProviderQuotaSnapshot `json:"provider_quotas,omitempty"`
DownloadSpeedBytesPerSec float64 `json:"download_speed_bytes_per_sec"`
MaxDownloadSpeedBytesPerSec float64 `json:"max_download_speed_bytes_per_sec"`
UploadSpeedBytesPerSec float64 `json:"upload_speed_bytes_per_sec"`
Timestamp time.Time `json:"timestamp"`
StartedAt time.Time `json:"started_at"`
ProviderMissingRates map[string]float64 `json:"provider_missing_rates"`
ProviderMissingWarning map[string]bool `json:"provider_missing_warning"`
ProviderSpeeds map[string]float64 `json:"provider_speeds"`
}
MetricsSnapshot represents pool metrics at a point in time with calculated values
type MetricsTracker ¶
type MetricsTracker struct {
// contains filtered or unexported fields
}
MetricsTracker tracks pool metrics over time and calculates rates
func NewMetricsTracker ¶
func NewMetricsTracker(pool *nntppool.Client, repo StatsRepository) *MetricsTracker
NewMetricsTracker creates a new metrics tracker
func (*MetricsTracker) GetSnapshot ¶
func (mt *MetricsTracker) GetSnapshot() MetricsSnapshot
GetSnapshot returns the current metrics with calculated speeds
func (*MetricsTracker) IncArticlesDownloaded ¶
func (mt *MetricsTracker) IncArticlesDownloaded()
IncArticlesDownloaded increments the count of articles successfully downloaded
func (*MetricsTracker) IncArticlesPosted ¶
func (mt *MetricsTracker) IncArticlesPosted()
IncArticlesPosted increments the count of articles successfully posted
func (*MetricsTracker) Reset ¶
Reset resets cumulative metrics both in memory and in the database based on flags
func (*MetricsTracker) ResetProviderErrors ¶
func (mt *MetricsTracker) ResetProviderErrors(ctx context.Context) error
ResetProviderErrors zeroes out all per-provider error counts by offsetting the live pool error counts. Bytes, speed, and history are untouched.
func (*MetricsTracker) SetProviderIDs ¶
func (mt *MetricsTracker) SetProviderIDs(mapping map[string]string)
SetProviderIDs sets the mapping from pool names to config IDs
func (*MetricsTracker) Start ¶
func (mt *MetricsTracker) Start(ctx context.Context)
Start begins collecting metrics samples
func (*MetricsTracker) Stop ¶
func (mt *MetricsTracker) Stop()
Stop stops collecting metrics samples
func (*MetricsTracker) UpdateDownloadProgress ¶
func (mt *MetricsTracker) UpdateDownloadProgress(id string, bytesDownloaded int64)
UpdateDownloadProgress updates the live bytes downloaded counter
type NntpClient ¶
type NntpClient interface {
// Fetch downloads and decodes an article body. Req.Lane selects the queue
// (normal for imports, priority for playback reads, background for PAR2
// repair), and Req.Writer streams the decoded bytes instead of buffering
// them. A streamed fetch that has delivered bytes is not failed over, so
// callers retrying it must supply a fresh writer.
Fetch(ctx context.Context, r nntppool.Req) (*nntppool.ArticleBody, error)
// FetchAsync is Fetch on its own goroutine; the returned channel yields
// exactly one BodyResult. The importer uses it to overlap a segment
// download with its own bookkeeping.
FetchAsync(ctx context.Context, r nntppool.Req) <-chan nntppool.BodyResult
// Exists checks whether an article is retrievable from at least one
// provider without downloading the body. Used by health checks, import
// validation, and the mid-stream miss re-check (which passes
// LanePriority, since a playback read is blocked on the answer and on the
// normal lane could spend its whole budget queued behind a large body).
Exists(ctx context.Context, r nntppool.Req) (*nntppool.StatResult, error)
// ExistsMany checks many articles concurrently, streaming a result per
// message-id as each completes. Used by health checks and fast-fail import
// validation to batch existence sweeps instead of issuing one Exists per
// segment.
ExistsMany(ctx context.Context, messageIDs []string, opts nntppool.ManyOptions) <-chan nntppool.ExistsResult
// Stats returns a snapshot of pool/provider statistics used by the metrics
// tracker and the system handlers.
Stats() nntppool.ClientStats
}
NntpClient is the narrow surface of the underlying nntppool.Client that the rest of AltMount calls through Manager.GetPool. Defining it here lets tests inject a deterministic fake (see internal/testsupport/fakepool) without standing up real NNTP connections, and pins exactly which operations the streaming, import, validation, and metrics paths depend on.
Implementations must be safe for concurrent use. The production implementation is *nntppool.Client; the contract below intentionally mirrors its signatures so the existing client satisfies the interface without an adapter.
Keep this interface small. Anything that needs a behavior not listed here should add the method explicitly so callers stay observable. Note that what used to be eight methods is five: nntppool v5 moved lane selection and buffered-vs-streamed delivery into fields on nntppool.Req, so a new combination of those needs no new method here.
type ProviderQuotaSnapshot ¶
type ProviderQuotaSnapshot struct {
QuotaBytes int64 `json:"quota_bytes"`
QuotaUsed int64 `json:"quota_used"`
QuotaResetAt time.Time `json:"quota_reset_at,omitempty"`
QuotaExceeded bool `json:"quota_exceeded"`
}
ProviderQuotaSnapshot holds the quota state for a single provider.
type SpeculativeBudget ¶
type SpeculativeBudget struct {
// contains filtered or unexported fields
}
SpeculativeBudget bounds read-ahead fetches across every stream on the pool. Without it each open handle schedules its own full prefetch window, so several players on one title flood the priority lane with speculative bodies and starve each other's demand reads. Demand fetches never take a slot; only fetches ahead of the read position do, and only when one is free right now.
func NewSpeculativeBudget ¶
func NewSpeculativeBudget() *SpeculativeBudget
func (*SpeculativeBudget) Capacity ¶
func (b *SpeculativeBudget) Capacity() int
Capacity returns the current slot count; 0 means unlimited.
func (*SpeculativeBudget) InFlight ¶
func (b *SpeculativeBudget) InFlight() int
InFlight returns how many speculative fetches currently hold a slot.
func (*SpeculativeBudget) SetCapacity ¶
func (b *SpeculativeBudget) SetCapacity(totalConns int)
SetCapacity derives the slot count from total provider connections: four bodies per connection, minus one so a demand read never finds every slot taken by read-ahead. Zero or negative disables the budget (unlimited).
func (*SpeculativeBudget) TryAcquire ¶
func (b *SpeculativeBudget) TryAcquire() (release func(), ok bool)
TryAcquire never blocks. ok=false means the caller should skip this fetch for now and try again when its reader advances. release must be called exactly once when ok is true; extra calls are ignored.
type StatsRepository ¶
type StatsRepository interface {
UpdateSystemStat(ctx context.Context, key string, value int64) error
BatchUpdateSystemStats(ctx context.Context, stats map[string]int64) error
MigrateSystemStats(ctx context.Context, stats map[string]int64, deleteKeys []string) error
GetSystemStats(ctx context.Context) (map[string]int64, error)
AddBytesDownloadedToDailyStat(ctx context.Context, bytes int64) error
AddProviderBytesToHourlyStat(ctx context.Context, providerID string, bytes int64) error
RecordProviderSpeedTest(ctx context.Context, providerID string, speedMbps float64) error
GetProviderHourlyStats(ctx context.Context, hours int) (map[string]int64, error)
ClearProviderHourlyStats(ctx context.Context) error
GetOldestStatDate(ctx context.Context) (time.Time, error)
GetOldestProviderStatDates(ctx context.Context) (map[string]time.Time, error)
}
StatsRepository defines the interface for persisting pool statistics
type StreamActivitySource ¶
type StreamActivitySource interface {
ActiveStreams() int
}
StreamActivitySource reports how many streams are currently active. Implemented by api.StreamTracker; kept here so the dependency flows api -> pool.