pool

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 18, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Index

Constants

View Source
const MissingRateWarningThreshold = 10.0

MissingRateWarningThreshold is the missing articles per minute rate that triggers a warning.

Variables

This section is empty.

Functions

func RegisterConfigHandlers

func RegisterConfigHandlers(ctx context.Context, configManager *config.Manager, poolManager Manager)

RegisterConfigHandlers registers handlers for pool-related configuration changes

func StatManyTimeout added in v0.3.0

func StatManyTimeout(count, concurrency int, perItem time.Duration) time.Duration

StatManyTimeout scales a per-item Stat timeout into an overall deadline for a StatMany batch of count IDs run at the given concurrency, preserving the same worst-case bound a per-item context.WithTimeout gave when every check ran as its own goroutine: ceil(count/concurrency) waves, each capped at perItem.

Types

type ImportAdmission added in v0.3.0

type ImportAdmission struct {
	// contains filtered or unexported fields
}

ImportAdmission is a counting semaphore that gates how many NZB imports may run concurrently end-to-end. A cap of 0 means "unlimited" (the controller is a no-op), which is the default so deployments behave as before until the max_concurrent_imports config knob is set.

Connection-level balancing between imports and streams is handled separately by ImportBudget; this gate only bounds whole-import parallelism (CPU, disk, queue pressure).

func NewImportAdmission added in v0.3.0

func NewImportAdmission() *ImportAdmission

NewImportAdmission constructs an admission controller with the cap disabled (0 = unlimited). Use SetCap to configure it.

func (*ImportAdmission) Acquire added in v0.3.0

func (a *ImportAdmission) Acquire(ctx context.Context) (release func(), err error)

Acquire blocks until an admission slot is available or ctx is cancelled. The returned release function MUST be called exactly once when the import is done. When the cap is 0 the call is a fast-path no-op.

func (*ImportAdmission) SetCap added in v0.3.0

func (a *ImportAdmission) SetCap(cap int)

SetCap updates the cap. Queued waiters are woken if the cap grew. A cap of 0 disables the gate (unlimited).

type ImportBudget added in v0.3.0

type ImportBudget struct {
	// contains filtered or unexported fields
}

ImportBudget bounds the total number of in-flight import segment (body) fetches pool-wide, across all concurrent imports. Its capacity tracks the pool's total connection count and automatically shrinks while streams are active:

effective cap = capacity − min(streamHeadroom × activeStreams, capacity−1)

so imports expand to the full pool when idle, yield headroom to streams under playback, and always keep at least 1 connection so a lone import can make progress. A capacity of 0 disables the budget (no-op), which keeps pool-less paths and test fakes deadlock-free.

func NewImportBudget added in v0.3.0

func NewImportBudget() *ImportBudget

NewImportBudget constructs a budget with capacity 0 (disabled). Use SetCapacity and SetStreamSource to configure it.

func (*ImportBudget) Acquire added in v0.3.0

func (b *ImportBudget) Acquire(ctx context.Context) (release func(), err error)

Acquire blocks until a connection token is available or ctx is cancelled. The returned release function MUST be called exactly once when the fetch is done. When the capacity is 0 the call is a fast-path no-op.

func (*ImportBudget) Capacity added in v0.3.0

func (b *ImportBudget) Capacity() int

Capacity returns the configured total capacity (not the stream-adjusted effective cap). Useful for sizing worker pools.

func (*ImportBudget) NotifyStreamChange added in v0.3.0

func (b *ImportBudget) NotifyStreamChange()

NotifyStreamChange should be called when the stream count changes so the budget can wake or hold waiters according to the new effective cap.

func (*ImportBudget) SetCapacity added in v0.3.0

func (b *ImportBudget) SetCapacity(totalConns int)

SetCapacity updates the total connection capacity (sum of provider connections). Queued waiters are woken if the effective cap grew; on shrink, in-flight fetches drain naturally.

func (*ImportBudget) SetStreamSource added in v0.3.0

func (b *ImportBudget) SetStreamSource(src StreamActivitySource)

SetStreamSource wires the activity signal. nil sources are tolerated and pin the effective cap to the full capacity.

type Manager

type Manager interface {
	// GetPool returns the current connection pool or error if not available.
	// The returned client exposes the narrow NntpClient surface so tests can
	// substitute a fake (see internal/testsupport/fakepool). In production it
	// is backed by *nntppool.Client.
	GetPool() (NntpClient, error)

	// SetProviders creates/recreates the pool with new providers
	SetProviders(providers []nntppool.Provider) error

	// ClearPool shuts down and removes the current pool
	ClearPool() error

	// HasPool returns true if a pool is currently available
	HasPool() bool

	// GetMetrics returns the current pool metrics with calculated speeds
	GetMetrics() (MetricsSnapshot, error)

	// ResetMetrics resets specific cumulative metrics
	ResetMetrics(ctx context.Context, resetPeak bool, resetTotals bool) error

	// ResetProviderErrors zeroes all per-provider error counts without
	// affecting bytes downloaded, peak speed, or history.
	ResetProviderErrors(ctx context.Context) error

	// IncArticlesDownloaded increments the count of articles successfully downloaded
	IncArticlesDownloaded()

	// UpdateDownloadProgress updates the bytes downloaded for a specific stream
	UpdateDownloadProgress(id string, bytesDownloaded int64)

	// IncArticlesPosted increments the count of articles successfully posted
	IncArticlesPosted()

	// AddProvider adds a single provider to the running pool.
	// If no pool exists, a new one is created with this provider.
	AddProvider(provider nntppool.Provider) error

	// RemoveProvider removes a provider by its nntppool name (host:port or host:port+username).
	// If the last provider is removed, the pool is closed.
	RemoveProvider(name string) error

	// ResetProviderQuota resets the download quota counter for a provider,
	// clearing its consumed-bytes counter and exceeded flag in-place.
	ResetProviderQuota(ctx context.Context, poolName string) error

	// SetProviderIDs sets a mapping between pool names and configuration IDs.
	SetProviderIDs(mapping map[string]string)

	// AcquireImportSlot blocks until an admission slot is available for an
	// NZB import to start, or ctx is cancelled. The returned release function
	// must be called exactly once when the import has finished (success or
	// failure). When the admission cap is unconfigured (0) it is a no-op.
	AcquireImportSlot(ctx context.Context) (release func(), err error)

	// SetAdmissionCap configures the cap on concurrently running NZB imports.
	// A cap of 0 means unlimited.
	SetAdmissionCap(cap int)

	// AcquireImportConnection blocks until the global import connection
	// budget grants a token for one segment (body) fetch, or ctx is
	// cancelled. The returned release function must be called exactly once
	// when the fetch is done. No-op while the budget capacity is unset (0).
	AcquireImportConnection(ctx context.Context) (release func(), err error)

	// SetImportConnCapacity sets the import connection budget to the pool's
	// total connection count (sum of provider max connections).
	SetImportConnCapacity(total int)

	// ImportConnCapacity returns the current budget capacity snapshot,
	// useful for sizing import worker pools.
	ImportConnCapacity() int

	// SetStreamSource wires the activity signal so the import connection
	// budget can shrink while streams are active.
	SetStreamSource(src StreamActivitySource)

	// NotifyStreamChange must be called by the stream source whenever its
	// active stream count changes, so the budget can re-evaluate.
	NotifyStreamChange()
}

Manager provides centralized NNTP connection pool management.

func NewManager

func NewManager(ctx context.Context, repo StatsRepository) Manager

NewManager creates a new pool manager

type MetricsSnapshot

type MetricsSnapshot struct {
	BytesDownloaded             int64                            `json:"bytes_downloaded"`
	BytesUploaded               int64                            `json:"bytes_uploaded"`
	ArticlesDownloaded          int64                            `json:"articles_downloaded"`
	ArticlesPosted              int64                            `json:"articles_posted"`
	TotalErrors                 int64                            `json:"total_errors"`
	ProviderErrors              map[string]int64                 `json:"provider_errors"`
	ProviderBytes               map[string]int64                 `json:"provider_bytes"`
	ProviderBytes24h            map[string]int64                 `json:"provider_bytes_24h"`
	ProviderStartedAt           map[string]time.Time             `json:"provider_started_at"`
	ProviderQuotas              map[string]ProviderQuotaSnapshot `json:"provider_quotas,omitempty"`
	DownloadSpeedBytesPerSec    float64                          `json:"download_speed_bytes_per_sec"`
	MaxDownloadSpeedBytesPerSec float64                          `json:"max_download_speed_bytes_per_sec"`
	UploadSpeedBytesPerSec      float64                          `json:"upload_speed_bytes_per_sec"`
	Timestamp                   time.Time                        `json:"timestamp"`
	StartedAt                   time.Time                        `json:"started_at"`
	ProviderMissingRates        map[string]float64               `json:"provider_missing_rates"`
	ProviderMissingWarning      map[string]bool                  `json:"provider_missing_warning"`
	ProviderSpeeds              map[string]float64               `json:"provider_speeds"`
}

MetricsSnapshot represents pool metrics at a point in time with calculated values

type MetricsTracker

type MetricsTracker struct {
	// contains filtered or unexported fields
}

MetricsTracker tracks pool metrics over time and calculates rates

func NewMetricsTracker

func NewMetricsTracker(pool *nntppool.Client, repo StatsRepository) *MetricsTracker

NewMetricsTracker creates a new metrics tracker

func (*MetricsTracker) GetSnapshot

func (mt *MetricsTracker) GetSnapshot() MetricsSnapshot

GetSnapshot returns the current metrics with calculated speeds

func (*MetricsTracker) IncArticlesDownloaded

func (mt *MetricsTracker) IncArticlesDownloaded()

IncArticlesDownloaded increments the count of articles successfully downloaded

func (*MetricsTracker) IncArticlesPosted

func (mt *MetricsTracker) IncArticlesPosted()

IncArticlesPosted increments the count of articles successfully posted

func (*MetricsTracker) Reset

func (mt *MetricsTracker) Reset(ctx context.Context, resetPeak bool, resetTotals bool) error

Reset resets cumulative metrics both in memory and in the database based on flags

func (*MetricsTracker) ResetProviderErrors added in v0.3.0

func (mt *MetricsTracker) ResetProviderErrors(ctx context.Context) error

ResetProviderErrors zeroes out all per-provider error counts by offsetting the live pool error counts. Bytes, speed, and history are untouched.

func (*MetricsTracker) SetProviderIDs added in v0.3.0

func (mt *MetricsTracker) SetProviderIDs(mapping map[string]string)

SetProviderIDs sets the mapping from pool names to config IDs

func (*MetricsTracker) Start

func (mt *MetricsTracker) Start(ctx context.Context)

Start begins collecting metrics samples

func (*MetricsTracker) Stop

func (mt *MetricsTracker) Stop()

Stop stops collecting metrics samples

func (*MetricsTracker) UpdateDownloadProgress

func (mt *MetricsTracker) UpdateDownloadProgress(id string, bytesDownloaded int64)

UpdateDownloadProgress updates the live bytes downloaded counter

type NntpClient added in v0.3.0

type NntpClient interface {
	// Body fetches an article body via the default (non-priority) lane.
	// Used by the importer to download NZB segments during scanning.
	Body(ctx context.Context, messageID string, onMeta ...func(nntppool.YEncMeta)) (*nntppool.ArticleBody, error)

	// BodyAsync fetches an article body asynchronously, streaming the decoded
	// payload to w. The returned channel yields exactly one BodyResult.
	BodyAsync(ctx context.Context, messageID string, w io.Writer, onMeta ...func(nntppool.YEncMeta)) <-chan nntppool.BodyResult

	// BodyPriority fetches an article body via the priority lane. Streaming
	// reads use this so live playback isn't queued behind a background import.
	BodyPriority(ctx context.Context, messageID string, onMeta ...func(nntppool.YEncMeta)) (*nntppool.ArticleBody, error)

	// Stat checks whether an article exists on at least one provider without
	// downloading the body. Used by health checks and validation.
	Stat(ctx context.Context, messageID string) (*nntppool.StatResult, error)

	// StatMany checks the existence of many articles concurrently, streaming a
	// result per message-id as each completes. Used by health checks and
	// fast-fail import validation to batch existence sweeps instead of
	// issuing one Stat per segment.
	StatMany(ctx context.Context, messageIDs []string, opts nntppool.StatManyOptions) <-chan nntppool.StatManyResult

	// Stats returns a snapshot of pool/provider statistics used by the metrics
	// tracker and the system handlers.
	Stats() nntppool.ClientStats
}

NntpClient is the narrow surface of the underlying nntppool.Client that the rest of AltMount calls through Manager.GetPool. Defining it here lets tests inject a deterministic fake (see internal/testsupport/fakepool) without standing up real NNTP connections, and pins exactly which operations the streaming, import, validation, and metrics paths depend on.

Implementations must be safe for concurrent use. The production implementation is *nntppool.Client; the contract below intentionally mirrors its signatures so the existing client satisfies the interface without an adapter.

Keep this interface small. Anything that needs a behavior not listed here should add the method explicitly so callers stay observable.

type ProviderQuotaSnapshot added in v0.3.0

type ProviderQuotaSnapshot struct {
	QuotaBytes    int64     `json:"quota_bytes"`
	QuotaUsed     int64     `json:"quota_used"`
	QuotaResetAt  time.Time `json:"quota_reset_at,omitempty"`
	QuotaExceeded bool      `json:"quota_exceeded"`
}

ProviderQuotaSnapshot holds the quota state for a single provider.

type StatsRepository

type StatsRepository interface {
	UpdateSystemStat(ctx context.Context, key string, value int64) error
	BatchUpdateSystemStats(ctx context.Context, stats map[string]int64) error
	GetSystemStats(ctx context.Context) (map[string]int64, error)
	AddBytesDownloadedToDailyStat(ctx context.Context, bytes int64) error
	AddProviderBytesToHourlyStat(ctx context.Context, providerID string, bytes int64) error
	RecordProviderSpeedTest(ctx context.Context, providerID string, speedMbps float64) error
	GetProviderHourlyStats(ctx context.Context, hours int) (map[string]int64, error)
	ClearProviderHourlyStats(ctx context.Context) error
	GetOldestStatDate(ctx context.Context) (time.Time, error)
	GetOldestProviderStatDates(ctx context.Context) (map[string]time.Time, error)
}

StatsRepository defines the interface for persisting pool statistics

type StreamActivitySource added in v0.3.0

type StreamActivitySource interface {
	ActiveStreams() int
}

StreamActivitySource reports how many streams are currently active. Implemented by api.StreamTracker; kept here so the dependency flows api -> pool.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL