scrape

package

v0.8.0 Latest Latest Go to latest Published: Jul 7, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/datallboy/gonzb

Links

Open Source Insights

Documentation ¶

Index ¶

type GroupStats
type NNTPAdapter
- func NewNNTPAdapter(p nntpClient) *NNTPAdapter
type Options
type OverviewHeader
type Service
- func NewService(repo repository, p provider, log logger, opts Options) *Service

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type GroupStats ¶

type GroupStats struct {
	Low        int64
	High       int64
	ProviderID string
}

type NNTPAdapter ¶

type NNTPAdapter struct {
	// contains filtered or unexported fields
}

func NewNNTPAdapter ¶

func NewNNTPAdapter(p nntpClient) *NNTPAdapter

func (*NNTPAdapter) GroupStats ¶

func (a *NNTPAdapter) GroupStats(ctx context.Context, group string) (GroupStats, error)

func (*NNTPAdapter) ID ¶

func (a *NNTPAdapter) ID() string

func (*NNTPAdapter) XOver ¶

func (a *NNTPAdapter) XOver(ctx context.Context, group string, from, to int64) ([]OverviewHeader, error)

func (*NNTPAdapter) XOverWithProvider ¶ added in v0.8.0

func (a *NNTPAdapter) XOverWithProvider(ctx context.Context, group string, from, to int64) ([]OverviewHeader, string, error)

type Options ¶

type Options struct {
	Newsgroups               []string
	BatchSize                int64
	Concurrency              int
	MaxBatches               int
	BackfillUntilDateByGroup map[string]time.Time
}

type OverviewHeader ¶

type OverviewHeader struct {
	ArticleNumber int64
	MessageID     string
	Subject       string
	Poster        string
	DateUTC       *time.Time
	Bytes         int64
	Lines         int
	Xref          string
	RawOverview   map[string]any
}

type Service ¶

type Service struct {
	// contains filtered or unexported fields
}

func NewService ¶

func NewService(repo repository, p provider, log logger, opts Options) *Service

func (*Service) RunBackfillOnce ¶

func (s *Service) RunBackfillOnce(ctx context.Context) error

backfill mode walks backward from the most recent known boundary.

func (*Service) RunBackfillOnceWithMetrics ¶ added in v0.7.0

func (s *Service) RunBackfillOnceWithMetrics(ctx context.Context) (map[string]any, error)

func (*Service) RunLatestOnce ¶

func (s *Service) RunLatestOnce(ctx context.Context) error

latest mode prioritizes the head of the group and continues forward.

func (*Service) RunLatestOnceWithMetrics ¶ added in v0.7.0

func (s *Service) RunLatestOnceWithMetrics(ctx context.Context) (map[string]any, error)

func (*Service) RunOnce ¶

func (s *Service) RunOnce(ctx context.Context) error

backward-compatible alias. Default scrape mode is latest.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL