archiver

package

v2.0.26 Latest Latest Go to latest Published: Jun 2, 2026 License: AGPL-3.0 Imports: 20 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/internetarchive/Zeno

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func GetClients() (clients []*warc.CustomHTTPClient)
func GetWARCWritingQueueSize() (total int)
func Start(inputChan, outputChan chan *models.Item) error
func Stop()
type WARCStats
- func GetStats() WARCStats

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	// ErrArchiverAlreadyInitialized is the error returned when the preprocess is already initialized
	ErrArchiverAlreadyInitialized = errors.New("archiver already initialized")
)

Functions ¶

func GetClients ¶

func GetClients() (clients []*warc.CustomHTTPClient)

func GetWARCWritingQueueSize ¶

func GetWARCWritingQueueSize() (total int)

This function is used in multiple places so it can't be replaced by GetStats()

func Start ¶

func Start(inputChan, outputChan chan *models.Item) error

Start initializes the internal archiver structure, start the WARC writer and start routines, should only be called once and returns an error if called more than once

func Stop ¶

func Stop()

Stop stops the archiver routines and the WARC writer

Types ¶

type WARCStats ¶

type WARCStats struct {
	WARCWritingQueueSize         int64
	WARCTotalBytesArchived       int64
	CDXDedupeTotalBytes          int64
	DoppelgangerDedupeTotalBytes int64
	LocalDedupeTotalBytes        int64
	CDXDedupeTotal               int64
	DoppelgangerDedupeTotal      int64
	LocalDedupeTotal             int64
}

func GetStats ¶

func GetStats() WARCStats

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
connutil
discard
discarder/akamai
discarder/cloudflare
discarder/contentlength
discarder/warcdiscardstatus
reasoncode
general
headless
ratelimiter

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL