feedsync

package
v1.0.126 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 18, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package feedsync is the UT1 URL-category feed syncer: it periodically downloads the UT1 Capitole blacklist tarball (via the NethServer mirror), extracts the categories in the ingest map, and bulk-writes them into the community URL-category database (internal/catdb). Extracted from package main per ADR-0002; the feedSyncer singleton and the urlcategories startup slice stay in main (feedsync.go shim).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func MappedCategories

func MappedCategories() []string

MappedCategories returns the URLCategory values the UT1 ingest map produces (with duplicates — multiple UT1 directories can map to one category). Consumers: the category-store seeding at startup and the admin UI's feed-backed badge enrichment.

func SyncFailures

func SyncFailures() int64

SyncFailures returns the cumulative UT1 sync failure count.

Types

type Syncer

type Syncer struct {
	// contains filtered or unexported fields
}

Syncer manages periodic synchronisation of UT1 data into the community DB.

func New

func New(db *catdb.CommunityDB, feedURL string, syncInterval time.Duration) *Syncer

New creates a Syncer for the given DB. feedURL defaults to defaultUT1FeedURL when empty. syncInterval defaults to 24h when zero.

func (*Syncer) SeedStats

func (fs *Syncer) SeedStats(lastSync time.Time, totalDomains int64)

SeedStats sets the last-sync timestamp and domain count directly. Test support for the metrics surface (production values are set by Sync).

func (*Syncer) Start

func (fs *Syncer) Start(ctx context.Context)

Start launches the background sync goroutine. An immediate sync is performed on first start when the DB is empty.

func (*Syncer) Stats

func (fs *Syncer) Stats() (int64, time.Time, time.Duration)

Stats returns (totalDomains, lastSyncTime, syncInterval) for the metrics endpoint.

func (*Syncer) Sync

func (fs *Syncer) Sync()

Sync downloads the UT1 tarball, parses all mapped categories, and performs a bulk write into CommunityDB. The previous DB contents remain readable during the import; BadgerDB's WriteBatch overwrites keys as they arrive.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL