Documentation
¶
Overview ¶
Package feedsync is the UT1 URL-category feed syncer: it periodically downloads the UT1 Capitole blacklist tarball (via the NethServer mirror), extracts the categories in the ingest map, and bulk-writes them into the community URL-category database (internal/catdb). Extracted from package main per ADR-0002; the feedSyncer singleton and the urlcategories startup slice stay in main (feedsync.go shim).
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MappedCategories ¶
func MappedCategories() []string
MappedCategories returns the URLCategory values the UT1 ingest map produces (with duplicates — multiple UT1 directories can map to one category). Consumers: the category-store seeding at startup and the admin UI's feed-backed badge enrichment.
func SyncFailures ¶
func SyncFailures() int64
SyncFailures returns the cumulative UT1 sync failure count.
Types ¶
type Health ¶ added in v1.0.220
type Health struct {
LastAttempt time.Time
LastSuccess time.Time
ConsecutiveFailures int64
TotalFailures int64
LastFailure string
SyncInterval time.Duration
Domains int64
}
Health is a consistent snapshot of the UT1 syncer's state for the metrics and diagnostics surfaces.
type Syncer ¶
type Syncer struct {
// contains filtered or unexported fields
}
Syncer manages periodic synchronisation of UT1 data into the community DB.
func New ¶
New creates a Syncer for the given DB. feedURL defaults to defaultUT1FeedURL when empty. syncInterval defaults to 24h when zero.
func (*Syncer) SeedStats ¶
SeedStats sets the last-sync timestamp and domain count directly. Test support for the metrics surface (production values are set by Sync).
func (*Syncer) Start ¶
Start launches the background sync goroutine. An immediate sync is performed on first start when the DB is empty.
The loop is a feedsched.Scheduler, not a bare 24-hour ticker. Two reasons, both reachable without any infrastructure fault:
- a failed round was not retried for a full DAY. The origin is a raw.githubusercontent.com mirror of a 50+ MB tarball; a rate-limit, a GitHub incident, or a proxy restart froze category coverage until the next day. On a node whose community DB is empty (first boot, or after the CHAOS-50 quarantine of a damaged store) that means a full day of Layer-1-only categorisation.
- a fixed ticker keeps a fleet in phase forever, and every Culvert deployment pulls that same 50+ MB object from that same host. A synchronised fleet is answered by rate-limiting — which produces exactly the failure the missing backoff then holds for 24 hours.
Source Files