manager

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var FeedRefreshRetention = 20

FeedRefreshRetention bounds how many recent refresh runs (and their per-feed results) the monitor keeps. 0 disables pruning. Mirrors services.LLMMonitorRetention.

Functions

func FeedIDForURL added in v0.4.0 

func FeedIDForURL(rawURL string) (string, error)

FeedIDForURL returns the catalog feed id for a URL (the same id RegisterFeed and ApplyFeeds use), so callers outside this package can map a feed URL to its row.

func InitFeedManager 

func InitFeedManager(db store.Store)

func LoadProfilesFile added in v0.4.0 

func LoadProfilesFile(path string) (*models.ProfilesFile, error)

LoadProfilesFile parses a profiles.yml catalog from disk.

Types

type ApplyResult 

type ApplyResult struct {
	Created  []string
	Updated  []string
	Disabled []string
}

ApplyResult reports the outcome of an ApplyFeeds reconciliation. Each list holds human-readable feed labels ("title — url"), not ids.

type DeleteResult 

type DeleteResult struct {
	Deleted  []string
	NotFound []string
}

DeleteResult reports the outcome of a DeleteFeeds call. Deleted holds human-readable feed labels; NotFound holds the requested ids with no match.

type FeedManager 

type FeedManager struct {
	// contains filtered or unexported fields
}

FeedManager manages feeds and articles 

var (
	Manager *FeedManager
)

func NewFeedManager 

func NewFeedManager(db store.Store) *FeedManager

NewFeedManager creates a new FeedManager instance

func (*FeedManager) AllTopics added in v0.4.0 

func (m *FeedManager) AllTopics() ([]string, error)

AllTopics returns the distinct topics in use across all configured feeds.

func (*FeedManager) ApplyFeeds 

func (m *FeedManager) ApplyFeeds(configs []models.FeedConfig, defaults *models.Selectors, dryRun bool) (ApplyResult, error)

ApplyFeeds reconciles the stored feeds against the desired set: feeds in configs are created or updated, feeds with enabled:false in configs are disabled, and feeds absent from configs that are still enabled get disabled (articles preserved). When dryRun is true the plan is computed but nothing is written.

func (*FeedManager) ApplyProfiles added in v0.4.0 

func (m *FeedManager) ApplyProfiles(file *models.ProfilesFile) (ProfileApplyResult, error)

ApplyProfiles reconciles the stored profiles against profiles.yml: each entry is upserted and its feed pool (referenced by URL) is set to exactly the resolved feed ids. Feeds are resolved the same way the feed catalog stores them (by domain id), so a profile picks up whichever feed serves that URL. Profiles absent from the file are left untouched (the default profile, in particular, is never removed here).

func (*FeedManager) ArticleMainText added in v0.4.0 

func (m *FeedManager) ArticleMainText(rawURL, mode string, headers map[string]string) (string, error)

ArticleMainText scrapes a page in the given mode and returns the plain text of its most likely article-body element (the top SuggestSelectors candidate). It backs the feed-content check, which matches a feed entry against the real page to decide whether the body is already fully in the feed. Read-only; stores nothing.

func (*FeedManager) DeleteFeeds 

func (m *FeedManager) DeleteFeeds(feedIds []string, dryRun bool) (DeleteResult, error)

DeleteFeeds removes the given feeds (by id), cascading article deletion. When dryRun is true the targets are reported but nothing is deleted.

func (*FeedManager) DiagnoseFeed added in v0.2.0 

func (m *FeedManager) DiagnoseFeed(id string) (models.FeedDiagnosis, error)

DiagnoseFeed fetches a feed's raw HTTP response and returns a structured diagnosis of what came back. It is read-only: nothing is stored and the feed's last-fetch time is left untouched. This backs the `feeds diagnose` command.

func (*FeedManager) FetchFeed 

func (m *FeedManager) FetchFeed(feed models.Feed, from *time.Time, to *time.Time, overwrite bool, restore bool, lastN int) (models.FetchResult, error)

FetchFeed fetches a feed and stores its articles Optional time window filtering: from and to can be nil to disable filtering If provided, only articles with PublishedAt between from and to (inclusive) will be stored If overwrite is true, existing articles will be overwritten instead of skipped If restore is true, only existing articles with no content will be overwritten

func (*FeedManager) FinishRefreshRun added in v0.4.0 

func (m *FeedManager) FinishRefreshRun(runID string)

FinishRefreshRun stamps a run's completion and prunes old runs. Best-effort.

func (*FeedManager) GetFeed 

func (m *FeedManager) GetFeed(id string) (models.Feed, error)

GetFeed returns a feed by Id

func (*FeedManager) GetScraper 

func (m *FeedManager) GetScraper(feedType string) (scrapers.Scraper, error)

GetScraper returns the scraper for a feed type

func (*FeedManager) InspectArticle added in v0.2.0 

func (m *FeedManager) InspectArticle(rawURL, mode string, headers map[string]string, sel *models.Selectors, htmlLimit int) models.ArticleInspection

InspectArticle scrapes a single article URL in the requested mode and, when an article selector is supplied, returns the extracted content. It is read-only and stores nothing — it backs the feed-config builder's selector testing and mode probing. Modes, in escalating cost: "" / static, dynamic, full_browser.

func (*FeedManager) InspectFeedURL added in v0.2.0 

func (m *FeedManager) InspectFeedURL(feedURL string, headers map[string]string, maxLinks int) scrapers.FeedInspection

InspectFeedURL fetches and parses a feed URL (read-only), returning a diagnosis plus sample article links and the detected title. It is the pre-registration counterpart to DiagnoseFeed, working on a raw URL rather than a stored feed.

func (*FeedManager) ListFeeds 

func (m *FeedManager) ListFeeds() ([]models.Feed, error)

ListFeeds returns all registered feeds

func (*FeedManager) RecomputeProfileFeeds added in v0.4.0 

func (m *FeedManager) RecomputeProfileFeeds() error

RecomputeProfileFeeds re-resolves every profile's feed membership from its stored selection rule. Call it after the feed catalog changes (e.g. dlk feeds apply) so new feeds flow into the profiles whose topics they match.

func (*FeedManager) RecordRefresh added in v0.4.0 

func (m *FeedManager) RecordRefresh(runID string, feed models.Feed, fr models.FetchResult, ferr error, dur time.Duration)

RecordRefresh persists one feed's refresh outcome against a run. A blank runID (failed StartRefreshRun) is a no-op. Best-effort: store errors are logged.

func (*FeedManager) RefreshAllFeeds 

func (m *FeedManager) RefreshAllFeeds(wg *sync.WaitGroup)

RefreshAllFeeds refreshes all enabled feeds

func (*FeedManager) RefreshFeed 

func (m *FeedManager) RefreshFeed(id string) (models.FetchResult, error)

RefreshFeed refreshes a feed by Id

func (*FeedManager) RefreshFeedWithTimeWindow 

func (m *FeedManager) RefreshFeedWithTimeWindow(id string, from *time.Time, to *time.Time, overwrite bool, restore bool, lastN int) (models.FetchResult, error)

RefreshFeedWithTimeWindow refreshes a feed by Id with optional time window filtering from and to can be nil to disable filtering

func (*FeedManager) RegisterFeed 

func (m *FeedManager) RegisterFeed(config models.FeedConfig) error

RegisterFeed registers a feed

func (*FeedManager) RegisterScraper 

func (m *FeedManager) RegisterScraper(feedType string, scraper scrapers.Scraper)

RegisterScraper registers a scraper for a feed type

func (*FeedManager) RemoveFeed 

func (m *FeedManager) RemoveFeed(feedId string) error

RemoveFeed removes a feed and its data from the system

func (*FeedManager) SetSolimenAddr 

func (m *FeedManager) SetSolimenAddr(addr string)

SetSolimenAddr sets the solimen service address used for full_browser feeds.

func (*FeedManager) StartRefreshRun added in v0.4.0 

func (m *FeedManager) StartRefreshRun(trigger string) string

StartRefreshRun opens a refresh-monitoring run and returns its id. Recording is best-effort: a store failure is logged and an empty id returned, which the subsequent Record/Finish calls treat as a no-op.

func (*FeedManager) SuggestLinkSelectors added in v0.4.0 

func (m *FeedManager) SuggestLinkSelectors(indexURL, mode string, headers map[string]string, max int) ([]models.LinkListCandidate, error)

SuggestLinkSelectors scrapes an HTML index page in the given mode and returns ranked guesses at the selector scoping its repeating post links, plus the date_selector and url_filter implied by the repeating block. Relative hrefs are resolved against the index URL. Read-only; backs the html autoconfig pre-phase.

func (*FeedManager) SuggestSelectors added in v0.2.0 

func (m *FeedManager) SuggestSelectors(rawURL, mode string, headers map[string]string, max int) ([]models.SelectorCandidate, error)

SuggestSelectors scrapes an article page in the given mode and returns ranked candidate article-content selectors (by text length, penalized by link density). Read-only; backs the autobuild agent's selector discovery.

func (*FeedManager) UpdateFeedEnabled 

func (m *FeedManager) UpdateFeedEnabled(id string, enabled bool) error

UpdateFeedEnabled updates the enabled status of a feed

type ProfileApplyResult added in v0.4.0 

type ProfileApplyResult struct {
	Upserted []string // profile slugs created or updated
	Skipped  []string // profile slugs skipped (e.g. invalid)
}

ProfileApplyResult summarizes a profiles.yml apply.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.