Affected by 
Documentation
¶
Overview ¶
Package rss implements the RSS/Atom data source connector for WeKnora.
It syncs articles from one or more RSS/Atom/JSON feeds into a WeKnora knowledge base. Each configured feed URL is treated as a selectable resource; each feed item becomes a knowledge entry whose body is the article's full text rendered as Markdown.
Capabilities:
- Private feeds: optional custom HTTP headers (e.g. "Authorization: Bearer …") are attached only to feed fetches, never to third-party article pages.
- Full text: when an item exposes a link, the article page is fetched and run through a readability extractor; the cleaned HTML is converted to Markdown. Feed-provided content (content:encoded / description) is used as a fallback.
- Incremental: feed-level signals skip full-text article fetches when a feed entry is unchanged; item fingerprints detect title, link and body changes. Article-only edits without feed updates may be missed until the feed entry changes. Deletions are NOT synced — feeds routinely drop old items.
All outbound requests go through the SSRF-safe HTTP client so a malicious feed cannot redirect WeKnora to internal services.
Index ¶
- type Config
- type Connector
- func (c *Connector) FetchAll(ctx context.Context, config *types.DataSourceConfig, resourceIDs []string) ([]types.FetchedItem, error)
- func (c *Connector) FetchIncremental(ctx context.Context, config *types.DataSourceConfig, cursor *types.SyncCursor) ([]types.FetchedItem, *types.SyncCursor, error)
- func (c *Connector) ListResources(ctx context.Context, config *types.DataSourceConfig, parentID string) ([]types.Resource, error)
- func (c *Connector) ResolveResourceAncestors(ctx context.Context, config *types.DataSourceConfig, resourceIDs []string) ([]string, error)
- func (c *Connector) Type() string
- func (c *Connector) Validate(ctx context.Context, config *types.DataSourceConfig) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// FeedURLs is a newline- or comma-separated list of feed URLs.
FeedURLs string `json:"feed_urls"`
// AuthHeaders is an optional newline-separated list of custom request
// headers in "Name: Value" form, applied only to feed fetches.
AuthHeaders string `json:"auth_headers,omitempty"`
}
Config holds RSS-specific configuration.
FeedURLs are stored in DataSourceConfig.Settings (non-secret, editable in the UI without replacing credentials). AuthHeaders live in Credentials because they may carry secrets that must be encrypted at rest. Credentials may still carry feed_urls for backward compatibility with older rows.
type Connector ¶
type Connector struct{}
Connector implements datasource.Connector for RSS/Atom/JSON feeds.
func (*Connector) FetchAll ¶
func (c *Connector) FetchAll( ctx context.Context, config *types.DataSourceConfig, resourceIDs []string, ) ([]types.FetchedItem, error)
FetchAll performs a full sync of the specified feeds (or all configured feeds when resourceIDs is empty).
func (*Connector) FetchIncremental ¶
func (c *Connector) FetchIncremental( ctx context.Context, config *types.DataSourceConfig, cursor *types.SyncCursor, ) ([]types.FetchedItem, *types.SyncCursor, error)
FetchIncremental returns only items whose item fingerprint changed since the prior cursor. Deletions are intentionally not emitted (feeds drop old items as a matter of course).
func (*Connector) ListResources ¶
func (c *Connector) ListResources( ctx context.Context, config *types.DataSourceConfig, parentID string, ) ([]types.Resource, error)
ListResources returns one resource per configured feed URL. The feed is fetched so the resource can carry its real title; a feed that fails to fetch still appears (named by URL) with an error note, so the user can deselect it instead of the whole listing failing.
func (*Connector) ResolveResourceAncestors ¶
func (c *Connector) ResolveResourceAncestors( ctx context.Context, config *types.DataSourceConfig, resourceIDs []string, ) ([]string, error)
ResolveResourceAncestors has nothing to do: feeds are a flat list with no nesting, so a selection has no ancestors to reveal.
Source Files