vgi-feed

A Query.Farm VGI worker for DuckDB.

Parse RSS, Atom & JSON Feeds into Rows in DuckDB

vgi-feed · a Query.Farm VGI worker · powered by gofeed

A VGI worker, written in Go, that fetches and parses RSS, Atom, and JSON Feed documents into DuckDB/SQL rows. The feed format is auto-detected. Input may be an http(s) URL (fetched over HTTP) or a raw feed document supplied inline (RSS/Atom XML or JSON Feed text).

Built on the vgi-go SDK; it speaks the VGI protocol over stdio. Catalog name: feed. Universal feed parsing is provided by gofeed.

INSTALL vgi FROM community; LOAD vgi;

-- LOCATION is the path to the compiled worker binary.
ATTACH 'feed' AS feed (TYPE vgi, LOCATION '/path/to/vgi-feed-worker');

-- One row per feed item (input is a URL → fetched over HTTP).
SELECT seq, title, link, published, author, categories
FROM feed.feed_items('https://news.ycombinator.com/rss');

-- Feed-level metadata (one row); feed_type is 'rss' | 'atom' | 'json'.
SELECT title, feed_type, language, item_count
FROM feed.feed_info('https://news.ycombinator.com/rss');

-- Raw feed text input (not a URL) is parsed directly.
SELECT title FROM feed.feed_items('<?xml version="1.0"?><rss version="2.0">...</rss>');

-- Named options: HTTP timeout and an item cap.
SELECT * FROM feed.feed_items('https://example.com/feed.xml',
  timeout_ms := 5000, max_items := 50);

-- categories is a VARCHAR[] — UNNEST to one row per category.
SELECT title, UNNEST(categories) AS category
FROM feed.feed_items('https://example.com/feed.xml');

Functions

Both are table functions (each parses one feed and returns rows). The first argument is always input — either an http:///https:// URL to fetch, or a raw feed document (RSS/Atom XML or JSON Feed text).

Named options

URL vs. raw text

The input argument is classified by its leading character (after trimming whitespace and an optional UTF-8 BOM):

• starts with <, {, or [ → treated as a raw feed document and parsed directly (no network access);
• starts with http:// or https:// → fetched over HTTP and then parsed;
• anything else → a clear error.

Behaviour

• Format auto-detection — RSS, Atom, and JSON Feed are detected by gofeed; feed_info.feed_type reports which.
• Dates — published / updated come from the feed's parsed dates; a missing or unparseable date is emitted as a NULL TIMESTAMP.
• Categories — a non-nil VARCHAR[] (empty list when the item has none).
• Robustness — HTTP errors/timeouts, unreachable hosts, and malformed feeds become clear DuckDB errors (with a short body excerpt for non-2xx responses); the worker never crashes or hangs. Response bodies are bounded (64 MiB) and the item count is capped.

Building

make build        # builds vgi-feed-worker + mockserver
make test-unit    # Go unit/integration tests (in-process httptest feed server)
make test-sql     # haybarn SQL end-to-end against a local mock feed server
make test         # both

make test-sql needs haybarn-unittest on PATH:

uv tool install haybarn-unittest
export PATH="$HOME/.local/bin:$PATH"

Licensing

This worker is licensed MIT (see LICENSE).

Dependencies:

• gofeed — universal RSS/Atom/JSON feed parser, MIT License (Copyright (c) 2016 mmcdole; verified from its repository LICENSE).
• The VGI SDK stack — vgi-go (the worker SDK, MIT here), vgi-rpc-go, and arrow-go (Apache-2.0).

Authorship & License

Written by Query.Farm.

Copyright 2026 Query Farm LLC - https://query.farm