downlink

Downlink

A feed aggregator and content-analysis platform for security news. Downlink collects RSS/Atom feeds, scrapes full article content, runs each article through an LLM for rubric-based scoring and summarization, and assembles ranked digests that it can publish to GitHub Pages and Discord.

Features

• Multi-provider LLMs Claude, ChatGPT Codex, Mistral, vLLM, Ollama, and llama.cpp (any OpenAI-compatible endpoint)
• Scoring articles are either vibe-scored by the LLM or scored across six dimensions (specificity, severity, breadth, novelty, actionability, credibility) instead of a single opaque number. See pkg/scoring.
• Flexible scraping plain RSS, dynamic scraping (Lightpanda), and full-browser scraping (Solimen) with per-feed CSS selectors, triggers, blacklists, and headers.
• Digest generation categorized, importance-ranked digests with deduplication.
• Glossary mode optional plain-language explanations and a jargon glossary per article, toggled on the digest page.
• Publishing push static digests to GitHub Pages and notify a Discord webhook.
• Interfaces gRPC API (default :50051), an Atom feed export (:65261), and the dlk command-line client.

Architecture

The server exposes gRPC services for articles, analysis, feeds, digests, queue management, config, and auth. The Atom feed server publishes analyzed articles at http://localhost:65261.

Install

With Go 1.25+:

go install github.com/ma111e/downlink/cmd/dlk@latest # CLI
go install github.com/ma111e/downlink/cmd/server@latest # server

Or build from source:

make all # builds server + cli
make server # build just the server
make cli # build just dlk

Or with Docker:

docker build -t downlink .
docker run --rm -p 50051:50051 -p 65261:65261 \
  -v "$PWD/config.json:/app/config.json" \
  -v "$PWD/feeds.yml:/app/feeds.yml" \
  downlink

A docker-compose.yml is provided that also wires up the optional Solimen full-browser scraper.

Quickstart

1. Build the binaries (see Install):

make all # produces ./server and ./dlk

2. Create your config and enable at least one LLM provider (analysis needs one; fetching feeds does not):

cp config.example.json config.json
$EDITOR config.json # enable a provider + fill in its api_key

3. Write a demo feeds.yml with a few public security feeds:

feeds:
  - url: https://cert.gov.ua/api/articles/rss
    title: Cert-UA
    type: rss
    enabled: true
    scraping: dynamic # "dynamic" (Lightpanda) or "full_browser" (Solimen); omit for static RSS
    scraper:
      article: div.article-item__content # CSS selector for the article body
  - url: https://www.bleepingcomputer.com/feed/
    title: Bleeping Computer
    type: rss
    enabled: true
    scraper:
      triggers:
          loaded:
              - article .article_section
    scraping: full_browser
    selectors:
      article: div.articleBody
  - url: https://feeds.feedburner.com/TheHackersNews
    title: The Hacker News
    type: rss
    enabled: true
    scraper:
      triggers:
          loaded:
              - div.main-box
    scraping: full_browser
    selectors:
      article: '#articlebody'
      cutoff: .stophere

The quickstart mixes scraping modes: scraping: dynamic needs Lightpanda and scraping: full_browser needs Solimen. The --auto-start-* flags in step 4 launch both in Docker. For a no-dependencies first run, drop the scraping/scraper/selectors keys to use plain RSS.

4. Start the server (in one terminal). The quickstart feeds use both scrapers, so start them too (requires Docker):

./server --auto-start-lightpanda --auto-start-solimen

5. Apply the feeds. dlk feed apply reconciles the database to match the file. Feeds in the file are created or updated, and feeds no longer listed are disabled (their articles are kept). Preview first with --dry-run:

./dlk feed apply -f feeds.yml --dry-run # show what would change
./dlk feed apply -f feeds.yml # apply it

6. Fetch and generate a digest. digest generate analyzes any not-yet-scored articles with your LLM provider, then assembles the ranked digest:

./dlk feed refresh all # pull the latest articles
./dlk digest generate  # analyze + assemble the ranked digest

7. View the result:

./dlk digest list # list generated digests
./dlk digest get  # pick one and view it (add --markdown for prose)

./dlk feed export -f feeds.yml does the reverse of step 5: it writes the feeds currently in the database back out to a YAML file.

Documentation

Full guides live in docs/: getting started, the configuration and CLI references, feeds and scraping, analysis and scoring, LLM providers, digests, publishing, and deployment.

Configuration

Downlink reads configuration from three sources. Copy the bundled examples and fill in your values:

cp config.example.json config.json # LLM providers, analysis, notifications
cp feeds.example.yml feeds.yml # feed sources and per-feed scraping rules
cp .env.example .env # runtime/env overrides

• config.json LLM provider definitions (name, type, model, base URL, API key), the active analysis provider and persona/writing style, and notification settings (Discord webhook, GitHub Pages repo/token/branch). See config.example.json.
• feeds.yml the list of feeds with per-feed scraping strategy, CSS selectors, triggers, blacklists, and custom HTTP headers. See feeds.example.yml.
• .env every server flag has a DOWNLINK_* env var equivalent (loaded automatically via Viper). See .env.example. Precedence: CLI flag -> env variables/.env -> config.json -> default.

Publishing digests

Downlink can publish generated digests to GitHub Pages and announce them on Discord. See docs/github-pages.md for the full setup, including --init-gh-pages and the required token scopes.

Deployment

A sample systemd unit is provided at etc/downlink.service for running the server under /opt/downlink.

License

MIT © 2026 ma111e

Contributions are not being accepted at this time.