yarilo

yarilo

Production-grade IMAP / POP3 / LMTP / ManageSieve / Submission mail server with built-in full-text search, written in Go. Multi-binary architecture - each protocol component is a separate process. Kubernetes-native via Helm. Website: yarilomail.org · Docs: doc.yarilomail.org

Architecture

Yarilo is a multi-binary server. Each protocol and infrastructure role is a separate compiled binary - no mode flags, no combined processes. Deployment topology is configured purely through Helm values; the same binaries serve standalone and clustered installations.

Login pods - terminate TLS (SNI per-domain), authenticate via passdb, enforce per-user connection limits (warden), pass the raw fd to session pods via SCM_RIGHTS. Stateless; scale independently.

Session pods - full mail processing: IMAP / POP3 / LMTP / Sieve / Submission / ManageSieve. Each protocol scales as a separate StatefulSet so IMAP can scale independently of LMTP.

yarilo-auth - shared passdb chain (MySQL / Postgres / SQLite / passwd-file / static), auth-token cache, SASL dispatch, master protocol for userdb lookups.

yarilo-locks - cross-process write coordination. TCP mTLS :9104, Redis-backed state. All Kubernetes deployments (standalone and backend) use remote mode - embedded Unix-socket mode is reserved for unit tests and single-process CLI runs.

yarilo-warden - connection rate limiting and penalty tracking. Shared across all login pods.

Protocol support

Storage

All index mutations go through the cross-process mailbox lock (yarilo-locks). Sessions sharing a pod serialise on an in-process sync.RWMutex - the Redis lock is only ever contested across pods, not within a single pod.

Self-healing (Maildir sync-on-open, dbox/mdbox reactive heal), the operator rebuild path, and mdbox rotation/tuning knobs: see STORAGE.

Full-text search

SEARCH BODY, SEARCH TEXT and SEARCH HEADER are backed by a per-user full-text index instead of a linear message scan. SEARCH RETURN (RELEVANCY) (RFC 4731/6203) surfaces the engine's ranking as scores 1-100, min-max normalized per result set - requires a yarilo-patches go-imap fork, since upstream has no RELEVANCY support.

The yarilo-fts service owns the index end-to-end (indexing and lookups) and is the only process linking libxapian (cgo); session binaries stay pure-Go and send LOOKUP over the internal TAB protocol. Enable it with fts.enabled + fts_engine: flatcurve. Multi-language indexing/search, attachment decoders, and the full config surface are documented in FTS.

Acceptance benchmark (app/fts-bench, synthetic corpus, local Xapian glass shards):

Search stays sub-millisecond as the mailbox grows; the linear scan it replaces grows with message count. See https://doc.yarilomail.org/latest/features/fts.html for the full design and the phased roadmap (relevancy / strict-substring / multi-language, then attachment decoders).

Cluster components

All intra-cluster protocols are TAB-delimited text with LF termination and a version handshake.

Session routing (backend_addr / director_addr precedence), sticky assignments, username-hash templates (username_hash), domain placement and the operator's domain move (yarctl director domains move), backend evacuation, the per-user flush hook, tag sharding models, and the self-organizing ring formation (with its design history) are all documented in DIRECTOR.

Quick start (Helm)

# Add the chart (local checkout)
helm upgrade --install yarilo ./helm \
  -f helm_values/values-sandbox.yaml \
  -n yarilo --create-namespace

See the installation guide for a full Kubernetes walkthrough with cert-manager, Let's Encrypt, and an external MySQL passdb.

Minimal yarilo.yaml for bare-metal single-node:

hostname: mail.example.com

auth:
  passdb:
    - driver: mysql
      dsn: "yarilo:secret@tcp(127.0.0.1:3306)/yarilo"

storage:
  persistence:
    enabled: true
    size: 50Gi

locks:
  mode: embedded   # single-node only; use remote in k8s

LOG_LEVEL=debug yarilo-imap -config yarilo.yaml

LOG_LEVEL=debug is per-service - set it on the process whose code path you are tracing (a delivery breadcrumb lives in yarilo-lmtp, a search one in yarilo-imap). At debug level every write path emits an explicit "wrote UID=N file=Q" line (lmtp: delivered, imap: append saved, imapsieve: fileinto saved, sieve/pipe: invoked, sieve/sender: notification sent), and the read side logs what it actually scanned when a lookup comes back empty (imap: search matched no messages with the folder's record count and UID range, imap: fetch skipped uid absent from client view, fileindex: reload applied / reset folder with record counts before/after) - so a delivery→visibility mismatch is diagnosable from the log alone. The full-text search pipeline is instrumented the same way: yarilo-fts logs every request it handles (fts: indexed / expunged / lookup / rescanned / status) and the indexing worker reports each run (fts: index run start / done with checkpoint, indexed/skipped counts and duration, plus per-message fts: message indexed), while the yarilo-imap/yarilo-lmtp sides log the handoff (imap: fts notify sent, lmtp: fts autoindex queued) and how many candidates a search got back (imap: fts search candidates). FTS lines log result and term COUNTS, never the query terms (private mail content).

These lines carry only metadata (user, folder, UID, filename, counts); passwords, tokens, SASL response data, and search query terms are never logged.

Quick start (Docker Compose)

Single-host, no Kubernetes - the standalone topology (login proxies + session backends + auth/warden/locks + Redis) on one host, SQLite userdb, one image:

cd deploy/compose
cp .env.example .env
./gen-certs.sh mail.example.test     # self-signed TLS for local use
docker compose up -d

Full walkthrough - creating users, TLS, MTA (Postfix) integration, verifying, backups - in DOCKER-COMPOSE.

Mailbox format migration (offline)

yarilo-migrate is an offline, on-disk format converter for a per-user mailbox tree - sources maildir / dbox-v1 / mdbox-v1, destinations sdbox / mdbox. It is not a cross-server (dsync/imapc) migration that pulls mail over IMAP from another server.

yarilo-migrate \
  --from /var/mail/vhosts \
  --to   /var/mail/dbox \
  --format dbox   # or mdbox; --dry-run to preview

GUID pre-migration

Mail stored before yarilo 2.3.8 carries no per-message GUID, so its EMAILID (RFC 8474) is stamped the first time a client selects the folder. That one-off pass is automatic and needs no operator action; this command only moves the cost off the first SELECT, which is worth doing for very large folders.

yarilo-migrate --guid-backfill \
  --config /etc/yarilo/yarilo.yaml \   # layout, driver, userdb and yarilo-locks
  --user   u1@example.com               # optional: one user instead of all
                                        # --dry-run reports what is pending

Per-user INDEX=, CONTROL=, ALT= and mail_path overrides live in the userdb, not in yarilo.yaml, so by default the tool looks each user up through backend_api.auth_master_addr exactly as a session does. A store whose auth is not running is handled by --offline plus the templates; the two sources are mutually exclusive, because a template disagreeing with userdb would address a mailbox the sessions never use.

The templates take ~/ or %h for the user's home, plus %u/%n/%d, so a userdb value of INDEX=~/index is written the same way here.

The tool never creates an index. A path holding no index is an error naming the path, not an empty folder reported as complete.

Without --config both --driver and --root are required, and the run is unlocked. Users are enumerated only for a layout whose leaf directory names the user (%u, or %n with %d above it); any other mail_home_template has to be driven one user at a time with --user.

The command writes to shared storage, so pass --config to make it take the same locks the services take; it is then safe to run against a live store. Without --config it runs unlocked, which is only safe with the store stopped. Repeat runs are no-ops: a stamped folder is recorded as done and an already assigned GUID is never rewritten.

Test coverage

The coverage badge reports main, the released code, measured by .github/workflows/coverage.yml on every release and on demand: the whole tree with -coverpkg=./..., so a package's statements count when any package's tests run them, plus the flatcurve engine under its build tag. A branch is measured by running the workflow on it by hand (Actions → Coverage → Run workflow); Codecov shows each branch apart.

Left out (codecov.yml): the main packages in app/ that are entry points, which wire the config into components tested in internal/ and start them, and the stand tools (auth-bench, fts-bench, smoketest-e2e) that run against a live deployment. Counted: yarctl, smoketest and yarilo-migrate, which hold command logic of their own.

Documentation

Full documentation lives at doc.yarilomail.org (source: yarilomail/documentation).

License

GNU Affero General Public License v3.0 - see LICENSE.