shared

module
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 20, 2026 License: AGPL-3.0

README

Shared

Common utilities for the Atomic Blend backend microservices.

Overview

Contains infrastructure-level packages used by two or more services: MongoDB connection management, RabbitMQ AMQP utilities, Age encryption helpers, Firebase Cloud Messaging, S3 storage, Rspamd integration, subscription validation, and testing utilities.

Packages

Package Description
amqp/ RabbitMQ producer/consumer
db/ MongoDB / PostgreSQL datastore selection (Open)
id/ Backend-neutral identifier (Mongo ObjectID or uuid)
authid/ Parse the authenticated user id into an id.ID
migrate/ PostgreSQL migration runner
syncquery/ since sync filters and keyset cursors
age_encryption/ Age encryption utilities
fcm/ Firebase Cloud Messaging
s3/ AWS S3 / MinIO storage
rspamd/ Spam filtering
ratelimit/ Per-IP token-bucket gin middleware + TRUSTED_PROXIES setup
subscription/ Subscription validation
shortcuts/ Utility helpers
test_utils/ In-memory MongoDB and throwaway PostgreSQL for tests

Upgrading from v0.8.x (breaking API changes)

Repository and handler code no longer hands raw primitive.ObjectIDs to this module; it uses id.ID, which holds either a Mongo ObjectID or a uuid.

  • subscription.CheckStorageQuota(ctx, userID, client): userID is now an id.ID (was primitive.ObjectID). A zero id is rejected with HTTP 401 and true is returned, even with a nil auth client.
  • syncquery.Cursor.ID is an id.ID (was primitive.ObjectID) and syncquery.EncodeCursor(updatedAt, id) takes an id.ID. DecodeCursor accepts either id form.
  • (*syncquery.Envelope).SetNextCursor(..., lastID id.ID) takes an id.ID and now returns an error (syncquery.ErrZeroCursorID for a zero id) instead of returning nothing.
  • db.UserBSONSize(ctx, db, spec, userID) and db.UserBSONSizes(ctx, db, specs, userID) take an id.ID (were primitive.ObjectID) and do the ObjectID conversion themselves. A zero or uuid-kind id is an error wrapping id.ErrInvalid (never a 0-byte measurement with partial=false); callers should answer it like any other invalid user_id.
  • db.Open returns every Mongo failure as an error instead of panicking, and publishes db.MongoClient / db.Database only on success.
  • id.ID behaves like the primitive.ObjectID it replaced on the Mongo datasource. The zero id.ID encodes as the nil ObjectID in BSON (not null), is omitted by bson:",omitempty", and marshals to JSON as "000000000000000000000000"; id.FromObjectID(primitive.NilObjectID), id.FromUUID(uuid.Nil) and id.Parse of either all-zero form all return the same zero id.ID, so a lookup by it matches nothing and answers 404 (never 400). pgid.DeriveID maps the zero id to the nil uuid for the same reason. authid.Parse still rejects it. binding:"required" on an id.ID field rejects a zero id with the usual 400 (registered by the id package's init for gin's binding validator; a standalone validator.New() must call id.RegisterValidator(v)). Conditional tags (required_with, excluded_*) and *id.ID fields evaluate as they did on origin/dev. pgid.ServiceName never panics (MongoDB ignores MONGO2PG_SERVICE_NAME); an invalid value refuses to boot under DATASOURCE=postgres (db.Open, VerifyDerivation). syncquery.EncodeCursor of the zero id encodes 000000000000000000000000 and round-trips. The Postgres dial + ping is bounded to 10s.
  • id.Parse does not trim whitespace, and accepts a uuid only in its canonical hyphenated form and only when id.SetUUIDAccepted(true) has been called. db.Open calls it from the datasource: PostgreSQL accepts uuid text, MongoDB (and any process that never calls db.Open) rejects it with id.ErrInvalid, a 400 at the API boundary exactly as on origin/dev, so a uuid can never be persisted as a BSON string. Tests that exercise uuid-kind ids through id.Parse call id.SetUUIDAccepted(true) first.

Migrations (migrate)

migrate.Run(ctx, pool, fsys, versionTable) applies a service's embedded .sql migrations with tern, under tern's global advisory lock (held for the whole run, so concurrent instances and services sharing one database are serialised; there is no per-service lock to configure).

migrate is new: there is no pre-existing or default version table to carry over. versionTable is required and must be specific to the service, because in a single-database deployment a shared table would be one version row for every service. A new consumer picks its own, e.g. public.auth_schema_version, public.calendar_schema_version.

Testing against PostgreSQL

test_utils.RequirePostgres(t) returns a fresh, empty PostgreSQL 18 database. Locally it starts a testcontainers container (and skips the test when there is no Docker daemon). On a CI runner without a container runtime it needs a real server:

  • add a postgres service to the job that runs the tests, in the merge-request-only overrides file the CI includes (see .gitlab/ci/merge-request-overrides.yml in this repository):

    pre-commit-all:
      services:
        - name: postgres:18-alpine
          alias: postgres
    
  • set POSTGRES_TEST_DSN (and POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DB, which seed that service) in the variables: of .gitlab-ci.yml, e.g. postgres://ci_user:ci_password@postgres:5432/ci_db?sslmode=disable. Each test still gets its own isolated database carved out of that server.

When POSTGRES_TEST_DSN is set, or CI is, RequirePostgres fails instead of skipping if no database is reachable, so a broken CI service cannot turn the PostgreSQL tests into silent skips.

Getting Started

# From the backend root (go.work must exist)
go test ./shared/...

Documentation

Shared Package Docs

Contributing

See Contributing Guide. Commits must follow Conventional Commits.

Directories

Path Synopsis
Package ageencryptionservice contains the age encryption service
Package ageencryptionservice contains the age encryption service
interfaces
Package ageinterfaces contains the interfaces for the age encryption service
Package ageinterfaces contains the interfaces for the age encryption service
Package amqpservice contains the AMQP service
Package amqpservice contains the AMQP service
interfaces
Package amqpinterfaces contains the interfaces for the AMQP service
Package amqpinterfaces contains the interfaces for the AMQP service
utils
Package amqputils contains the AMQP consumer utils
Package amqputils contains the AMQP consumer utils
Package authid turns the user id carried by an authenticated request into the neutral id.ID every repository takes, so each controller doesn't repeat the same parse-and-map-the-error block.
Package authid turns the user id carried by an authenticated request into the neutral id.ID every repository takes, so each controller doesn't repeat the same parse-and-map-the-error block.
Package id defines the canonical identifier used at repository interface boundaries, so a repository's callers never have to know whether a record lives in MongoDB (a 24-character hex ObjectID) or PostgreSQL (a UUID).
Package id defines the canonical identifier used at repository interface boundaries, so a repository's callers never have to know whether a record lives in MongoDB (a 24-character hex ObjectID) or PostgreSQL (a UUID).
Package migrate applies a service's PostgreSQL schema migrations from an embedded fs.FS of .sql files (github.com/jackc/tern/v2/migrate's own file format and layout — see its docs for how to name and write them), coordinating concurrent instances via a PostgreSQL advisory lock.
Package migrate applies a service's PostgreSQL schema migrations from an embedded fs.FS of .sql files (github.com/jackc/tern/v2/migrate's own file format and layout — see its docs for how to name and write them), coordinating concurrent instances via a PostgreSQL advisory lock.
Package pgid derives the PostgreSQL primary key that mongo2pg mints for a MongoDB row, without consulting the database.
Package pgid derives the PostgreSQL primary key that mongo2pg mints for a MongoDB row, without consulting the database.
Package pgjson encodes and decodes the jsonb columns of a database that mongo2pg produced.
Package pgjson encodes and decodes the jsonb columns of a database that mongo2pg produced.
Package ratelimit is a per-key token-bucket gin middleware for public routes.
Package ratelimit is a per-key token-bucket gin middleware for public routes.
Package rspamdservice contains the mock Rspamd service
Package rspamdservice contains the mock Rspamd service
client
Package rspamdclient contains the Rspamd client models
Package rspamdclient contains the Rspamd client models
interfaces
Package rspamdinterfaces contains the interfaces for the Rspamd service
Package rspamdinterfaces contains the interfaces for the Rspamd service
s3
Package s3service defines s3 operations available
Package s3service defines s3 operations available
Package absentry wires Sentry into the backend services.
Package absentry wires Sentry into the backend services.
Package syncquery builds the filters, options and cursors used by every `since` sync endpoint, so all services share one implementation.
Package syncquery builds the filters, options and cursors used by every `since` sync endpoint, so all services share one implementation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL