redis

package
v1.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: MIT Imports: 24 Imported by: 0

README

Redis backend

Redis stores execution state and histories and coordinates workflow/activity consumers through streams and Lua scripts. See backend operations for deployment and durability requirements.

Client ownership and retention

NewRedisBackend accepts an application-configured redis.UniversalClient. Configure authentication, TLS and connection policy in that client. The backend's Close closes the supplied Redis client; do not share it with an application component that must outlive the backend.

WithKeyPrefix separates key namespaces, not authorization. Use separate credentials or Redis deployments when access must be isolated. WithBlockTimeout controls blocking queue reads.

Finished executions do not expire by default. Configure WithAutoExpiration for finished runs and WithAutoExpirationContinueAsNew for continued executions. The client-managed SQL expiration workflow is not supported by Redis; Feature_Expiration reports false. Redis persistence and replication settings determine whether stored state survives server failure.

Instances and their state

Execution state uses instance:{instanceID}:{executionID} keys, preceded by the configured key prefix. Additional indexes track active instances, creation order and expiration. keys.go owns the exact names; do not construct or mutate them in application code.

History and pending events

Recorded history and pending events use separate per-execution streams, keyed by history:{instanceID}:{executionID} and pending-events:{instanceID}:{executionID}, with the configured prefix. The worker replays recorded events and consumes pending events when processing the next workflow task.

Timer events

Timer events are stored in a sorted set (ZSET). Whenever a worker checks for a new workflow instance task, the sorted set is checked to see if any of the pending timer events is ready yet. If it is, it's added to the pending events before those are returned for pending workflow tasks.

Task queues

We need queues for activities and workflow instances. In both cases, we have tasks being enqueued, workers polling for works, and we have to guarantee that every task is eventually processed. So if a worker has dequeued a task and crashed, for example, eventually we need another worker to pick up the task and finish it.

Task queues are implemented using Redis STREAMs. In addition for queues where we only want a single instance of a task to be in the queue, we maintain an additional SET.

Alternatives considered

Option 1 - ZSET:

Store keys for queue items in a ZSET. The score for the sorted set is the timestamp when the task is unlocked. For new tasks the SCORE is the current timestamp.

Workers make a ZRANGE query to that sorted set, looking for tasks where the SCORE is in -inf, now). That finds tasks whose unlock timestamp is in the past, either because they were just queued or because the lock expired.

Once a worker picks up a task, the score is updated to now + lock_timeout. Query and update are done in a transaction with a WATCH on the queue key.

When a worker is done with a task, it removes it from the ZSET (ZREM).

Pro:

  • No special handling for recovering crashed tasks, they'll automatically unlock

Con:

  • Need for polling and cannot use any of the blocking redis commands
  • WAIT with transaction, or a script required

Option 2 - LISTS

Use two LISTs, one for the pending queue, one for the processing queue. To enqueue a new task, LPUSH it onto the pending list. Also add an entry in a separate ZSET where the score is the unlock timestamp. Initially that timestamp will be the current timestamp. The LPUSH and ZADD are done in a transaction with a WATCH on the queue key, and retried if another client modified the queue in the mean time. Alternatively, the two operations can be done in a script.

For picking up tasks, we use a blocking BLMOVE .. RIGHT LEFT command to pick up the next available task from pending and move it to processing as a single atomic operation. Once picked up, the SCORE in the ZSET is adjusted to now + lock_timeout.

When a worker is done with a task, it removes it from the processing list (LREM), and the ZSET (ZREM).

To recover abandoned tasks, we periodically scan the processing list

Pro:

  • Blocking call does not require constant polling

Con:

  • Requires periodic scans of the processing list to find tasks that have been abandoned
  • Picking up a task, adjusting its ZSET value and the periodic scan could run into race conditions

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewRedisBackend

func NewRedisBackend(client redis.UniversalClient, opts ...RedisBackendOption) (*redisBackend, error)

Types

type KeyInfo

type KeyInfo struct {
	StreamKey string
	SetKey    string
}

type RedisBackendOption

type RedisBackendOption func(*RedisOptions)

func WithAutoExpiration

func WithAutoExpiration(expireFinishedRunsAfter time.Duration) RedisBackendOption

WithAutoExpiration sets the duration after which finished runs will expire from the data store. If set to 0 (default), runs will never expire and need to be manually removed.

func WithAutoExpirationContinueAsNew

func WithAutoExpirationContinueAsNew(expireContinuedAsNewRunsAfter time.Duration) RedisBackendOption

WithAutoExpirationContinueAsNew sets the duration after which runs that were completed with `ContinueAsNew` automatically expire. If set to 0 (default), the overall expiration setting set with `WithAutoExpiration` will be used.

func WithBackendOptions

func WithBackendOptions(opts ...backend.BackendOption) RedisBackendOption

func WithBlockTimeout

func WithBlockTimeout(timeout time.Duration) RedisBackendOption

WithBlockTimeout sets the timeout for blocking operations like dequeuing a workflow or activity task

func WithKeyPrefix

func WithKeyPrefix(prefix string) RedisBackendOption

WithKeyPrefix sets the prefix for all keys used in the Redis backend.

type RedisOptions

type RedisOptions struct {
	*backend.Options

	BlockTimeout time.Duration

	AutoExpiration              time.Duration
	AutoExpirationContinueAsNew time.Duration

	KeyPrefix string
}

type TaskItem

type TaskItem[T any] struct {
	// TaskID is the generated ID of the task item
	TaskID string

	// ID is the provided id
	ID string

	// Optional data stored with a task, needs to be serializable
	Data T
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL