Documentation
¶
Overview ¶
Package dlq provides the dead letter queue for jobs that have exhausted their retry budget. It supports inspection, replay, and purging.
When a job fails and MaxRetries has been reached, the executor calls Service.Push to move it into the DLQ. The original payload, error message, and retry counts are preserved for debugging.
Entry ¶
A Entry captures:
- JobID / JobName / Queue: original job identity
- Payload: the raw JSON payload at time of failure
- Error: the final error message
- RetryCount / MaxRetries: exhausted retry budget
- FailedAt: when the terminal failure occurred
- ReplayedAt: set when the entry is replayed (nil if not yet replayed)
Service ¶
Service wraps the DLQ store with high-level operations:
svc := dlq.NewService(store, jobStore)
// Push is called automatically by the executor on terminal failure.
svc.Push(ctx, failedJob, err)
// Access the underlying store for list/get/purge/count.
svc.DLQStore().ListDLQ(ctx, dlq.ListOpts{Limit: 50})
svc.DLQStore().PurgeDLQ(ctx)
Replay ¶
Replaying an entry re-enqueues the original job with the same payload. Use engine.Engine.ReplayDLQ, the admin API (POST /v1/dlq/:entryId/replay) or Service.Replay. Every one of them claims the entry first (ReplayClaimer.ClaimReplay), which sets ReplayedAt and ReplayedJobID, so an entry is replayed into at most one job however many callers try.
Admin API ¶
The DLQ is exposed via the HTTP admin API:
- GET /v1/dlq — list entries
- GET /v1/dlq/:entryId — get a single entry
- POST /v1/dlq/:entryId/replay — replay one entry
- POST /v1/dlq/purge — purge all entries
- GET /v1/dlq/count — entry count
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CountOpts ¶
CountOpts filters CountDLQEntries. A zero FailedBefore counts every entry; a set one counts entries whose FailedAt is strictly before it, the same boundary PurgeDLQ deletes on, so a purge confirmation can show exactly how many rows will go.
type Enqueuer ¶
Enqueuer puts a prepared job, ID and all, on its queue. Replay hands it the job it built from an entry. The engine supplies one that checks the job still fits a worker, wakes the local pool and reports the enqueue to extensions; without one, Replay writes the job straight to the job store.
type Entry ¶
type Entry struct {
ID id.DLQID `json:"id"`
JobID id.JobID `json:"job_id"`
JobName string `json:"job_name"`
Queue string `json:"queue"`
Payload []byte `json:"payload"`
Error string `json:"error"`
RetryCount int `json:"retry_count"`
MaxRetries int `json:"max_retries"`
ScopeAppID string `json:"scope_app_id,omitempty"`
ScopeOrgID string `json:"scope_org_id,omitempty"`
FailedAt time.Time `json:"failed_at"`
ReplayedAt *time.Time `json:"replayed_at,omitempty"`
CreatedAt time.Time `json:"created_at"`
// ReplayedJobID is the job a replay created, or for a retry the
// retried job itself. ClaimReplay sets it together with ReplayedAt,
// and it is nil while the entry is unreplayed. An entry marked by the
// older ReplayDLQ has ReplayedAt set and this nil.
ReplayedJobID *id.JobID `json:"replayed_job_id,omitempty"`
// Priority is the claim ordering the job was enqueued with.
Priority int `json:"priority,omitempty"`
// Timeout is how long the handler was allowed to run.
Timeout time.Duration `json:"timeout,omitempty"`
// LeaseTTL is how long each renewal extended the job's lease. Losing
// it is the reason this block exists: a six-hour job replayed without
// it falls back to the pool default, which is measured in seconds, so
// its lease lapses mid-run and it is reclaimed and restarted over and
// over without ever finishing. That is the exact failure long-running
// jobs carry a per-job TTL to avoid.
LeaseTTL time.Duration `json:"lease_ttl,omitempty"`
// ArtifactBindings is the encoded map of declared input names to
// artifacts, carried verbatim. A handler that declares inputs cannot
// run without them.
ArtifactBindings []byte `json:"artifact_bindings,omitempty"`
// Resources and ResourceLimits are what the job asked for and what it
// was capped at. Without them a replayed job looks free to schedule
// and can be claimed by a worker that cannot actually host it.
Resources resource.Set `json:"resources,omitempty"`
ResourceLimits resource.Set `json:"resource_limits,omitempty"`
// ResourceClass is the named class the job was placed in.
ResourceClass string `json:"resource_class,omitempty"`
// InputBytes and PrimaryInputHash are derived from the bindings by the
// engine at enqueue. Replay does not run that derivation, so they are
// carried rather than recomputed, and stay consistent with the
// bindings above.
InputBytes int64 `json:"input_bytes,omitempty"`
PrimaryInputHash string `json:"primary_input_hash,omitempty"`
}
Entry represents a job that has exhausted its retry budget and been moved to the dead letter queue for inspection or replay.
The fields below the identity block exist so Replay can rebuild a job that behaves like the one that failed. Replay enqueues the job it builds as it is, through the engine's prepared-job path or straight to the job store, and neither re-derives these from the definition: whatever the entry does not carry, the replayed job silently takes a default for. They are stored as the effective values from the failed job rather than looked up from the definition by name, because a definition's declaration is only half the story. The enqueue site can override every one of them, and a definition can be changed or removed between the failure and the replay.
type ListOpts ¶
type ListOpts struct {
// Limit is the maximum number of entries to return. Zero means no limit.
Limit int
// Offset is the number of entries to skip.
Offset int
// Queue filters by queue name. Empty means all queues.
Queue string
}
ListOpts controls pagination and filtering for DLQ list queries.
type Page ¶
Page is one page of ListDLQPage. NextCursor and Complete mean what they mean on job.Page.
type PageLister ¶
type PageLister interface {
ListDLQPage(ctx context.Context, opts PageOpts) (Page, error)
CountDLQEntries(ctx context.Context, opts CountOpts) (int64, error)
}
PageLister is the paged DLQ read and the filtered count the dashboard uses. ListDLQ and CountDLQ are kept, unchanged, for existing callers.
type PageOpts ¶
type PageOpts struct {
Queue string
NamePrefix string
ScopeAppID string
ScopeOrgID string
Replayed *bool
Cursor string
Limit int
}
PageOpts filters and pages ListDLQPage, newest first by ID. NamePrefix matches the entry's JobName. Replayed nil lists both; a pointer to false lists only entries nobody has replayed yet, which is the dashboard's default view.
type ReplayClaimer ¶
type ReplayClaimer interface {
// ClaimReplay sets replayed_at = now and replayed_job_id = jobID only
// if replayed_at is null. It returns dispatch.ErrDLQAlreadyReplayed
// when replayed_at was already set, and dispatch.ErrDLQNotFound when
// the entry does not exist. The check and the write are one atomic
// step on every backend: of two concurrent claims, exactly one wins.
ClaimReplay(ctx context.Context, entryID id.DLQID, jobID id.JobID) error
// ReleaseReplay clears replayed_at and replayed_job_id, but only if
// replayed_job_id still equals jobID, so a release never undoes
// somebody else's claim. When it does not match, including on an
// entry nobody claimed, it is a no-op and returns nil. It returns
// dispatch.ErrDLQNotFound when the entry does not exist. Like the
// claim, the comparison and the write are one atomic step.
ReleaseReplay(ctx context.Context, entryID id.DLQID, jobID id.JobID) error
// GetDLQByJobID returns the newest entry for a failed job, newest by
// entry ID (the order ListDLQPage uses), or dispatch.ErrDLQNotFound
// when the job has none.
GetDLQByJobID(ctx context.Context, jobID id.JobID) (*Entry, error)
// DeleteDLQ hard-deletes one entry. It returns dispatch.ErrDLQNotFound
// when the entry does not exist.
DeleteDLQ(ctx context.Context, entryID id.DLQID) error
}
ReplayClaimer is the conditional write that makes replay idempotent.
A replay used to read the entry, enqueue a new job, then mark the entry replayed, so two operators clicking replay at once both got a job. The claim moves the check into the store: whoever sets replayed_at first owns the replay, and everyone after them is told so.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service provides high-level DLQ operations over a Store.
func NewService ¶
func NewService(store Store, jobStore job.Store, opts ...ServiceOption) *Service
NewService creates a DLQ service.
func (*Service) DLQStore ¶
DLQStore returns the underlying DLQ store for direct access to List, Get, Purge, and Count operations.
func (*Service) Push ¶
Push builds a DLQ Entry from a failed job and persists it. The error string is captured from the original handler error.
func (*Service) Replay ¶
Replay re-enqueues a DLQ entry as a new pending job and marks the entry as replayed. The new job gets a fresh ID, zero retry count, and runs immediately. See ReplayEntry for how a replay is made safe to run twice.
func (*Service) ReplayEntry ¶
ReplayEntry is Replay for an entry the caller has already read.
The entry is claimed before anything is enqueued: the new job's ID is minted first and ClaimReplay records it on the entry only if nobody replayed or retried the entry before. Of two concurrent replays, exactly one gets past the claim and the other gets an error wrapping dispatch.ErrDLQAlreadyReplayed, so one failure is never turned back into two jobs. If the enqueue then fails, the claim is released and the entry can be replayed again.
A process that dies between the claim and the enqueue leaves the entry marked replayed with no job behind it. That is the safe side to fail on: the operator sees an entry replayed into a job that does not exist, which is visible, where the opposite order could run the job twice, which is not.
type ServiceOption ¶
type ServiceOption func(*Service)
ServiceOption configures a Service.
func WithEnqueuer ¶
func WithEnqueuer(fn Enqueuer) ServiceOption
WithEnqueuer routes Replay's enqueue through fn.
type Store ¶
type Store interface {
ReplayClaimer
// PushDLQ adds a failed job entry to the dead letter queue.
PushDLQ(ctx context.Context, entry *Entry) error
// ListDLQ returns DLQ entries matching the given options.
ListDLQ(ctx context.Context, opts ListOpts) ([]*Entry, error)
// GetDLQ retrieves a DLQ entry by ID.
GetDLQ(ctx context.Context, entryID id.DLQID) (*Entry, error)
// ReplayDLQ marks a DLQ entry as replayed. The actual re-enqueue is
// handled at the service layer.
ReplayDLQ(ctx context.Context, entryID id.DLQID) error
// PurgeDLQ removes DLQ entries with FailedAt before the given time.
// Returns the number of entries removed.
PurgeDLQ(ctx context.Context, before time.Time) (int64, error)
// CountDLQ returns the total number of entries in the dead letter queue.
CountDLQ(ctx context.Context) (int64, error)
}
Store defines the persistence contract for the dead letter queue.