otel

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package otel adds OpenTelemetry tracing and metrics to any workflow driver, as one step middleware and one enqueue middleware:

inst, err := otel.New(otel.WithTracerProvider(tp), otel.WithMeterProvider(mp))
c := memory.New(memory.WithMiddleware(inst.Step()))
client := middleware.WrapClient(c, inst.Enqueue())

Spans

Enqueue opens a span named "workflow.enqueue <workflow>" (kind Producer) around every Client.Enqueue, and Step opens one named "workflow.step <workflow>/<step>" (kind Consumer) around every step attempt. Both carry workflow.name and workflow.queue (see Metrics for the enqueue side); the step span also carries workflow.step, workflow.run_id and workflow.attempt, and the enqueue span workflow.run_id once the driver returns it. Job labels are not recorded by default, because they can carry tenant or user ids into a third-party trace backend; WithLabelFilter (chosen keys) or WithAllLabels opts in, recording them as workflow.label.<key>. Payloads and step outputs are never recorded: they can hold personal data, and a codec may not have run yet. WithStepSpanName and WithEnqueueSpanName replace the span names; keep a name to a bounded set of values.

Trace context propagation

The enqueue middleware injects the caller's trace context into the job's labels and the step middleware extracts it, so a step span is a child of the span that enqueued its run even when the step runs on another machine. WithParenting changes that relation: ParentLink keeps the span ctx already holds (an engine SDK's) as the parent and links the enqueue span, as the OTel messaging conventions do; ParentNewRootLink starts each step span as a new trace linked to the enqueue span, so a fan-out of thousands of jobs does not build one trace too large to keep. The carrier prefixes every propagator key with LabelPrefix, so with the default W3C TraceContext propagator the labels are

otel.traceparent
otel.tracestate (only when the trace has state)

The prefix is reserved for this package (it is the core's workflow.LabelPrefixOTel): Enqueue refuses a job that already sets a label under it with workflow.ErrInvalid, so use Enqueue once per chain. It is chosen so every driver accepts it: the core validates no label keys, and the Hatchet driver reserves only "wf__", "hatchet__" and the bare "traceparent" its SDK writes itself. The caller's Labels map is never modified; the middleware copies it before adding keys. When a step's labels carry no trace context, the step span's parent is whatever span ctx already holds.

Metrics

  • workflow.step.duration (histogram, seconds): one per step attempt, with workflow.name, workflow.step, workflow.queue and workflow.outcome.

  • workflow.step.outcomes (counter): one per step attempt, same attributes.

  • workflow.enqueue.count (counter): one per Enqueue, with workflow.name, workflow.outcome and, when Job.Queue is set, workflow.queue.

  • workflow.queue.wait (histogram, seconds, QueueWaitBuckets): one per step attempt whose StepInfo.ReadyAt the driver set: attempt start minus ReadyAt, so for a run's first step it is enqueue (or RunAt) to start. With workflow.name, workflow.step and workflow.queue. A driver that leaves ReadyAt zero records no sample; a ReadyAt after the start (clock skew between nodes) records zero.

  • workflow.queue.depth (gauge): runs waiting per queue, one point per queue with workflow.queue, read through ObserveQueueDepth's QueueDepthFunc at every collection; admin.Admin.QueueDepths fits it.

  • workflow.runs.finished (counter): one per run end delivered to the observer RunFinished returns (register it with the driver's OnRunFinished), with workflow.name and workflow.run.status: "succeeded", "failed", "cancelled", or "discarded" on a driver with CapRunStates. Delivery is at least once, so a redelivered end is counted again. "retrying" never appears: it is not an end.

Step spans and step metrics carry workflow.queue (StepInfo.Queue); the enqueue span and counter carry the queue the job asked for (Job.Queue), and no workflow.queue when the job left the choice to the workflow, whose queue only the driver resolves. workflow.outcome is "ok", "error" (retryable), "permanent", "cancelled" or "discarded" (the step returned workflow.Discard). Run ids never become metric attributes, and labels do only when named one by one with WithMetricLabels (recorded as workflow.label.<key>): each named key multiplies the series by its number of values, so name only labels with a small, bounded set of values, such as a tier or a region.

Step spans also carry workflow.node (StepInfo.Node) when the driver names the process that ran the attempt. Job.Tenant is recorded as workflow.tenant on step and enqueue spans only with WithTenant, since a tenant ID can identify a customer, and is never a metric attribute.

Providers

No global is read unless asked for: a nil or absent TracerProvider or MeterProvider means the otel global one (otel.GetTracerProvider, otel.GetMeterProvider), which is a no-op until the app installs one. The propagator defaults to W3C TraceContext, not the global propagator, whose default propagates nothing; WithPropagator replaces it.

Index

Constants

View Source
const (
	AttrWorkflow = attribute.Key("workflow.name")
	AttrStep     = attribute.Key("workflow.step")
	AttrRunID    = attribute.Key("workflow.run_id")
	AttrAttempt  = attribute.Key("workflow.attempt")
	AttrOutcome  = attribute.Key("workflow.outcome")
	AttrQueue    = attribute.Key("workflow.queue")
	// AttrNode is StepInfo.Node, the process that ran the attempt, on
	// step spans when the driver sets it.
	AttrNode = attribute.Key("workflow.node")
	// AttrTenant is Job.Tenant (StepInfo.Tenant) on step and enqueue
	// spans, recorded only with WithTenant.
	AttrTenant = attribute.Key("workflow.tenant")
	// AttrRunStatus is the terminal workflow.Status of a run on
	// MetricRunsFinished: succeeded, failed, cancelled, or discarded on a
	// driver with CapRunStates.
	AttrRunStatus        = attribute.Key("workflow.run.status")
	LabelAttributePrefix = "workflow.label."
)

Attribute keys. Labels are recorded under LabelAttributePrefix + key.

View Source
const (
	MetricStepDuration = "workflow.step.duration"
	MetricStepOutcomes = "workflow.step.outcomes"
	MetricEnqueueCount = "workflow.enqueue.count"
	// MetricQueueWait is how long a step attempt waited for a worker:
	// from the moment it became ready (StepInfo.ReadyAt) to the moment it
	// started. For a run's first step that is enqueue to start.
	MetricQueueWait = "workflow.queue.wait"
	// MetricQueueDepth is how many runs wait in each queue, read through a
	// QueueDepthFunc at every collection.
	MetricQueueDepth = "workflow.queue.depth"
	// MetricRunsFinished counts run ends by workflow and AttrRunStatus,
	// fed by RunFinished.
	MetricRunsFinished = "workflow.runs.finished"
)

Metric names.

View Source
const LabelPrefix = workflow.LabelPrefixOTel

LabelPrefix starts every job label this package writes to carry trace context; see the package doc for why this prefix. It is the core's workflow.LabelPrefixOTel, which this package owns.

View Source
const ScopeName = "github.com/ubgo/workflow/otel"

ScopeName is the instrumentation scope of every tracer and meter this package creates.

Variables

This section is empty.

Functions

func QueueWaitBuckets

func QueueWaitBuckets() []float64

QueueWaitBuckets returns the bucket boundaries, in seconds, of MetricQueueWait: from 5ms to an hour, because a queue wait ranges from an idle worker picking a job up at once to a backlog of an hour, and the SDK's default boundaries (0 to 10000, meant for milliseconds) would put almost every wait in the first two buckets. An SDK View replaces them. It returns a new slice on every call, so no caller can change another's boundaries.

Types

type Instrumentation

type Instrumentation struct {
	// contains filtered or unexported fields
}

Instrumentation holds the tracer, meter instruments and propagator the middleware share. Build it once with New and reuse it.

func New

func New(opts ...Option) (*Instrumentation, error)

New builds the instrumentation. It fails when an option is invalid (workflow.ErrInvalid) or the meter provider refuses to create an instrument.

func (*Instrumentation) Enqueue

Enqueue returns the enqueue middleware: a span and a count per Enqueue, and the span's trace context written into a copy of job.Labels under LabelPrefix so the run's steps continue the trace. A job that arrives with a label under LabelPrefix is refused with workflow.ErrInvalid, before any span or count, rather than having it silently replaced: it would otherwise let a caller choose the trace the run's steps join. So put Enqueue in a chain once.

func (*Instrumentation) ObserveQueueDepth

func (i *Instrumentation) ObserveQueueDepth(fn QueueDepthFunc) (metric.Registration, error)

ObserveQueueDepth reports fn's depths as the workflow.queue.depth gauge, one data point per queue with a workflow.queue attribute, read at every metric collection (no poller of its own). An error from fn skips that collection's points and is handed to the OpenTelemetry error handler. Unregister the returned registration to stop. A nil fn is refused with workflow.ErrInvalid.

func (*Instrumentation) RunFinished

func (i *Instrumentation) RunFinished() workflow.RunFinishedFunc

RunFinished returns a run-finished observer that counts each run end on MetricRunsFinished with the workflow name and AttrRunStatus. Register it with the driver's workflow.RunNotifier (OnRunFinished) before Start. Delivery is at least once, so a run end delivered twice is counted twice; the counter is a rate, not a ledger. Only terminal statuses arrive here: a run in retry backoff (StatusRetrying) is not an end and is seen through RunQuery or admin instead.

func (*Instrumentation) Step

Step returns the step middleware: a span, a duration and an outcome per attempt. Put it outermost (first in middleware.Chain) so the span covers the other middleware and sees a recovered panic as an error. A ctx without a workflow.StepInfo still gets a span, with no workflow attributes.

type Option

type Option func(*config)

Option configures New.

func WithAllLabels

func WithAllLabels() Option

WithAllLabels records every job label (except LabelPrefix keys) as a span attribute. Use it only when no label holds personal data or secrets; WithLabelFilter picks labels by key instead.

func WithClock

func WithClock(clk clock.Clock) Option

WithClock sets the clock step durations are measured with. Nil means clock.Real.

func WithEnqueueSpanName

func WithEnqueueSpanName(fn func(workflow.Job) string) Option

WithEnqueueSpanName names enqueue spans with fn instead of "workflow.enqueue <workflow>". The same advice as WithStepSpanName applies; fn must not read the payload into the name.

func WithLabelFilter

func WithLabelFilter(keep func(key string) bool) Option

WithLabelFilter records the job labels for which keep reports true as span attributes. By default no label is recorded: labels can carry tenant or user ids, and a trace backend is often a third party, so recording them is opt-in, as Logging never logs them either. Keys under LabelPrefix are never recorded. A nil keep records none.

func WithMeterProvider

func WithMeterProvider(mp metric.MeterProvider) Option

WithMeterProvider sets where metrics go. Nil means otel.GetMeterProvider.

func WithMetricLabels

func WithMetricLabels(keys ...string) Option

WithMetricLabels adds the job labels named by keys to every metric as workflow.label.<key> attributes, for example a tenant label for per-tenant SLOs. Only the keys named are ever recorded, never all labels: each key multiplies the number of time series by its number of distinct values, so name only labels with a small, bounded set of values (a tier, a region, a few hundred tenants at most), never a user or order id. A job without a named label records it as "". Keys under LabelPrefix and empty keys are refused by New with workflow.ErrInvalid.

func WithParenting

func WithParenting(p Parenting) Option

WithParenting picks how a step span relates to its enqueue span; the default is ParentChild. New refuses any other value with workflow.ErrInvalid.

func WithPropagator

func WithPropagator(p propagation.TextMapPropagator) Option

WithPropagator sets how trace context is written into and read from job labels. Nil keeps the default, W3C TraceContext.

func WithStepSpanName

func WithStepSpanName(fn func(workflow.StepInfo) string) Option

WithStepSpanName names step spans with fn instead of "workflow.step <workflow>/<step>". fn sees the driver's StepInfo (zero when ctx has none); an empty result falls back to the default. A span name is indexed by most backends, so keep it to a bounded set of values: never put a run ID or a label value with unbounded values in it.

func WithTenant

func WithTenant() Option

WithTenant records Job.Tenant as the workflow.tenant attribute on step and enqueue spans. It is off by default, as labels are: a tenant ID can identify a customer. It is never a metric dimension (one series per tenant); WithMetricLabels is the way to opt into that, through a label.

func WithTracerProvider

func WithTracerProvider(tp trace.TracerProvider) Option

WithTracerProvider sets where spans go. Nil means otel.GetTracerProvider.

type Outcome

type Outcome string

Outcome is the value of the workflow.outcome attribute.

const (
	OutcomeOK        Outcome = "ok"
	OutcomeError     Outcome = "error"
	OutcomePermanent Outcome = "permanent"
	OutcomeCancelled Outcome = "cancelled"
	// OutcomeDiscarded is a step that returned workflow.Discard: the run
	// is dropped, not retried and not failed.
	OutcomeDiscarded Outcome = "discarded"
)

The outcomes a step attempt or an enqueue can have.

type Parenting

type Parenting string

Parenting is how a step span relates to the span that enqueued its run, whose context the step reads from the job's labels.

const (
	// ParentChild makes the step span a child of the enqueue span, so the
	// whole run is one trace with its caller. The default.
	ParentChild Parenting = "child"
	// ParentLink keeps whatever span ctx already holds as the parent (an
	// engine SDK's span, or none) and links the enqueue span, as the OTel
	// messaging conventions do for async consumers.
	ParentLink Parenting = "link"
	// ParentNewRootLink starts every step span as the root of its own
	// trace, linked to the enqueue span, so a fan-out of thousands of jobs
	// does not build one trace too large for the backend to keep.
	ParentNewRootLink Parenting = "new_root_link"
)

The parentings WithParenting accepts.

type QueueDepthFunc

type QueueDepthFunc func(ctx context.Context) (map[string]int64, error)

QueueDepthFunc returns how many runs wait in each queue, by queue name. admin.Admin.QueueDepths has this signature, so a method value plugs in directly; an app that can count with one SQL query passes its own.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL