Documentation
¶
Overview ¶
Package otel adds OpenTelemetry tracing and metrics to any workflow driver, as one step middleware and one enqueue middleware:
inst, err := otel.New(otel.WithTracerProvider(tp), otel.WithMeterProvider(mp)) c := memory.New(memory.WithMiddleware(inst.Step())) client := middleware.WrapClient(c, inst.Enqueue())
Spans ¶
Enqueue opens a span named "workflow.enqueue <workflow>" (kind Producer) around every Client.Enqueue, and Step opens one named "workflow.step <workflow>/<step>" (kind Consumer) around every step attempt. Both carry workflow.name and workflow.queue (see Metrics for the enqueue side); the step span also carries workflow.step, workflow.run_id and workflow.attempt, and the enqueue span workflow.run_id once the driver returns it. Job labels are not recorded by default, because they can carry tenant or user ids into a third-party trace backend; WithLabelFilter (chosen keys) or WithAllLabels opts in, recording them as workflow.label.<key>. Payloads and step outputs are never recorded: they can hold personal data, and a codec may not have run yet. WithStepSpanName and WithEnqueueSpanName replace the span names; keep a name to a bounded set of values.
Trace context propagation ¶
The enqueue middleware injects the caller's trace context into the job's labels and the step middleware extracts it, so a step span is a child of the span that enqueued its run even when the step runs on another machine. WithParenting changes that relation: ParentLink keeps the span ctx already holds (an engine SDK's) as the parent and links the enqueue span, as the OTel messaging conventions do; ParentNewRootLink starts each step span as a new trace linked to the enqueue span, so a fan-out of thousands of jobs does not build one trace too large to keep. The carrier prefixes every propagator key with LabelPrefix, so with the default W3C TraceContext propagator the labels are
otel.traceparent otel.tracestate (only when the trace has state)
The prefix is reserved for this package (it is the core's workflow.LabelPrefixOTel): Enqueue refuses a job that already sets a label under it with workflow.ErrInvalid, so use Enqueue once per chain. It is chosen so every driver accepts it: the core validates no label keys, and the Hatchet driver reserves only "wf__", "hatchet__" and the bare "traceparent" its SDK writes itself. The caller's Labels map is never modified; the middleware copies it before adding keys. When a step's labels carry no trace context, the step span's parent is whatever span ctx already holds.
Metrics ¶
workflow.step.duration (histogram, seconds): one per step attempt, with workflow.name, workflow.step, workflow.queue and workflow.outcome.
workflow.step.outcomes (counter): one per step attempt, same attributes.
workflow.enqueue.count (counter): one per Enqueue, with workflow.name, workflow.outcome and, when Job.Queue is set, workflow.queue.
workflow.queue.wait (histogram, seconds, QueueWaitBuckets): one per step attempt whose StepInfo.ReadyAt the driver set: attempt start minus ReadyAt, so for a run's first step it is enqueue (or RunAt) to start. With workflow.name, workflow.step and workflow.queue. A driver that leaves ReadyAt zero records no sample; a ReadyAt after the start (clock skew between nodes) records zero.
workflow.queue.depth (gauge): runs waiting per queue, one point per queue with workflow.queue, read through ObserveQueueDepth's QueueDepthFunc at every collection; admin.Admin.QueueDepths fits it.
workflow.runs.finished (counter): one per run end delivered to the observer RunFinished returns (register it with the driver's OnRunFinished), with workflow.name and workflow.run.status: "succeeded", "failed", "cancelled", or "discarded" on a driver with CapRunStates. Delivery is at least once, so a redelivered end is counted again. "retrying" never appears: it is not an end.
Step spans and step metrics carry workflow.queue (StepInfo.Queue); the enqueue span and counter carry the queue the job asked for (Job.Queue), and no workflow.queue when the job left the choice to the workflow, whose queue only the driver resolves. workflow.outcome is "ok", "error" (retryable), "permanent", "cancelled" or "discarded" (the step returned workflow.Discard). Run ids never become metric attributes, and labels do only when named one by one with WithMetricLabels (recorded as workflow.label.<key>): each named key multiplies the series by its number of values, so name only labels with a small, bounded set of values, such as a tier or a region.
Step spans also carry workflow.node (StepInfo.Node) when the driver names the process that ran the attempt. Job.Tenant is recorded as workflow.tenant on step and enqueue spans only with WithTenant, since a tenant ID can identify a customer, and is never a metric attribute.
Providers ¶
No global is read unless asked for: a nil or absent TracerProvider or MeterProvider means the otel global one (otel.GetTracerProvider, otel.GetMeterProvider), which is a no-op until the app installs one. The propagator defaults to W3C TraceContext, not the global propagator, whose default propagates nothing; WithPropagator replaces it.
Index ¶
- Constants
- func QueueWaitBuckets() []float64
- type Instrumentation
- type Option
- func WithAllLabels() Option
- func WithClock(clk clock.Clock) Option
- func WithEnqueueSpanName(fn func(workflow.Job) string) Option
- func WithLabelFilter(keep func(key string) bool) Option
- func WithMeterProvider(mp metric.MeterProvider) Option
- func WithMetricLabels(keys ...string) Option
- func WithParenting(p Parenting) Option
- func WithPropagator(p propagation.TextMapPropagator) Option
- func WithStepSpanName(fn func(workflow.StepInfo) string) Option
- func WithTenant() Option
- func WithTracerProvider(tp trace.TracerProvider) Option
- type Outcome
- type Parenting
- type QueueDepthFunc
Constants ¶
const ( AttrWorkflow = attribute.Key("workflow.name") AttrStep = attribute.Key("workflow.step") AttrRunID = attribute.Key("workflow.run_id") AttrAttempt = attribute.Key("workflow.attempt") AttrOutcome = attribute.Key("workflow.outcome") AttrQueue = attribute.Key("workflow.queue") // AttrNode is StepInfo.Node, the process that ran the attempt, on // step spans when the driver sets it. AttrNode = attribute.Key("workflow.node") // AttrTenant is Job.Tenant (StepInfo.Tenant) on step and enqueue // spans, recorded only with WithTenant. AttrTenant = attribute.Key("workflow.tenant") // AttrRunStatus is the terminal workflow.Status of a run on // MetricRunsFinished: succeeded, failed, cancelled, or discarded on a // driver with CapRunStates. AttrRunStatus = attribute.Key("workflow.run.status") LabelAttributePrefix = "workflow.label." )
Attribute keys. Labels are recorded under LabelAttributePrefix + key.
const ( MetricStepDuration = "workflow.step.duration" MetricStepOutcomes = "workflow.step.outcomes" MetricEnqueueCount = "workflow.enqueue.count" // MetricQueueWait is how long a step attempt waited for a worker: // from the moment it became ready (StepInfo.ReadyAt) to the moment it // started. For a run's first step that is enqueue to start. MetricQueueWait = "workflow.queue.wait" // MetricQueueDepth is how many runs wait in each queue, read through a // QueueDepthFunc at every collection. MetricQueueDepth = "workflow.queue.depth" // MetricRunsFinished counts run ends by workflow and AttrRunStatus, // fed by RunFinished. MetricRunsFinished = "workflow.runs.finished" )
Metric names.
const LabelPrefix = workflow.LabelPrefixOTel
LabelPrefix starts every job label this package writes to carry trace context; see the package doc for why this prefix. It is the core's workflow.LabelPrefixOTel, which this package owns.
const ScopeName = "github.com/ubgo/workflow/otel"
ScopeName is the instrumentation scope of every tracer and meter this package creates.
Variables ¶
This section is empty.
Functions ¶
func QueueWaitBuckets ¶
func QueueWaitBuckets() []float64
QueueWaitBuckets returns the bucket boundaries, in seconds, of MetricQueueWait: from 5ms to an hour, because a queue wait ranges from an idle worker picking a job up at once to a backlog of an hour, and the SDK's default boundaries (0 to 10000, meant for milliseconds) would put almost every wait in the first two buckets. An SDK View replaces them. It returns a new slice on every call, so no caller can change another's boundaries.
Types ¶
type Instrumentation ¶
type Instrumentation struct {
// contains filtered or unexported fields
}
Instrumentation holds the tracer, meter instruments and propagator the middleware share. Build it once with New and reuse it.
func New ¶
func New(opts ...Option) (*Instrumentation, error)
New builds the instrumentation. It fails when an option is invalid (workflow.ErrInvalid) or the meter provider refuses to create an instrument.
func (*Instrumentation) Enqueue ¶
func (i *Instrumentation) Enqueue() workflow.EnqueueMiddleware
Enqueue returns the enqueue middleware: a span and a count per Enqueue, and the span's trace context written into a copy of job.Labels under LabelPrefix so the run's steps continue the trace. A job that arrives with a label under LabelPrefix is refused with workflow.ErrInvalid, before any span or count, rather than having it silently replaced: it would otherwise let a caller choose the trace the run's steps join. So put Enqueue in a chain once.
func (*Instrumentation) ObserveQueueDepth ¶
func (i *Instrumentation) ObserveQueueDepth(fn QueueDepthFunc) (metric.Registration, error)
ObserveQueueDepth reports fn's depths as the workflow.queue.depth gauge, one data point per queue with a workflow.queue attribute, read at every metric collection (no poller of its own). An error from fn skips that collection's points and is handed to the OpenTelemetry error handler. Unregister the returned registration to stop. A nil fn is refused with workflow.ErrInvalid.
func (*Instrumentation) RunFinished ¶
func (i *Instrumentation) RunFinished() workflow.RunFinishedFunc
RunFinished returns a run-finished observer that counts each run end on MetricRunsFinished with the workflow name and AttrRunStatus. Register it with the driver's workflow.RunNotifier (OnRunFinished) before Start. Delivery is at least once, so a run end delivered twice is counted twice; the counter is a rate, not a ledger. Only terminal statuses arrive here: a run in retry backoff (StatusRetrying) is not an end and is seen through RunQuery or admin instead.
func (*Instrumentation) Step ¶
func (i *Instrumentation) Step() workflow.Middleware
Step returns the step middleware: a span, a duration and an outcome per attempt. Put it outermost (first in middleware.Chain) so the span covers the other middleware and sees a recovered panic as an error. A ctx without a workflow.StepInfo still gets a span, with no workflow attributes.
type Option ¶
type Option func(*config)
Option configures New.
func WithAllLabels ¶
func WithAllLabels() Option
WithAllLabels records every job label (except LabelPrefix keys) as a span attribute. Use it only when no label holds personal data or secrets; WithLabelFilter picks labels by key instead.
func WithEnqueueSpanName ¶
WithEnqueueSpanName names enqueue spans with fn instead of "workflow.enqueue <workflow>". The same advice as WithStepSpanName applies; fn must not read the payload into the name.
func WithLabelFilter ¶
WithLabelFilter records the job labels for which keep reports true as span attributes. By default no label is recorded: labels can carry tenant or user ids, and a trace backend is often a third party, so recording them is opt-in, as Logging never logs them either. Keys under LabelPrefix are never recorded. A nil keep records none.
func WithMeterProvider ¶
func WithMeterProvider(mp metric.MeterProvider) Option
WithMeterProvider sets where metrics go. Nil means otel.GetMeterProvider.
func WithMetricLabels ¶
WithMetricLabels adds the job labels named by keys to every metric as workflow.label.<key> attributes, for example a tenant label for per-tenant SLOs. Only the keys named are ever recorded, never all labels: each key multiplies the number of time series by its number of distinct values, so name only labels with a small, bounded set of values (a tier, a region, a few hundred tenants at most), never a user or order id. A job without a named label records it as "". Keys under LabelPrefix and empty keys are refused by New with workflow.ErrInvalid.
func WithParenting ¶
WithParenting picks how a step span relates to its enqueue span; the default is ParentChild. New refuses any other value with workflow.ErrInvalid.
func WithPropagator ¶
func WithPropagator(p propagation.TextMapPropagator) Option
WithPropagator sets how trace context is written into and read from job labels. Nil keeps the default, W3C TraceContext.
func WithStepSpanName ¶
WithStepSpanName names step spans with fn instead of "workflow.step <workflow>/<step>". fn sees the driver's StepInfo (zero when ctx has none); an empty result falls back to the default. A span name is indexed by most backends, so keep it to a bounded set of values: never put a run ID or a label value with unbounded values in it.
func WithTenant ¶
func WithTenant() Option
WithTenant records Job.Tenant as the workflow.tenant attribute on step and enqueue spans. It is off by default, as labels are: a tenant ID can identify a customer. It is never a metric dimension (one series per tenant); WithMetricLabels is the way to opt into that, through a label.
func WithTracerProvider ¶
func WithTracerProvider(tp trace.TracerProvider) Option
WithTracerProvider sets where spans go. Nil means otel.GetTracerProvider.
type Outcome ¶
type Outcome string
Outcome is the value of the workflow.outcome attribute.
const ( OutcomeOK Outcome = "ok" OutcomeError Outcome = "error" OutcomePermanent Outcome = "permanent" OutcomeCancelled Outcome = "cancelled" // OutcomeDiscarded is a step that returned workflow.Discard: the run // is dropped, not retried and not failed. OutcomeDiscarded Outcome = "discarded" )
The outcomes a step attempt or an enqueue can have.
type Parenting ¶
type Parenting string
Parenting is how a step span relates to the span that enqueued its run, whose context the step reads from the job's labels.
const ( // ParentChild makes the step span a child of the enqueue span, so the // whole run is one trace with its caller. The default. ParentChild Parenting = "child" // ParentLink keeps whatever span ctx already holds as the parent (an // engine SDK's span, or none) and links the enqueue span, as the OTel // messaging conventions do for async consumers. ParentLink Parenting = "link" // ParentNewRootLink starts every step span as the root of its own // trace, linked to the enqueue span, so a fan-out of thousands of jobs // does not build one trace too large for the backend to keep. ParentNewRootLink Parenting = "new_root_link" )
The parentings WithParenting accepts.
type QueueDepthFunc ¶
QueueDepthFunc returns how many runs wait in each queue, by queue name. admin.Admin.QueueDepths has this signature, so a method value plugs in directly; an app that can count with one SQL query passes its own.