obs

package
v1.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package obs holds the tracing and metrics plumbing every Ctrl Plane subsystem shares: the tracer, the span helpers that keep error details off spans, and the labelled metric families.

Spans go to the global OpenTelemetry tracer provider, which forge.New installs. Library level work (provider calls, deployments, renders, managed transactions) only starts a span when the context already carries one, so a background poller that has no trace of its own never mints a root trace per tick. The background workers start their own root spans, with trace.WithNewRoot, once they know there is work.

Metrics are recorded through a go-utils metrics sink. In a Forge app that is app.Metrics(), which the ctrlplane extension passes in. A nil sink records nothing.

Index

Constants

View Source 
const (
	AttrProviderType      = attribute.Key("ctrlplane.provider.type")
	AttrProviderOperation = attribute.Key("ctrlplane.provider.operation")
	AttrProviderHealthy   = attribute.Key("ctrlplane.provider.healthy")
	AttrProviderNotFound  = attribute.Key("ctrlplane.provider.not_found")
	AttrProvidersChecked  = attribute.Key("ctrlplane.providers.checked")
	AttrProvidersFailing  = attribute.Key("ctrlplane.providers.unhealthy")
	AttrSourceType        = attribute.Key("ctrlplane.source.type")
	AttrInstanceID        = attribute.Key("ctrlplane.instance.id")
	AttrDatacenterID      = attribute.Key("ctrlplane.datacenter.id")

	AttrDeploymentID       = attribute.Key("ctrlplane.deployment.id")
	AttrDeploymentStrategy = attribute.Key("ctrlplane.deployment.strategy")
	AttrDeploymentRollback = attribute.Key("ctrlplane.deployment.rollback")
	AttrDeploymentServices = attribute.Key("ctrlplane.deployment.services")

	AttrReadinessPolls   = attribute.Key("ctrlplane.readiness.polls")
	AttrReadinessOutcome = attribute.Key("ctrlplane.readiness.outcome")

	AttrReconcileDatacenters = attribute.Key("ctrlplane.reconcile.datacenters")
	AttrReconcileFailed      = attribute.Key("ctrlplane.reconcile.failed")

	AttrBootstrapID          = attribute.Key("ctrlplane.bootstrap.id")
	AttrBootstrapHook        = attribute.Key("ctrlplane.bootstrap.hook")
	AttrBootstrapDesired     = attribute.Key("ctrlplane.bootstrap.desired")
	AttrBootstrapProvisioned = attribute.Key("ctrlplane.bootstrap.provisioned")
	AttrBootstrapRetired     = attribute.Key("ctrlplane.bootstrap.retired")
	AttrBootstrapFailed      = attribute.Key("ctrlplane.bootstrap.failed")

	AttrGCTenants       = attribute.Key("ctrlplane.gc.tenants")
	AttrGCTenantsFailed = attribute.Key("ctrlplane.gc.tenants_failed")
	AttrGCReaped        = attribute.Key("ctrlplane.gc.reaped")

	AttrHealthCheckID   = attribute.Key("ctrlplane.health.check.id")
	AttrHealthCheckType = attribute.Key("ctrlplane.health.check_type")
	AttrHealthStatus    = attribute.Key("ctrlplane.health.status")

	AttrWorkloadID      = attribute.Key("ctrlplane.workload.id")
	AttrWorkloadReplica = attribute.Key("ctrlplane.workload.replica")
	AttrReplicaAdopted  = attribute.Key("ctrlplane.workload.replica_adopted")

	AttrManagedStep    = attribute.Key("ctrlplane.managed.step")
	AttrManagedRefusal = attribute.Key("ctrlplane.managed.refusal")
)

Span attribute keys. Values are counts, booleans, enums and opaque resource IDs. Tenant, subject and user identifiers, names, URLs, error messages and payloads never go on a span.

View Source 
const (
	MetricWorkerRunDuration    = "ctrlplane_worker_run_duration_seconds"
	MetricWorkerRunsSkipped    = "ctrlplane_worker_runs_skipped_total"
	MetricSchedulerLeader      = "ctrlplane_scheduler_leader"
	MetricLeaderChecks         = "ctrlplane_leader_checks_total"
	MetricDatacenterReconciles = "ctrlplane_datacenter_reconciles_total"
	MetricBootstrapActions     = "ctrlplane_bootstrap_actions_total"
	MetricGCTenantSweeps       = "ctrlplane_gc_tenant_sweeps_total"
	MetricGCInstancesReaped    = "ctrlplane_gc_instances_reaped_total"
	MetricProviderCallDuration = "ctrlplane_provider_call_duration_seconds"
	MetricResolverDuration     = "ctrlplane_provider_credentials_duration_seconds"
	MetricReadinessWait        = "ctrlplane_readiness_wait_duration_seconds"
	MetricDeploymentDuration   = "ctrlplane_deployment_duration_seconds"
	MetricProviderHealthChecks = "ctrlplane_provider_health_check_duration_seconds"
	MetricResourceSamples      = "ctrlplane_resource_samples_total"
	MetricResourcePollers      = "ctrlplane_resource_pollers"
	MetricHealthCheckDuration  = "ctrlplane_health_check_duration_seconds"
	MetricLifecycleEvents      = "ctrlplane_lifecycle_events_total"
	MetricPluginHookErrors     = "ctrlplane_plugin_hook_errors_total"
)

Metric names. Forge exports a name that isn't an OTel semantic convention under its namespace, so ctrlplane_worker_run_duration_seconds scrapes as forge_ctrlplane_worker_run_duration_seconds. Forge adds no unit suffix to such names, which is why _seconds and _total are spelled out here.

View Source 
const (
	LabelWorker    = "worker"
	LabelResult    = "result"
	LabelAction    = "action"
	LabelProvider  = "provider"
	LabelOperation = "operation"
	LabelStrategy  = "strategy"
	LabelCheckType = "check_type"
	LabelStatus    = "status"
	LabelEvent     = "event"
	LabelHook      = "hook"
)

Label keys. Every series of a family carries the same keys.

View Source 
const (
	LeaderLeader   = "leader"
	LeaderFollower = "follower"
)

Values of the leader check result label.

View Source 
const (
	ActionProvision = "provision"
	ActionRetire    = "retire"
)

Values of the bootstrap action label.

View Source 
const (
	ActionDeploy   = "deploy"
	ActionRollback = "rollback"
)

Values of the deployment action label.

View Source 
const (
	HealthHealthy   = "healthy"
	HealthUnhealthy = "unhealthy"
)

Values of the provider health check result label.

View Source 
const (
	ResultOK    = "ok"
	ResultError = "error"
)

Values of the result label.

View Source 
const (
	OpProvision   = "provision"
	OpDeprovision = "deprovision"
	OpStart       = "start"
	OpStop        = "stop"
	OpRestart     = "restart"
	OpScale       = "scale"
	OpDeploy      = "deploy"
	OpStatus      = "status"
	OpLogs        = "logs"
	OpResources   = "resources"
)

Provider operations, the values of the operation label and of the ctrlplane.provider.operation span attribute.

View Source 
const InstrumentationName = "github.com/xraph/ctrlplane"

InstrumentationName is the instrumentation scope on every span Ctrl Plane starts.

View Source 
const ProviderCustom = "custom"

ProviderCustom is the provider label value for every provider that isn't one of the built-in ones.

View Source 
const ResultNotFound = "not_found"

ResultNotFound is the provider call result for a readiness status read that found nothing yet.

View Source 
const SampleEmpty = "empty"

SampleEmpty is the resource sample result label value for a provider that had nothing to report.

Variables

This section is empty.

Functions

func Detach 

func Detach(ctx context.Context) context.Context

Detach returns ctx without the span it carries, keeping its values and cancellation. Long-lived goroutines started from a lifecycle hook use it: Forge hands Start a context holding the boot span, and anything traced under it later would join the boot trace.

func ErrorType 

func ErrorType(err error, extra ...Sentinel) string

ErrorType returns a short, value free class for err: the name of a known sentinel it wraps (the shared ones, then any in extra), or else the Go type of the innermost error in its chain. It is safe on a span and as a span status description. It returns "" for a nil err.

func Fail 

func Fail(span trace.Span, err error, extra ...Sentinel)

Fail marks span failed. The status description and the exception event carry only the error's class (see ErrorType), never its message: driver and provider messages quote names, addresses and payload fragments.

extra names package specific sentinels for ErrorType.

func FailClass 

func FailClass(span trace.Span, class string)

FailClass marks span failed with class as both the status description and the exception type. Use it for failures that are outcomes rather than Go errors, such as an unhealthy probe.

func IsNotFound 

func IsNotFound(err error) bool

IsNotFound reports whether err says the thing asked about doesn't exist (yet): it wraps ctrlplane.ErrNotFound, or carries the NotFound() marker method errdefs-style errors use, as the Docker SDK's do. Errors that only spell it out in their text, or Kubernetes API errors, aren't recognised.

func MethodSpanName 

func MethodSpanName(_ string, req *http.Request) string

MethodSpanName names a client span after the request method alone. The Docker SDK's default adds the URL path, which carries container IDs.

func ProviderLabel 

func ProviderLabel(providerType string) string

ProviderLabel maps a provider's Info().Name to the provider label: docker, kubernetes and nomad stay as they are, and anything else, which a host chooses, becomes custom. Spans keep the real name.

func RedactedTracerProvider 

func RedactedTracerProvider() trace.TracerProvider

RedactedTracerProvider returns a tracer provider for HTTP instrumentation that lives inside a third-party SDK, such as the otelhttp transport the Docker client always installs. Spans it starts go to the global tracer provider, looked up on every span, but keep only the method, status code, server address and port and error.type. A failed span's status description is replaced by its error.type, or left empty, because otelhttp writes err.Error() there and a *url.Error quotes the full URL. Recorded errors become an exception event carrying only exception.type.

func Result 

func Result(err error) string

Result is the result label for err: "ok" or "error".

func Start 

func Start(ctx context.Context, name string, opts ...trace.SpanStartOption) (context.Context, trace.Span)

Start starts a span under whatever span ctx carries, or a root span when it carries none. End it when the work is done.

func StartChild 

func StartChild(ctx context.Context, name string, opts ...trace.SpanStartOption) (context.Context, trace.Span)

StartChild starts a span only when ctx already carries a valid one. Otherwise it returns ctx unchanged and the non-recording span ctx holds, so the caller can use the result the same way either way.

func StartRoot 

func StartRoot(ctx context.Context, name string, opts ...trace.SpanStartOption) (context.Context, trace.Span)

StartRoot starts a span that begins a new trace, whatever ctx carries. Background workers use it for one unit of work so the span never hangs off the boot span of the context the worker was started with.

func Traced 

func Traced(req *http.Request) bool

Traced reports whether req was made under a span. Clients that bring their own otelhttp transport, like the Docker SDK, use it as the transport's filter so an untraced poll doesn't start a root trace.

func Tracer 

func Tracer() trace.Tracer

Tracer returns the Ctrl Plane tracer from the global provider. It is looked up on each call so a provider installed after construction, as forge.New does, still receives the spans.

func Transport 

func Transport(rt http.RoundTripper) http.RoundTripper

Transport wraps rt so a request made under a span gets a client span and carries the trace context, without any inbound baggage. A request whose context has no span goes out untouched, so a poller with no trace of its own adds nothing. A nil rt means http.DefaultTransport.

The span is named after the method and records the method, the server address and port and the response status. It never records the URL path or query: on a cluster API those name tenant namespaces and workloads.

Types

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics records Ctrl Plane's metric families on a go-utils sink. Build it with NewMetrics; a nil *Metrics is valid and records nothing, so every subsystem can hold one whether metrics are wired or not.

func NewMetrics 

func NewMetrics(m gometrics.MetricFactory) *Metrics

NewMetrics declares Ctrl Plane's metric families on m. Labelled series are created on first use. A nil m returns nil, which records nothing.

func (*Metrics) BootstrapAction 

func (m *Metrics) BootstrapAction(action string, err error)

BootstrapAction records a bootstrap workload provision or retirement.

func (*Metrics) CredentialsResolved 

func (m *Metrics) CredentialsResolved(took time.Duration, err error)

CredentialsResolved records one credentials resolver call.

func (*Metrics) DatacenterReconciled 

func (m *Metrics) DatacenterReconciled(err error)

DatacenterReconciled records one datacenter's bootstrap reconcile.

func (*Metrics) DeploymentFinished 

func (m *Metrics) DeploymentFinished(strategy, action string, took time.Duration, err error)

DeploymentFinished records one deployment or rollback.

func (*Metrics) GCInstanceReaped 

func (m *Metrics) GCInstanceReaped(err error)

GCInstanceReaped records one orphan delete attempt.

func (*Metrics) GCTenantSwept 

func (m *Metrics) GCTenantSwept(err error)

GCTenantSwept records one tenant's orphan sweep.

func (*Metrics) HealthChecked 

func (m *Metrics) HealthChecked(checkType, status string, took time.Duration)

HealthChecked records one instance health check. status is the result's health status, or error when the checker failed to run.

func (*Metrics) LeaderCheck 

func (m *Metrics) LeaderCheck(leader bool, err error)

LeaderCheck records one leader election answer: leader, follower, or error when the elector could not tell. The scheduler leader gauge follows the answer, with an error counting as not leading.

func (*Metrics) LifecycleEvent 

func (m *Metrics) LifecycleEvent(eventType string)

LifecycleEvent counts one lifecycle event delivered to plugins.

func (*Metrics) PluginHookFailed 

func (m *Metrics) PluginHookFailed(hook string)

PluginHookFailed counts one plugin hook that returned an error.

func (*Metrics) ProviderHealthChecked 

func (m *Metrics) ProviderHealthChecked(providerType, result string, took time.Duration)

ProviderHealthChecked records one provider health check. result is healthy, unhealthy, or error when the check itself failed.

func (*Metrics) ReadinessWaited 

func (m *Metrics) ReadinessWaited(outcome string, took time.Duration)

ReadinessWaited records one readiness wait and how it ended: ready, failed, timeout or canceled (provider.ReadinessOutcome).

func (*Metrics) ResourceSampled 

func (m *Metrics) ResourceSampled(result string)

ResourceSampled records one metrics poller sample: ok, empty when the provider had nothing to report, or error.

func (*Metrics) SetLeader 

func (m *Metrics) SetLeader(leader bool)

SetLeader sets the scheduler leader gauge without counting a check. The scheduler uses it when no elector is configured, and when it resigns.

func (*Metrics) SetResourcePollers 

func (m *Metrics) SetResourcePollers(n int)

SetResourcePollers sets how many instances the metrics poller tracks.

func (*Metrics) StartProviderCall 

func (m *Metrics) StartProviderCall(ctx context.Context, providerType, op string, attrs ...attribute.KeyValue) (context.Context, *ProviderCall)

StartProviderCall times one provider operation and, when ctx carries a span, traces it as a child. providerType is the provider's Info().Name and op one of the Op constants. m may be nil.

The returned context has no baggage. Providers call cloud and cluster APIs, often through instrumented clients that would forward whatever baggage a caller sent in, and that baggage often names users or tenants.

func (*Metrics) WorkerRun 

func (m *Metrics) WorkerRun(worker string, took time.Duration, err error)

WorkerRun records one worker pass. result is ok or error.

func (*Metrics) WorkerSkipped 

func (m *Metrics) WorkerSkipped(worker string)

WorkerSkipped records a tick a worker skipped because this replica did not lead.

type ProviderCall 

type ProviderCall struct {
	// contains filtered or unexported fields
}

ProviderCall is one provider operation in flight. End it with the call's error.

func (*ProviderCall) End 

func (c *ProviderCall) End(err error)

End records the call's duration and result and ends its span.

func (*ProviderCall) EndPoll 

func (c *ProviderCall) EndPoll(err error)

EndPoll ends a status read made while waiting for readiness. Right after a platform accepts a request it often answers "not found" for a moment, and the wait retries; such a read (IsNotFound) is recorded with result not_found and leaves the span unmarked. Any other error ends the call as End would.

type Sentinel 

type Sentinel struct {
	Err  error
	Name string
}

Sentinel names an error value ErrorType should report by name instead of by its Go type.

Source Files

View all Source files

Directories

Path Synopsis
Package obstest holds the test doubles for Ctrl Plane's telemetry: a span recorder installed as the global tracer provider, a metrics sink that remembers every series by name and labels, and a check that sensitive values never reach a span.
 Package obstest holds the test doubles for Ctrl Plane's telemetry: a span recorder installed as the global tracer provider, a metrics sink that remembers every series by name and labels, and a check that sensitive values never reach a span.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.