Documentation
¶
Overview ¶
Package observability holds the adapters for logging, metrics and tracing.
This file is the logger. Structured JSON in operation, text for a developer's terminal, and a redaction layer that no caller has to remember (threat T-18): the log is written by many hands, including hands that pass an error along without reading it, and an error is where a connection string travels.
Index ¶
- Constants
- func NewCorrelatingHandler(inner slog.Handler) slog.Handler
- func NewLogger(cfg env.Config, w io.Writer) *slog.Logger
- func NewRedactingHandler(inner slog.Handler) slog.Handler
- func ResultClass(err error) string
- type Metrics
- func (m *Metrics) AiCall(ctx context.Context, kind, operation, result string)
- func (m *Metrics) AiTokens(ctx context.Context, kind, operation string, input, output int)
- func (m *Metrics) AuditDefaultAgedRows(ctx context.Context, rows int64)
- func (m *Metrics) AuditPartitionsHeld(ctx context.Context, months int64)
- func (m *Metrics) AuthFailure(ctx context.Context, reason string)
- func (m *Metrics) BackupLastSuccess(ctx context.Context, targetID string, at time.Time)
- func (m *Metrics) BusPublished(ctx context.Context, eventType string)
- func (m *Metrics) BusRefused(ctx context.Context, reason string)
- func (m *Metrics) CircuitBreakerState(ctx context.Context, dependency string, level int64)
- func (m *Metrics) ConfigInvalid(ctx context.Context, key string)
- func (m *Metrics) DataSubjectDeadline(ctx context.Context, stage string, count int)
- func (m *Metrics) DegradedMode(ctx context.Context, feature string, degraded bool)
- func (m *Metrics) DependencyUp(ctx context.Context, dependency string, up bool)
- func (m *Metrics) HTTPRequest(ctx context.Context, route, method string, status int, seconds float64)
- func (m *Metrics) Handler() http.Handler
- func (m *Metrics) InflightDelta(ctx context.Context, role string, delta int64)
- func (m *Metrics) JobDeadLettered(ctx context.Context, kind string)
- func (m *Metrics) JobFailed(ctx context.Context, kind, attemptClass string)
- func (m *Metrics) JobFinished(ctx context.Context, kind string, seconds float64)
- func (m *Metrics) JobsPurged(ctx context.Context, state string, count int64)
- func (m *Metrics) MediaReclaimFailed(ctx context.Context, count int64)
- func (m *Metrics) MediaReclaimed(ctx context.Context, count int64)
- func (m *Metrics) MigrationVersion(ctx context.Context, version int64)
- func (m *Metrics) NotificationFailed(ctx context.Context, category, channel, reason string)
- func (m *Metrics) NotificationRecorded(ctx context.Context, category, channel, state string)
- func (m *Metrics) NotificationSent(ctx context.Context, category, channel string, seconds float64)
- func (m *Metrics) OccurrenceMaterialized(ctx context.Context, mode string, lagSeconds float64)
- func (m *Metrics) OutboundHTTP(ctx context.Context, targetClass string, seconds float64)
- func (m *Metrics) OutboxLag(ctx context.Context, seconds float64)
- func (m *Metrics) PanicRecovered(ctx context.Context, component string)
- func (m *Metrics) PanicRecoveredBy(ctx context.Context, component string, delta int64)
- func (m *Metrics) PasswordRehashed(ctx context.Context)
- func (m *Metrics) PasswordResetRequested(ctx context.Context)
- func (m *Metrics) PoolConnections(ctx context.Context, pool string, inUse, idle, maxConns int64)
- func (m *Metrics) PullRecords(ctx context.Context, count int)
- func (m *Metrics) PushResult(ctx context.Context, result string)
- func (m *Metrics) QueueDepth(ctx context.Context, kind string, pending int64)
- func (m *Metrics) QuotaUsage(ctx context.Context, quota string, tenant string, ratio float64)
- func (m *Metrics) RateLimited(ctx context.Context, scope string)
- func (m *Metrics) ReminderFired(ctx context.Context, channel string, delaySeconds float64)
- func (m *Metrics) RetentionBlocked(ctx context.Context, reason string, count int64)
- func (m *Metrics) RetentionDeleted(ctx context.Context, dataKind string, count int64)
- func (m *Metrics) RetentionRead(ctx context.Context, dataKind string, count int64)
- func (m *Metrics) RetentionRun(ctx context.Context, dataKind string, seconds float64)
- func (m *Metrics) RuleDisabled(ctx context.Context, reason string)
- func (m *Metrics) RuleRun(ctx context.Context, result, triggerType string)
- func (m *Metrics) SchedulerTickLag(ctx context.Context, seconds float64)
- func (m *Metrics) SecretResealed(ctx context.Context, store, outcome string, count int64)
- func (m *Metrics) Shutdown(ctx context.Context) error
- func (m *Metrics) SignInStep(ctx context.Context, step string)
- func (m *Metrics) StreamClosed(ctx context.Context, seconds float64)
- func (m *Metrics) StreamOpened(ctx context.Context)
- func (m *Metrics) StreamRecords(ctx context.Context, count int)
- func (m *Metrics) StreamRefused(ctx context.Context, reason string)
- func (m *Metrics) UseCase(ctx context.Context, useCase, result string, tenant string)
- func (m *Metrics) WebhookDelivery(ctx context.Context, result, statusClass string)
- func (m *Metrics) WebhookRetryBacklog(ctx context.Context, waiting int64)
- type Observer
- type Tracing
Constants ¶
const Redacted = "[REDACTED]"
Redacted is what replaces a value that must not be readable. A fixed marker rather than an empty string, so that the log still shows the field was present.
const ResultOK = "ok"
ResultOK is the `result` value of a use case that succeeded. Every other value is an error category of the domain error model, lower-cased - see ResultClass.
Variables ¶
This section is empty.
Functions ¶
func NewCorrelatingHandler ¶
NewCorrelatingHandler adds trace_id, span_id, request_id, tenant_id and component from the context to every record.
It is a handler rather than a rule every call site has to remember, because the call sites are many and the one that forgets is the one being debugged. The fields are mandatory in observability-reliability.md §3.1 for exactly that reason: without them a log line and a trace are two unrelated pieces of evidence.
It belongs outside the redacting handler, so the attributes it adds pass the redaction like any other - a handler that writes straight to the sink would be a hole in T-18.
func NewLogger ¶
NewLogger builds the process logger from the configuration.
The handler chain is correlation → redaction → sink. Correlation goes outermost so that the trace_id and request_id it adds pass the redaction like any other attribute; a handler writing straight to the sink would be a hole in T-18.
func NewRedactingHandler ¶
NewRedactingHandler wraps a handler so that every attribute passes the redaction on its way through - including the attributes of With and WithGroup, which are applied once and then reused for the lifetime of a logger.
func ResultClass ¶
ResultClass is the `result` label of hubtask_usecase_total: ok, or the error category in lower case (observability-reliability.md §4.1).
It derives rather than translates, and that is the whole design. A mapping table would be a second classification of the same failure, and the two would disagree the moment the domain grows a tenth category - which would then be folded into `internal` by a `default` branch and vanish. Deriving means a new category appears in the metrics by itself.
The coarse view an alert wants is a query, not a label: `result=~"internal|unavailable"`. Folding rate_limited into internal here would save that regular expression and cost the distinction between a defect and a limit doing its job - which is the one an alert must not get wrong (§1, "alert on symptoms").
AsError normalises anything unclassified to INTERNAL, so the result is total: every error produces exactly one of the values the domain defines.
Types ¶
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics owns the meter and the instruments that exist before any use case does. Handing out an instrument rather than the meter is deliberate: an instrument created at a call site is created per call, and the label set of one created ad hoc is nobody's decision.
func NewMetrics ¶
NewMetrics builds the meter and registers what the process needs from the first second: the panic counter above all, whose value is the one an alert watches (ADR-0016).
func (*Metrics) AiCall ¶
AiCall counts one finished AI provider call. Three labels, all closed sets: the provider kind, the operation, and the result as the domain's own error category in lower case. The endpoint is not among them - it is a tenant's configuration, and a label per endpoint would grow a series per customer (§3.2, rule 10).
What is deliberately absent is anything a person wrote. The one place in this system where a title reaches something outside it is the call this counts, so the counting is the place the rule is easiest to break.
func (*Metrics) AiTokens ¶
AiTokens records what a call consumed, in the one unit every provider reports. It feeds the tenant's budget and an operator's sense of cost, and it is not a price - this project does not have one.
func (*Metrics) AuditDefaultAgedRows ¶
AuditDefaultAgedRows is how many entries in the trail's default partition are past their period.
func (*Metrics) AuditPartitionsHeld ¶
AuditPartitionsHeld is how many of the trail's aged months a hold keeps.
func (*Metrics) AuthFailure ¶
AuthFailure counts one refused sign-in or refresh attempt, by reason. The reasons are the closed set the identity service declares - never a subject, never an address (§3.2, rule 10).
func (*Metrics) BackupLastSuccess ¶
BackupLastSuccess publishes when a target last had a backup that worked - the number alert A-12 watches (backup-restore.md §10).
A Unix timestamp rather than an age, which is the convention every backup exporter follows and the reason it works: an age computed here is stale the moment it is scraped, while a timestamp lets the alert compute the age at evaluation time and stay right between scrapes.
Labelled by target and not by tenant. Which targets exist is bounded by configuration; how many tenants there are is not, and an unbounded label is how a metrics backend falls over (observability-reliability.md §3.2).
func (*Metrics) BusPublished ¶
BusPublished counts one event that reached the bus. The type is a closed set - the event types this build emits - and the subject is deliberately not a label: it carries a tenant identifier, and a series per tenant is how a Prometheus dies and how a workspace becomes visible in somebody else's dashboard (§3.2, rule 10).
func (*Metrics) BusRefused ¶
BusRefused counts one that did not, by reason. The reason is written by hand where it is raised and stays a short closed set: `unavailable` for a bus that could not be reached, `event_expired` for an event swept before its publication ran.
func (*Metrics) CircuitBreakerState ¶
CircuitBreakerState publishes the state of one breaker: 0 closed, 1 half-open, 2 open. The values come from resilience.BreakerState.Level(), which owns them - the gauge only reports what it is handed, so that the dashboard's contract has a single source (§4).
The composition root passes this as the breaker's OnStateChange hook. It takes the level rather than the state type, so that the metrics adapter does not have to know the resilience adapter: adapters do not know each other (project-structure.md §2).
func (*Metrics) ConfigInvalid ¶
ConfigInvalid counts a rejected configuration variable - misconfiguration as a signal rather than as guesswork in a support conversation.
func (*Metrics) DataSubjectDeadline ¶
DataSubjectDeadline reports cases whose statutory deadline is near or past - alert A-19's number (data-protection.md §4).
A counter by stage rather than a gauge of how many are open, and that is about what is true in provider operation: a gauge would need a tenant label to be true there, and an unlabelled one would carry the last workspace's number rather than the installation's. "A deadline is approaching somewhere" is true either way, and it is what the alert asks.
func (*Metrics) DegradedMode ¶
DegradedMode reports a feature as restricted, which is what makes a partial outage visible without reading the health endpoint.
func (*Metrics) DependencyUp ¶
DependencyUp mirrors the health report into a time series, so that a disruption has a beginning and an end in the same place as everything else.
func (*Metrics) HTTPRequest ¶
func (m *Metrics) HTTPRequest(ctx context.Context, route, method string, status int, seconds float64)
HTTPRequest records one served request. route is the template (/items/{id}), never the resolved path - the resolved one carries identifiers and would be unbounded.
func (*Metrics) Handler ¶
Handler serves the Prometheus endpoint. It belongs on the ops port, never on the public one (§3.2): the series say what runs, how much of it, and how slowly.
func (*Metrics) InflightDelta ¶
InflightDelta moves the gauge of requests currently in flight.
func (*Metrics) JobDeadLettered ¶
JobDeadLettered counts a job that used up its attempts (alert A-07).
func (*Metrics) JobFailed ¶
JobFailed records one failed attempt. attemptClass is retry or final, which is the distinction an alert needs: retries are the system working, and a final failure is a person's problem.
func (*Metrics) JobFinished ¶
JobFinished records a job that ran to the end. kind is the job kind, which is a closed set in core/port/queue - never a tenant, never an identifier.
func (*Metrics) JobsPurged ¶
JobsPurged counts the finished jobs the leader removed, by terminal state. Written at zero too, for RetentionDeleted's reason.
func (*Metrics) MediaReclaimFailed ¶
MediaReclaimFailed counts the orphans a pass could not reclaim. They stay marked and the next pass tries again, so a number that keeps rising is a bucket that is not letting go rather than a backlog that will clear.
func (*Metrics) MediaReclaimed ¶
MediaReclaimed counts what a pass removed. Written even when it is zero, for the reason RetentionDeleted is.
func (*Metrics) MigrationVersion ¶
MigrationVersion publishes the migration this build embeds, once at startup. It deliberately reports the build, not the database: every pod asks the same database, so the database's answer could never differ between pods, and differing is the whole signal (A-13).
func (*Metrics) NotificationFailed ¶
NotificationFailed counts a send that did not work, by the detail code that says why. The reason separates "the server is down" from "the address was refused", which are different problems with different fixes.
func (*Metrics) NotificationRecorded ¶
NotificationRecorded counts a record as it is written, in the state it was written in. The state is the point: a rising SUPPRESSED count is people switching things off, and a rising PENDING count that never becomes SENT is a mail server nobody has noticed.
func (*Metrics) NotificationSent ¶
func (m *Metrics) NotificationSent( ctx context.Context, category, channel string, seconds float64, )
NotificationSent records how long one message took to leave.
func (*Metrics) OccurrenceMaterialized ¶
OccurrenceMaterialized records how late an occurrence was created against the moment it is for (the lag ADR-0008 promised).
The mode is the label because the two answer different questions: an ON_SCHEDULE series that is behind is a scheduler problem, while an ON_COMPLETION one is behind only for as long as nobody completed anything - which is a person's business rather than an operator's.
func (*Metrics) OutboundHTTP ¶
OutboundHTTP records the duration of one outbound call. targetClass is a class, never a host: a webhook target is chosen by a tenant, and a label per URL would grow a series per customer integration (§3.2, rule 10).
func (*Metrics) OutboxLag ¶
OutboxLag records the age of an event at delivery, which is what SLO-4 promises and what alert A-05 watches. Deliberately without a tenant label even when the tenant label is enabled: it is a histogram, and one per tenant multiplies buckets rather than series.
func (*Metrics) PanicRecovered ¶
PanicRecovered counts a caught panic. component is the location, not the message: a panic message can contain anything, including user content (rule 10).
func (*Metrics) PanicRecoveredBy ¶
PanicRecoveredBy exists for the zero seeding above; a delta of anything but 0 or 1 would be a misuse, and the counter is not the place to be clever.
func (*Metrics) PasswordRehashed ¶
PasswordRehashed counts one stored hash brought up to the current cost. No label: the counter shows how far an upgrade of the cost has reached, never whose.
func (*Metrics) PasswordResetRequested ¶
PasswordResetRequested counts one forgotten-password request. No label: which ones reached an account is what the request must not reveal, and a counter split by it would.
func (*Metrics) PoolConnections ¶
PoolConnections publishes one pool's connection states. The max joins in_use and idle as a third state rather than a separate metric, because it is the denominator A-11 divides by and belongs on the same series a dashboard already reads. The pool label tells the API's pool from the background one - their saturation means different things (§6).
func (*Metrics) PullRecords ¶
PullRecords counts what a pull handed out. Beside the stream's counter rather than folded into it: the two together say whether devices keep up with the log, and apart, by which door.
func (*Metrics) PushResult ¶
PushResult counts one mutation a push answered, by the contract's result - a closed set of four. What a device pushed and how much of it the server took is the one number that says whether offline work is landing.
func (*Metrics) QueueDepth ¶
QueueDepth publishes the backlog of one kind. Only the scheduler leader reports it: the number describes the installation, and one written by every replica would be summed across instances and read as several times the truth.
func (*Metrics) QuotaUsage ¶
QuotaUsage records how close one workspace stands to one of its ceilings. The label is the quota - a bounded set of six - and the tenant joins it only when the operator enabled the tenant label (§3.2): without it the series carries the last workspace's number per quota, which is still the alarm A-18 needs ("somebody stands at 90%"), just not the who.
func (*Metrics) RateLimited ¶
RateLimited counts a call turned away by a limit. scope says which limit: ip, token, tenant for the rate limiter, load_shed for the shedder, bulkhead:<compartment> for a full compartment. The set is written by hand and stays small - that is what keeps it a label.
func (*Metrics) ReminderFired ¶
ReminderFired records how late one reminder was against the moment it promised (SLO-5).
The delay rather than the send: a reminder that left on time and then waited an hour for a mail server is a different failure, and the notification metrics already measure that one. What this answers is whether the schedule kept its promise.
func (*Metrics) RetentionBlocked ¶
RetentionBlocked counts what a pass kept, by reason. The reason is the point: "twelve were kept" is not something an operator can act on, and "twelve were kept by a legal hold" is.
func (*Metrics) RetentionDeleted ¶
RetentionDeleted counts what a pass removed. Written even when it is zero: a counter that has never been written has no series, and an alert on a deletion run that never happens is one that reads "no data" and is believed (observability-reliability.md §4).
func (*Metrics) RetentionRead ¶
RetentionRead counts what the rule sweep judged in a pass. Written even when it is zero, for the reason RetentionDeleted is.
func (*Metrics) RetentionRun ¶
RetentionRun records how long one pass took.
func (*Metrics) RuleDisabled ¶
RuleDisabled counts one rule switching itself off. The reason is the engine's, and there is exactly one today; the label exists so a second protection would be told apart from the first.
func (*Metrics) RuleRun ¶
RuleRun counts one ended automation run. Both values come from the engine's closed sets - the domain's status vocabulary and the six trigger kinds - never a rule or a tenant (§3.2).
func (*Metrics) SchedulerTickLag ¶
SchedulerTickLag publishes how late the last tick was. Seconds as a whole number: the gauge exists to show drift, and a schedule that is off by less than a second is not drifting.
func (*Metrics) SecretResealed ¶
SecretResealed counts what one re-seal pass did in one store. Both labels are closed sets written by hand: the store is one of the five places a sealed value lives, the outcome is `rewrapped` or `skipped` - the latter a value naming a key the ring no longer holds, which is the state the operator reads from the census rather than from here.
func (*Metrics) Shutdown ¶
Shutdown flushes the meter. Metrics missing from the last seconds of a pod's life are exactly the ones an incident review wants.
func (*Metrics) SignInStep ¶
SignInStep counts one step a sign-in was handed, by step - a closed set, never a subject.
func (*Metrics) StreamClosed ¶
StreamClosed gives the connection back and records how long it lasted.
func (*Metrics) StreamOpened ¶
StreamOpened counts a connection this process has taken on.
func (*Metrics) StreamRecords ¶
StreamRecords counts what went down the streams. Against the change log's own growth it is what says whether the streams are keeping up.
func (*Metrics) StreamRefused ¶
StreamRefused counts a connection that was not taken on, by the limit that refused it. The reason is the point: "twelve refused" is not something an operator can act on, and "twelve refused by the per-credential cap" is.
func (*Metrics) UseCase ¶
UseCase records the outcome of a use case. The result is the error category, not the message: ok, or one of the domain's categories in lower case (§4.1) - see ResultClass, which is the only thing that should be producing this value.
func (*Metrics) WebhookDelivery ¶
WebhookDelivery counts one settled attempt. Both labels are closed sets the deliverer owns: what became of the attempt, and the class of the answer - never a target, never a tenant (§3.2, rule 10: a label per URL would grow a series per customer integration).
type Observer ¶
type Observer struct {
// contains filtered or unexported fields
}
Observer gives one use case execution its metric and its span - the two signals the Definition of Done requires of every use case, and what gate RT-12 checks for (ADR-0016 item 6).
It is a wrapper rather than two calls at each call site, because two calls are two chances to forget one, and the one that gets forgotten is the error path.
func NewObserver ¶
func (*Observer) Job ¶
Job runs one job execution as an observed unit (ADR-0008, §3.3).
It is the queue's counterpart to UseCase, and it exists for the same reason: a job that produced no span is a job nobody can follow through the pipeline dashboard, and the failure that most needs following is the one that happened at three in the morning.
What it does not do yet is continue the trace of whatever caused the job. §3.3 wants the traceparent persisted in the event and the job row, so that a chain of HTTP request -> event -> automation rule -> webhook is one trace; that needs a column and belongs with the dispatcher work in 0.5.0. Until then a job span is its own root, which is enough to see what a job did and how long it took, and not enough to see what asked for it.
func (*Observer) Registry ¶
func (o *Observer) Registry() usecase.Middleware
Registry returns the middleware the use case catalogue is built with.
This is what makes gate RT-12 structural rather than a rule people remember: the composition root passes it to usecase.NewRegistry, every entry's handler is wrapped on the way in, and there is then no path to a use case that produces neither a metric nor a span - whether the call arrived through REST, through MCP, or from an automation rule.
type Tracing ¶
type Tracing struct {
Enabled bool
// contains filtered or unexported fields
}
Tracing owns the tracer provider. It exists whether or not tracing is enabled: the W3C propagator is installed either way, so an incoming traceparent still reaches the log and still travels onwards to the next service (§3.3). Turning tracing off means "export nothing", not "lose the correlation".
func NewTracing ¶
NewTracing builds the tracer provider from the configuration and installs it globally, along with the W3C trace context and baggage propagators.
Disabled is the default and the documented self-hosting profile (§13): a no-op provider, no exporter, no goroutine, and no cost per span - but the propagator is still there.
Source Files