Documentation
¶
Overview ¶
Package observability wires up logging and metrics.
Index ¶
- func ContextWithLogger(ctx context.Context, l *slog.Logger) context.Context
- func LoggerFrom(ctx context.Context) *slog.Logger
- func NewIngestCollector(stats IngestStats) prometheus.Collector
- func NewLimiterCollector(limiters map[string]LimiterStats) prometheus.Collector
- func NewLogger(c config.Config, w *os.File) *slog.Logger
- func NewPoolCollector(pools map[string]*pgxpool.Pool) prometheus.Collector
- func SetWebPaths(paths []string)
- type AddonFetchOutcome
- type AddonFetchStats
- type AddonKills
- type AddonPerformance
- type AddonRedirectStats
- type IngestStats
- type LimiterStats
- type Metrics
- func (m *Metrics) AddonPerformance() map[string]AddonPerformance
- func (m *Metrics) ForgetAddon(addon, version string, abiVersion int, failureClass, permissions string)
- func (m *Metrics) Gather() *prometheus.Registry
- func (m *Metrics) HTTPMiddleware(next http.Handler) http.Handler
- func (m *Metrics) Handler() http.Handler
- func (m *Metrics) ObserveAddonFetch(addon, outcome string, d time.Duration)
- func (m *Metrics) ObserveAddonLoad(addon, outcome string)
- func (m *Metrics) ObserveAddonRedirect(addon, class string, d time.Duration)
- func (m *Metrics) ObserveAddonRedirectKill(addon, step string)
- func (m *Metrics) ObserveAddonRefusal(addon, permission string)
- func (m *Metrics) ObserveAutomationFiring(trigger, outcome string)
- func (m *Metrics) ObserveFeedCheck(result string)
- func (m *Metrics) ObserveJob(job string, err error)
- func (m *Metrics) ObserveJobSkipped(job string)
- func (m *Metrics) ObserveRedirect(outcome, cache string, d time.Duration)
- func (m *Metrics) ObserveThrottled(limit string)
- func (m *Metrics) ObserveWebhookDelivery(outcome, status string)
- func (m *Metrics) Register(c prometheus.Collector)
- func (m *Metrics) SetAddonInfo(addon, version string, abiVersion int, failureClass, permissions string)
- func (m *Metrics) SetAddonLargeObjects(addon string, n int64)
- func (m *Metrics) SetAddonSchemaBytes(addon string, n int64)
- func (m *Metrics) SetAuditLogBytes(n int64)
- func (m *Metrics) SetJobStaleness(job string, seconds float64)
- type Surface
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ContextWithLogger ¶
ContextWithLogger returns a context carrying the given logger.
func LoggerFrom ¶
LoggerFrom returns the request-scoped logger, or the default logger when there is none. It never returns nil, so callers need no nil check and a missing middleware degrades to unattributed logs rather than a panic.
func NewIngestCollector ¶
func NewIngestCollector(stats IngestStats) prometheus.Collector
NewIngestCollector reports the click pipeline's counters.
func NewLimiterCollector ¶
func NewLimiterCollector(limiters map[string]LimiterStats) prometheus.Collector
NewLimiterCollector reports the named limiters' bookkeeping.
None of these is about throttling — linkctrl_rate_limited_total covers that. They answer a different question: is the limiter still able to do its job. A climbing overflow count means it is not, and that failure is otherwise completely silent, because the design choice on a full table is to allow the request.
Disabled limits must be left out of the map by the caller rather than passed as a nil pointer: a nil pointer inside an interface is not a nil interface, so it would be collected as a working limiter reporting zeros — which reads as "enforcing, and nothing to report" instead of "off".
func NewLogger ¶
NewLogger builds the application logger.
JSON in production so logs are machine-readable; text in development because a human is reading them. Every record carries the service name and version, so logs from two versions during a rolling update are distinguishable.
func NewPoolCollector ¶
func NewPoolCollector(pools map[string]*pgxpool.Pool) prometheus.Collector
NewPoolCollector reports on the named pools.
Both pools are labelled separately because the entire point of splitting them is that they saturate independently: the alert worth having is "the redirect pool is exhausted", which an aggregate number hides.
func SetWebPaths ¶ added in v0.2.0
func SetWebPaths(paths []string)
SetWebPaths replaces the dashboard path set with the routes the application mux was given.
Called once at boot, before any request is served. Paths arrive as the mux spells them — an exact path like `/login`, or a subtree like `/links/` — and both are reduced to the prefix form this classifier matches on. The root pattern is dropped because `/` is handled explicitly below.
Types ¶
type AddonFetchOutcome ¶ added in v0.4.0
AddonFetchOutcome is one word of the vocabulary and how often it happened.
type AddonFetchStats ¶ added in v0.4.0
type AddonFetchStats struct {
// Count is how many requests this host attempted, which is the population the
// estimate below is over. A refusal this host decided itself — an unnamed
// origin, an address the policy will not dial — is in Outcomes and deliberately
// not here.
Count uint64 `json:"count"`
// Refused is every outcome but `ok`, added up. One number because the page has
// one line for it and the breakdown is Outcomes.
Refused uint64 `json:"refused"`
// Outcomes is each word of abi.FetchOutcomes this add-on has produced, with how
// often — ordered so a page does not reorder its own rows between renders.
Outcomes []AddonFetchOutcome `json:"outcomes,omitempty"`
P99 time.Duration `json:"-"`
Sum time.Duration `json:"-"`
P99Seconds float64 `json:"p99_seconds"`
SumSeconds float64 `json:"sum_seconds"`
}
AddonFetchStats is one add-on's outbound-request record (M68.5).
Beside the redirect figures rather than folded into them, because they are two different paths with two different bounds: a redirect invocation is priced in milliseconds against a deadline this product sets, and an outbound request is priced against a server somebody else runs.
func (AddonFetchStats) Observed ¶ added in v0.4.0
func (f AddonFetchStats) Observed() bool
Observed reports whether this add-on has made an outbound request at all. A module holding no egress grant has none, and the page draws nothing rather than zeros — the same rule the redirect figures follow.
All three counts, rather than the two that are drawn: every field this struct carries is written beside one of them, so *unobserved* and *all-zero* are the same state and AddonPerformance.IsZero can rest on that. Outcomes is the one that would otherwise be reachable alone — a counter incremented for `ok` with no duration observed beside it.
type AddonKills ¶ added in v0.4.0
AddonKills is how many invocations of one add-on the host stopped waiting for, split by the step whose bound they overran (F326's split — the two are different faults with different owners).
func (AddonKills) Total ¶ added in v0.4.0
func (k AddonKills) Total() uint64
Total is what the manager's list column shows: one number, because the list has one column and the split belongs on the detail page.
type AddonPerformance ¶ added in v0.4.0
type AddonPerformance struct {
// Classes is one entry per class this add-on has actually been observed in,
// ordered inline-then-observe. **A class with no observations is absent rather
// than zero**, which is what makes m68.md's "modules holding no redirect grant
// show no redirect figures rather than zeros" expressible: an add-on that never
// ran on the redirect path has an empty slice, and the page draws a dash.
Classes []AddonRedirectStats `json:"classes,omitempty"`
Kills AddonKills `json:"kills"`
// Fetch is the outbound-request record (M68.5), zero-valued and unobserved for
// an add-on that has never made one.
Fetch AddonFetchStats `json:"fetch"`
}
AddonPerformance is one add-on's redirect-path record.
func (AddonPerformance) IsZero ¶ added in v0.4.0
func (p AddonPerformance) IsZero() bool
IsZero is what `json:",omitzero"` asks of this struct, and it is a method because the encoder's own answer stopped being the right one when AddonFetchStats joined it.
omitzero compares the whole struct, and until M68.5 that comparison and AddonPerformance.Observed coincided — the only fields were the redirect ones. Fetch broke the coincidence: a module holding `network.fetch` and a route prefix, with no redirect class at all, has an all-zero redirect record and a non-zero struct after its first outbound request, and the field-by-field comparison would have published a `performance` object carrying nothing but zeros for the path the module never took. The API document's claim is now what this method says — absent for a module with **no record of either kind** — and there is one predicate rather than two that agreed by accident.
func (AddonPerformance) Observed ¶ added in v0.4.0
func (p AddonPerformance) Observed() bool
Observed reports whether this add-on has any redirect-path record at all.
The **redirect** path and not this struct as a whole, deliberately: it is what the manager's list draws a dash from, and m68.md's promise that a module holding no redirect grant shows no redirect figures rather than zeros is this predicate kept. A module that has only ever fetched has not run there and must still read false here. Whether the object is *published at all* is AddonPerformance.IsZero.
type AddonRedirectStats ¶ added in v0.4.0
type AddonRedirectStats struct {
// Class is `inline` or `observe` — addon.ClassInline and ClassObserve, spelled
// there rather than here.
Class string `json:"class"`
// Count is how many invocations are behind the estimate. Rendered beside it,
// because a p99 over four observations is a number with no meaning and the page
// must not present one as though it had.
Count uint64 `json:"count"`
// P99 is the estimate read off the histogram's buckets.
//
// A Duration in Go and **seconds in JSON**, which is why the two fields below
// exist rather than a tag on this one: a Duration marshals as a nanosecond
// integer, and the series this is read from is `_seconds`. A client comparing
// this answer against a scrape should not have to divide.
P99 time.Duration `json:"-"`
// Sum is the total time observed, so a mean is available without a second
// gather.
Sum time.Duration `json:"-"`
// P99Seconds and SumSeconds are the two above as the wire carries them, filled
// in beside them so that nothing has to convert at the call site and the two
// cannot come to disagree.
P99Seconds float64 `json:"p99_seconds"`
SumSeconds float64 `json:"sum_seconds"`
}
AddonRedirectStats is what one add-on cost the redirect path, per class.
The Add-on manager (M68) renders these on the page itself rather than linking to `/metrics`, which is the checkable form of the owner's "attribution without Prometheus": an operator asking *which add-on is slowing my redirects* gets an answer from the product they are already looking at, on an instance that scrapes nothing.
type IngestStats ¶
type IngestStats interface {
// QueueDepth is the leading indicator for the whole pipeline: it climbs
// minutes before drops start.
QueueDepth() int
Counters() (enqueued, dropped, flushed, failed, batches int64)
}
IngestStats is what the analytics ingester reports about itself.
An interface rather than the concrete type, because observability must not import analytics — analytics is where the click pipeline lives and a cycle through logging would be waiting to happen. The composition root adapts.
type LimiterStats ¶
type LimiterStats interface {
// Len is tracked keys, which is the memory the limiter is using.
Len() int
// Overflows counts requests allowed because the key table was full — the
// number that says the limiter has stopped limiting.
Overflows() int64
// Fallbacks counts decisions this replica made locally because the shared
// limiter did not answer. Always zero for a limiter with no shared backing.
Fallbacks() int64
}
LimiterStats is what a rate limiter reports about its own bookkeeping.
An interface for the same reason as IngestStats: observability must not import the packages it observes.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics is the instrument panel, built once and passed explicitly.
Its own registry rather than prometheus.DefaultRegisterer: a global registry makes two instances in one test process collide, and it lets any dependency that happens to import client_golang publish into our namespace. Passing the struct also means every metric has one obvious definition site.
Every method is nil-safe. Tests and the CLI build routers without metrics, and an instrumentation call site should not have to know whether metrics happen to be enabled.
func NewMetrics ¶
func NewMetrics() *Metrics
NewMetrics builds the registry and registers every collector.
func (*Metrics) AddonPerformance ¶ added in v0.4.0
func (m *Metrics) AddonPerformance() map[string]AddonPerformance
AddonPerformance reads the two per-module series M66 publishes and returns them by add-on name.
Why this reads the registry instead of keeping its own numbers ¶
The alternative is a second set of counters written beside the Prometheus ones, and it was rejected for the reason a second store of anything is: the page and the scrape would be two answers to one question, and the first time they disagreed the disagreement would be the thing an operator had to debug. What is published *is* the record; this asks it.
The cost is a `Gather()` per page render — every collector in the registry, including the Go runtime's — which is a few hundred microseconds and is on an authenticated dashboard page with a 250 ms budget, not on the redirect path. The manager is the only caller.
The p99 is the same estimate `histogram_quantile` makes, and it is an estimate ¶
A Prometheus histogram keeps bucket counts, not observations, so the quantile is interpolated linearly inside whichever bucket the rank falls in — the same arithmetic PromQL does, reproduced here rather than approximated differently, so the number on the page and the number on a dashboard agree. Two consequences are stated rather than left to be discovered: an estimate inside the last finite bucket cannot exceed that bucket's boundary, and one whose rank lands in `+Inf` saturates at that boundary, because there is no upper bound to interpolate towards.
**A saturated estimate is not marked as one.** It is returned as an ordinary Duration and `shortDuration` prints it as an ordinary figure, so a p99 in the `+Inf` bucket reads as exactly the last boundary — 1s, `redirectBuckets` — and understates whatever it actually was. What keeps that far away in a shipped configuration is the deadlines rather than this function: a successful invocation of either class spends at most `LINKCTRL_ADDON_INSTANTIATE_DEADLINE` plus `LINKCTRL_ADDON_INLINE_DEADLINE` under a bound, 525 ms together by default, and a killed one is not observed here at all. An operator who raises either past a second reaches the last bucket outright, and F331 carries what the page should say when they do.
**Bounded is not the same as all of it**, and the difference is stated rather than rounded away: the observed window closes after the instance is released, and releasing one copies the guest's linear memory back over itself — up to `maxGuestMemoryPages`, on this host's own CPU, under no deadline at all. It is microseconds in practice and it is a memcpy rather than guest execution, so nothing about an add-on's own code can stretch it; but a machine under enough pressure to make it matter is a machine where the two deadlines above are not the whole story either. The claim this comment makes is therefore about the deadlines and not about the histogram's window, which is wider than they are.
It is also **cumulative since this process started**, not a rate over a window: there is no time series here to take a rate of. An add-on that was slow an hour ago and is fine now still reads slow, and the manager says as much beside the figure rather than implying a live reading.
func (*Metrics) ForgetAddon ¶ added in v0.4.0
func (m *Metrics) ForgetAddon(addon, version string, abiVersion int, failureClass, permissions string)
ForgetAddon drops the gauges that describe an add-on this instance no longer runs (M67).
**Gauges only, and the counters deliberately stay.** `linkctrl_addon_info` and the two size gauges are statements about the present tense — *this instance runs this module, and its schema is this big* — and leaving them behind after a removal makes each one a lie that reads exactly like a fact. The counters are statements about the past: `linkctrl_addon_loads_total` and `linkctrl_addon_refusals_total` describe attempts that really happened, and deleting them would erase the record of an add-on that was installed and removed, which is the history an operator reading a scrape after an incident most needs.
The info series is deleted by exact label values because that is the only way to address one: its identity is every label it carries. The caller therefore passes what it published, which is why this takes five arguments rather than a name.
**The two size gauges go too, and that is the answer to a real question.** Removal leaves the add-on's schema behind — an orphan M63 makes enumerable and M68 offers to purge — so there is an argument for keeping the number. It loses, because nothing sets it any more: the maintenance job measures the *loaded* add-ons, so a gauge left standing is a value frozen at the last measurement before the removal, and a frozen gauge reads exactly like a live one. Silence is the honest answer, and what an operator reads instead is the boot warning naming every schema no loaded module owns.
func (*Metrics) Gather ¶
func (m *Metrics) Gather() *prometheus.Registry
Gather exposes the registry for tests.
func (*Metrics) HTTPMiddleware ¶
HTTPMiddleware counts and times every request.
Placed outermost, so the numbers include session lookup, CSRF checks and everything else a handler does not control. The redirect surface is also measured in finer detail inside its handler; this one is the outside view.
func (*Metrics) Handler ¶
Handler serves the scrape endpoint.
This is mounted on the metrics listener, never on the public one: the series below expose queue depths, pool saturation and the shape of traffic, which is operational detail rather than something to publish — and, since M60, the name and version of every installed add-on, which is an inventory of what this instance runs. docs/SECURITY.md says the same thing to the operator.
func (*Metrics) ObserveAddonFetch ¶ added in v0.4.0
ObserveAddonFetch records one outbound request an add-on made (M68.5): the outcome always, and the duration only when this host attempted the request rather than refusing it.
One method rather than two because the two series are written together on every path and a caller that could write one without the other would be a caller that eventually does. A zero duration means *this never dialled*, which is why it is the absence of an observation rather than an observation of zero.
func (*Metrics) ObserveAddonLoad ¶ added in v0.4.0
ObserveAddonLoad records one add-on load attempt (M60).
Called for every attempt including the refusals, which is the point: an operator whose add-on is silently not there needs a series that says so, and a counter that only ever counted successes would leave the failure visible in a log line nobody is scraping.
func (*Metrics) ObserveAddonRedirect ¶ added in v0.4.0
ObserveAddonRedirect records what one add-on cost one redirect (M66).
Called from the redirect path itself for the inline class and from the out-of-band worker for the observe one, so it is on the hot path and is a label lookup and an observation, exactly like ObserveRedirect beside it.
func (*Metrics) ObserveAddonRedirectKill ¶ added in v0.4.0
ObserveAddonRedirectKill counts an invocation the host stopped waiting for, at the step whose bound it overran.
step is addon.StepInstantiate or addon.StepCall, spelled there rather than here because that package is the one that knows which bound applies to which step.
func (*Metrics) ObserveAddonRefusal ¶ added in v0.4.0
ObserveAddonRefusal records one ABI call refused for want of a declared permission (M62).
Called from the host's own dispatch, on whatever goroutine the guest is running on, which from M66 is a request's. Both labels are bounded: the add-on's name comes from a validated manifest and the permission from a closed vocabulary, so neither is guest input however the module was written.
func (*Metrics) ObserveAutomationFiring ¶ added in v0.2.0
ObserveAutomationFiring records one rule firing (M43).
Called once per firing, not once per subject and not once per evaluation: the question this answers is "how much is the scheduler doing on somebody's behalf", and a rule that matched forty links did one thing.
func (*Metrics) ObserveFeedCheck ¶ added in v0.2.0
ObserveFeedCheck records one third-party reputation check.
The count is what makes a failing feed observable at all. A check that errors fails open to the built-in tiers by design, so the destination is accepted and nothing in the product's behaviour says the feed stopped answering — an operator who enabled a feed and is relying on it would otherwise find out by noticing nothing was ever refused.
func (*Metrics) ObserveJob ¶
ObserveJob records a background job run.
func (*Metrics) ObserveJobSkipped ¶
ObserveJobSkipped records a run that another replica held the lock for.
Counted rather than ignored: on a healthy multi-replica deployment most runs are skips, and a follower that never skips is a follower that never tried.
func (*Metrics) ObserveRedirect ¶
ObserveRedirect records one short-link request.
Called from the redirect handler with the elapsed time **less whatever an inline add-on held the redirect for** (M66), so instrumentation adds a map lookup and a histogram observation — tens of nanoseconds against a 20ms budget. That subtraction is why this no longer takes the same number the click event carries: the click records what the visitor waited, and this records what this product's own work cost.
One measurement caveat, verified rather than assumed: on a Windows host Go's monotonic clock cannot resolve an interval this short, and time.Since returns exactly zero for 100,000 out of 100,000 back-to-back samples. A cache-served redirect therefore lands in the zero bucket, making _sum and any average useless locally. Bucket counts, and so the "fraction under 20ms" ratio the SLO is stated as, remain correct — and the SLO itself is measured on Linux in containers, where the clock has nanosecond resolution.
func (*Metrics) ObserveThrottled ¶
ObserveThrottled records one request refused by a rate limit.
The label names the limit — "login", "api", "redirect_404" — not the client. That is what makes the series bounded, and it is also the more useful cut: an operator wants to know that logins are being throttled, and finds out who from the log if it matters.
func (*Metrics) ObserveWebhookDelivery ¶ added in v0.2.0
ObserveWebhookDelivery records one delivery attempt (M42).
Both labels come from a closed vocabulary the caller computes: internal/webhook reduces an HTTP code to its class before calling, so nothing user-chosen can reach a label from here. See the metric's definition for why that matters.
func (*Metrics) Register ¶
func (m *Metrics) Register(c prometheus.Collector)
Register adds a collector that reads live state — pool statistics, queue depth — rather than being written to by instrumentation.
func (*Metrics) SetAddonInfo ¶ added in v0.4.0
func (m *Metrics) SetAddonInfo(addon, version string, abiVersion int, failureClass, permissions string)
SetAddonInfo publishes the identity of an add-on that loaded (M60), and the permissions it holds (M62).
`permissions` is the *held* set, sorted and comma-separated — not what the manifest declared, since a permission the vocabulary carries and no host grants yet is declarable and not held. Sorted so a series does not change identity because a manifest listed the same grants in another order.
func (*Metrics) SetAddonLargeObjects ¶ added in v0.4.0
SetAddonLargeObjects records how many large objects one add-on's role owns (M63).
Zero for every add-on that behaves, which is why it is published at all: a large object is outside every schema, so SetAddonSchemaBytes cannot see one and an add-on's growth would otherwise be invisible between loads. Same replica rule as SetAddonSchemaBytes.
func (*Metrics) SetAddonSchemaBytes ¶ added in v0.4.0
SetAddonSchemaBytes records the on-disk size of one add-on's own schema (M63) — every relation in it that has storage, not a list of the kinds somebody thought of, which is store.AddonSchemaBytes's own argument and D254's.
Set by every replica, like SetAuditLogBytes and for the reason given there: a gauge the followers never set reads as zero. The add-on's name comes from a validated manifest, so the label is bounded whatever the module was written to do.
func (*Metrics) SetAuditLogBytes ¶ added in v0.2.0
SetAuditLogBytes records the audit log's on-disk size.
A plain gauge rather than a collector that queries at scrape time, because /metrics has to keep answering while the database is unwell — it is the endpoint an operator scrapes to find out that it is. The cost is that the value is up to an hour stale, which does not matter for a series whose whole purpose is a growth trend measured in days.
Set by every replica, not only the job leader. A gauge only the leader wrote would read as zero on every follower, so whether an alert fired would depend on which replica answered the scrape.
func (*Metrics) SetJobStaleness ¶ added in v0.2.0
SetJobStaleness records how long ago a job last succeeded.
Set by every replica, like SetAuditLogBytes and for the same reason: this is an observation of shared state rather than work that must happen once, and a gauge only the leader wrote would make an alert depend on which replica the scrape reached.
type Surface ¶
type Surface string
Surface is the coarse bucket a request belongs to.
Deliberately coarse. A label per URL path would let anyone mint unbounded series by requesting random aliases — the classic way a metrics endpoint becomes the reason a server falls over — and the redirect tree's whole namespace is attacker-chosen. Per-route detail for the API lives in the access log, which is sampled and does not accumulate.
func ClassifySurface ¶
ClassifySurface maps a request path to its surface.
Source Files