Documentation
¶
Overview ¶
Package metrics is the mlfs daemon's in-process instrument registry. It is the single source of truth for runtime counters (FUSE ops, bytes moved, cache hit/miss, write-back upload + staging depth) and is read through several surfaces without double-counting: a Prometheus-text render that the FUSE bridge serves as a virtual file inside the mount (so a workload pod can poll its own live feed with no extra access), an HTTP /metrics scrape, and an OTLP push — the last two for centralized reporting. WithIdentity tags every stream with the producer's domain/instance/region so tenants don't collide centrally.
Instruments are OpenTelemetry metric instruments fed into a Prometheus exporter, so the same definitions back every surface. All record methods are nil-safe: a nil *Registry is a no-op, letting callers instrument hot paths unconditionally without a per-call branch at the call site beyond the cheap nil check.
tracing.go stands up the daemon's OpenTelemetry TracerProvider — the trace half of the observability story (metrics.go is the metric half). The storage architecture is a fan-out (a cold mlfs read decomposes into a cache miss → a manifest read on PostgreSQL → a pack GET on S3), so tail latency lives in that fan-out and a request span that crosses the layer boundary is where it shows up (docs/OBSERVABILITY.md, "Traces"). The casstore backing wrapper (casstore/blobstore/obs) already opens leaf client spans on the GLOBAL tracer provider; the FUSE bridge opens the parent op spans on it too. This file is what makes those spans actually export: it builds an SDK TracerProvider over an OTLP/gRPC trace exporter and installs it as the global provider.
Tracing is built ONLY when an OTLP endpoint is configured — unlike metrics (which always have the local Prometheus surfaces), a trace is useless without a collector to receive it. When no endpoint is set NewTracing returns a nil *Tracing whose every method is a no-op and the global provider stays the OTEL default no-op, so casstore/bridge spans cost nothing and behavior is unchanged.
Index ¶
- func ErrorKind(err error) string
- type Option
- type Registry
- func (r *Registry) AddReadBytes(n int)
- func (r *Registry) AddWriteBytes(n int)
- func (r *Registry) AddWriteClassBytes(n int, uncompressed bool)
- func (r *Registry) CacheHit()
- func (r *Registry) CacheMiss()
- func (r *Registry) FenceRejected()
- func (r *Registry) FuseOp(op string)
- func (r *Registry) Handler() http.Handler
- func (r *Registry) IntegrityError(kind string)
- func (r *Registry) LeaderTransition(leader bool)
- func (r *Registry) LeaseAcquired()
- func (r *Registry) LeaseLost()
- func (r *Registry) Meter() metric.Meter
- func (r *Registry) MountEvent(event string)
- func (r *Registry) Prefetch(n int64)
- func (r *Registry) PrefetchPacks(n int64)
- func (r *Registry) RecordFuseOp(op string, start time.Time, ok bool)
- func (r *Registry) RecordMetaOp(op string, start time.Time, errp *error)
- func (r *Registry) RenderPrometheus() ([]byte, error)
- func (r *Registry) SetMetaPoolObserver(fn func() (inUse, idle, waitCount int64))
- func (r *Registry) SetOldestUnflushedObserver(fn func() float64)
- func (r *Registry) SetStagingObserver(fn func() (files, bytes int64))
- func (r *Registry) Shutdown(ctx context.Context) error
- func (r *Registry) Upload(n int64)
- type Tracing
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ErrorKind ¶
ErrorKind maps a metadata-DB error to the small bounded error.kind set in docs/OBSERVABILITY.md (never a raw error string — that is unbounded). It mirrors the casstore/blobstore/obs taxonomy so a meta error reads the same as a backing error on a dashboard. sql.ErrNoRows is not_found; context cancel/deadline are canceled/timeout; the rest fall back to string sniffs for the common Postgres conditions (serialization conflict, lock timeout, auth) and otherwise "other".
Types ¶
type Option ¶
type Option func(*config)
Option configures optional export surfaces on top of the always-on Prometheus reader (which backs the in-mount virtual file and the HTTP scrape).
func WithIdentity ¶
WithIdentity stamps the producer's identity onto every exported metric stream: the dedup/tenant domain, the node instance id, and the region. These become OTLP resource attributes (so a central collector can tell tenants apart) and constant labels on the Prometheus surfaces (so a federated scrape can too). Without it the streams are anonymous and collide across daemons.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds the daemon's instruments and the Prometheus registry they export through.
func New ¶
New builds a registry. A Prometheus reader is always present (it backs both the in-mount virtual file via RenderPrometheus and the HTTP scrape via Handler); WithOTLP adds a push reader for centralized reporting. The returned Registry is ready to record.
func (*Registry) AddReadBytes ¶
AddReadBytes records bytes returned to a read(2).
func (*Registry) AddWriteBytes ¶
AddWriteBytes records bytes accepted from a write(2).
func (*Registry) AddWriteClassBytes ¶
AddWriteClassBytes records write(2) bytes attributed by compression class so the model-vs-generic split is observable. uncompressed selects the "uncompressed" (model/tensor) series; otherwise "default". Same bytes as AddWriteBytes, cut by class.
func (*Registry) CacheHit ¶
func (r *Registry) CacheHit()
CacheHit / CacheMiss record a chunk read served locally vs from the backing store.
func (*Registry) FenceRejected ¶
func (r *Registry) FenceRejected()
FenceRejected records one metadata write rejected because the ownership generation moved out from under it (a stale-handle fence; multi-node).
func (*Registry) Handler ¶
Handler returns an HTTP handler that serves the current metrics in Prometheus exposition format, for a /metrics scrape endpoint. Backed by the same registry as the in-mount virtual file. Returns nil for a nil Registry.
func (*Registry) IntegrityError ¶
IntegrityError bumps the data-integrity canary for kind ("corrupt" / "not_found" / "range"). Caller classifies a read-path error with snapshot.IntegrityKind and calls this only when the kind is non-empty. A nil Registry is a no-op.
func (*Registry) LeaderTransition ¶
LeaderTransition records a GC-leader election transition. leader=true means this node just became leader; false means it stood down (multi-node).
func (*Registry) LeaseAcquired ¶
func (r *Registry) LeaseAcquired()
LeaseAcquired / LeaseLost record an ownership-lease transition (multi-node).
func (*Registry) Meter ¶
Meter returns the daemon's OTEL meter so opt-in adapters (e.g. casstore/blobstore/obs wrapping the S3 backing store) register their own instruments on the SAME provider — sharing this daemon's identity stream and the three export surfaces. Returns an otel noop meter for a nil Registry, so a caller (e.g. obs.Wrap) never has to branch on metrics being disabled.
func (*Registry) MountEvent ¶
MountEvent records a mount-lifecycle event ("mount" / "unmount"); repeated cycling shows up as mount flap.
func (*Registry) Prefetch ¶
Prefetch records one slice (n bytes) warmed into the cache by sequential-read readahead — the speculative work that converts later demand-read misses into local hits. Distinct from Upload (write-back) and from CacheMiss (the backing fetch a prefetch itself incurs).
func (*Registry) PrefetchPacks ¶
PrefetchPacks records the number of coalesced backing data GETs (packs) a prefetch batch issued. With mlfs.prefetch.slices it shows the round-trip reduction: slices/packs is the coalescing factor.
func (*Registry) RecordFuseOp ¶
RecordFuseOp records one served FUSE op's latency under {operation, outcome} and bumps the fuse.ops counter, so the op rate and its latency/error split come from one call. start is when the op began; ok=false records outcome=error. A nil Registry is a no-op.
func (*Registry) RecordMetaOp ¶
RecordMetaOp records one metadata-DB op's latency under {operation, outcome} and, on failure, bumps mlfs.meta.errors under {operation, error.kind}. start is when the op began; err nil means success. A nil Registry is a no-op. Callers wrap a query with a deferred call:
defer r.RecordMetaOp("read_slices", time.Now(), &err)
func (*Registry) RenderPrometheus ¶
RenderPrometheus returns the current snapshot in Prometheus text exposition format — the bytes the FUSE virtual file and any /metrics scrape serve.
func (*Registry) SetMetaPoolObserver ¶
SetMetaPoolObserver registers the callback the registry invokes on each collect to read the metadata DB connection-pool levels (in-use / idle / cumulative wait count). The daemon sets it to sql.DB.Stats(). Nil clears it.
func (*Registry) SetOldestUnflushedObserver ¶
SetOldestUnflushedObserver registers the callback the registry invokes on each collect to read the age (seconds) of the oldest still-unflushed staged write (0 = staging empty) — the GC safety-window canary. The cache sets it to a stat of its staging dir, so the cost lands only when metrics are scraped. Nil clears it.
func (*Registry) SetStagingObserver ¶
SetStagingObserver registers a callback the registry invokes on each collect to read the current write-back staging depth (files, bytes). The cache sets this to a directory walk of its staging dir, so the cost lands only when metrics are actually scraped, never on the drain path. Nil clears it.
type Tracing ¶
type Tracing struct {
// contains filtered or unexported fields
}
Tracing owns the daemon's SDK TracerProvider and its shutdown. A nil *Tracing is the disabled case (no OTLP endpoint): every method is a no-op, mirroring the nil-safety of *Registry, so callers never branch on tracing being off.
func NewTracing ¶
NewTracing builds the daemon's TracerProvider and installs it as the global OTEL provider (so casstore/blobstore/obs and the FUSE bridge, which both use otel.Tracer(...), export through it), along with a W3C trace-context propagator. It honors the SAME Options as metrics.New — WithOTLP supplies the collector endpoint (REQUIRED: with no endpoint NewTracing returns (nil, nil) and leaves the global no-op provider in place) and WithIdentity supplies the resource identity, so a trace carries the same service.name/instance/domain a metric stream does and a collector correlates them.
sampleRatio is the head-sampling probability for a ROOT span (a span with no remote parent); a span whose parent is already sampled is always recorded (ParentBased), so a request never decomposes into a half-sampled trace. Pass ratio ≥ 1 to record everything, ≤ 0 for the SDK default (AlwaysSample). The exemplar bridge on the metric side links a sampled trace to its latency bucket, so the sampler also governs how many exemplars land.
Source Files