metrics

package
v0.44.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 25, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package metrics is filex's Prometheus surface.

There was none before this: `grep -r prometheus` over the whole repo found only the plan that asked for it. So this package defines the exposition rather than matching an existing one, and everything it publishes is named `filex_*`.

Where it is served

`GET /metrics`, behind the SAME admin gate as every other operator endpoint (`auth.Middleware(true)` + `auth.RequireAdmin`). filex is a file server that is routinely exposed to the internet, and an open /metrics leaks storage names, user counts and traffic shape. A Prometheus job therefore scrapes it with an admin session/token; `docs/METRICS.md` shows the scrape config.

What is instrumented, and why exactly this

The list is the shortest one that answers the questions an operator actually asks when uploads on slow storage go wrong:

  • "is anything stuck?" → staged uploads in flight, and their age
  • "did it get worse?" → commits vs. failures vs. retries
  • "is the backend slow?" → transfer duration + rolling bytes/sec
  • "why was I refused?" → guard counters, one per guard
  • "is the disk filling?" → staged bytes, sweeper removals
  • "who is using the space?" → total quota usage, refusals

Nothing here is on a hot loop: counters are incremented once per upload, per commit or per refusal, and the two derived gauges are computed at scrape time from in-memory state.

Index

Constants

View Source
const (
	GuardDisk  = "disk"
	GuardQuota = "quota"
)

Guard label values.

Variables

View Source
var (
	// StagedInFlight is the number of staging directories currently open
	// (begun, not yet committed, aborted or swept). Gauge, not counter: the
	// question is "how many right now".
	StagedInFlight = prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "filex_staged_uploads_in_flight",
		Help: "Staged uploads that have begun and not yet committed, aborted or been swept.",
	})

	// StagedBytes is the number of bytes currently sitting in the staging
	// area. This is the disk-incident early warning.
	StagedBytes = prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "filex_staging_bytes",
		Help: "Bytes currently held in the upload staging area.",
	})

	// StagedBegun counts uploads accepted at `begin`.
	StagedBegun = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staged_uploads_begun_total",
		Help: "Staged uploads accepted at begin.",
	})

	// StagedBytesStaged counts bytes written into staging (chunk PUTs).
	StagedBytesStaged = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staged_upload_bytes_staged_total",
		Help: "Bytes accepted into the staging area.",
	})

	// StagedCommitted counts staged uploads whose bytes reached the driver.
	StagedCommitted = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staged_uploads_committed_total",
		Help: "Staged uploads successfully transferred to their storage driver.",
	})

	// StagedFailed counts transfers that failed (the staging bytes are kept
	// for retry, so this rising while committed does not is the signal that a
	// backend is down).
	StagedFailed = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staged_uploads_failed_total",
		Help: "Staged upload transfers that failed.",
	})

	// StagedChunkRetries counts chunk PUTs that overwrote a part already
	// present — i.e. a client resuming or re-sending. A rising line here with
	// no failures means flaky links, not a broken server.
	StagedChunkRetries = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staged_upload_chunk_retries_total",
		Help: "Chunk uploads that re-sent a part filex already held.",
	})

	// StagedAborted counts client-requested aborts.
	StagedAborted = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staged_uploads_aborted_total",
		Help: "Staged uploads aborted by the client.",
	})
)
View Source
var (
	// SweepRuns counts sweeper passes — a flat line means the GC stopped.
	SweepRuns = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "filex_staging_sweeps_total",
		Help: "Staging sweeper passes.",
	})

	// Swept counts what the sweeper removed. kind="row" is an expired upload
	// the DB knew about; kind="orphan" is a directory with no row at all
	// (debris from a crash between mkdir and INSERT).
	Swept = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "filex_staging_swept_total",
		Help: "Staging entries removed by the sweeper, by kind.",
	}, []string{"kind"})
)
View Source
var (
	// TransferDuration is how long one transfer op took, by storage and
	// direction. Buckets run from a tenth of a second to twenty minutes,
	// because the whole point of this work is the slow tail.
	TransferDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "filex_transfer_duration_seconds",
		Help:    "Duration of one storage transfer operation.",
		Buckets: []float64{0.1, 0.5, 1, 2, 5, 10, 30, 60, 180, 600, 1200},
	}, []string{"storage", "direction"})

	// TransferBytes counts bytes moved to/from a storage driver.
	TransferBytes = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "filex_transfer_bytes_total",
		Help: "Bytes transferred to or from a storage driver.",
	}, []string{"storage", "direction"})
)
View Source
var (
	// QuotaUsageBytes is the sum of every user's usage_bytes. Deliberately
	// UNLABELLED: a per-user gauge is a cardinality bomb on an instance with
	// a few thousand accounts, and the per-user number is already available
	// through the admin API. Seeded at boot from the users table and moved by
	// the accounting decorator afterwards.
	QuotaUsageBytes = prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "filex_quota_usage_bytes",
		Help: "Total storage attributed to users (sum of users.usage_bytes).",
	})

	// QuotaAccountedBytes counts absolute movement, so a stuck accounting
	// path is visible even when adds and subs cancel out.
	QuotaAccountedBytes = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "filex_quota_accounted_bytes_total",
		Help: "Bytes accounted to users, by direction (added/released).",
	}, []string{"direction"})
)
View Source
var (
	// PluginOps counts operations by plugin, operation and outcome
	// (ok | error | busy). `busy` is its own outcome on purpose: it means the
	// plugin hit its concurrency ceiling, which is a sizing problem, not a
	// fault to chase.
	PluginOps = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "filex_plugin_ops_total",
		Help: "Storage-plugin operations, by plugin, operation and outcome.",
	}, []string{"plugin", "op", "outcome"})

	// PluginOpDuration is how long the plugin took to answer.
	PluginOpDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "filex_plugin_op_duration_seconds",
		Help:    "How long a storage plugin takes to answer, by plugin and operation.",
		Buckets: []float64{0.005, 0.025, 0.1, 0.5, 1, 5, 30},
	}, []string{"plugin", "op"})

	// PluginInFlight is the live count of operations inside a plugin.
	PluginInFlight = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Name: "filex_plugin_in_flight",
		Help: "Operations currently inside a storage plugin.",
	}, []string{"plugin"})

	// PluginRestarts counts supervisor restarts. A plugin that restarts in a
	// loop still "works" for single requests, and this is how that shows up.
	PluginRestarts = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "filex_plugin_restarts_total",
		Help: "Times filex has restarted a storage plugin after it exited.",
	}, []string{"plugin"})

	// PluginUp is 1 while a plugin is running and registered.
	PluginUp = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Name: "filex_plugin_up",
		Help: "1 when a storage plugin is running and its driver is registered.",
	}, []string{"plugin"})
)

Storage plugins. A plugin is somebody else's program in filex's request path, so "is it slow, is it failing, is it saturated" has to be answerable from outside — the alternative is an operator watching a spinner.

View Source
var (
	// LazyFolders is how many folders of a storage are in each catalogue
	// state: uncatalogued (the filler's work list), catalogued, watched.
	LazyFolders = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Namespace: "filex", Name: "lazy_folders",
		Help: "Folders of a lazily catalogued storage per catalogue state (uncatalogued, catalogued, watched).",
	}, []string{"storage", "state"})
	// LazyWatches is how many fsnotify watches the storage holds right now.
	LazyWatches = prometheus.NewGaugeVec(prometheus.GaugeOpts{
		Namespace: "filex", Name: "lazy_watches",
		Help: "Visited folders currently watched for changes on a lazily catalogued storage.",
	}, []string{"storage"})
	// LazyReconciles counts folder reconciles by what asked for them: open,
	// watch, fill, refresh, pair.
	LazyReconciles = prometheus.NewCounterVec(prometheus.CounterOpts{
		Namespace: "filex", Name: "lazy_reconciles_total",
		Help: "Folder reconciles of the lazy catalogue, by reason (open, watch, fill, refresh, pair).",
	}, []string{"storage", "reason"})
	// LazyHeldBack counts deletions the per-folder guard refused to make.
	LazyHeldBack = prometheus.NewCounterVec(prometheus.CounterOpts{
		Namespace: "filex", Name: "lazy_held_back_total",
		Help: "Catalogue deletions the lazy catalogue's per-folder guard held back.",
	}, []string{"storage"})
)

── lazy catalogue (sync_mode lazy, issue #45) ─────────────────────────────

"How far has the catalogue come, and is the watch budget full?" — the two questions an operator of a large lazily catalogued storage asks.

View Source
var CacheBytes = prometheus.NewGauge(prometheus.GaugeOpts{
	Name: "filex_cache_bytes",
	Help: "Bytes currently held by the download cache.",
})

CacheBytes is how much the download cache currently holds. filecache calls CacheBytes.Set; zero until it exists.

View Source
var CacheEvents = prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "filex_cache_events_total",
	Help: "Download cache outcomes, by result.",
}, []string{"result"})

CacheEvents counts cache outcomes. Declared here — with the counters already registered — so filecache reports through the same exposition instead of growing its own registry. result="hit"|"miss"|"evict"|"store".

View Source
var GuardRefusals = prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "filex_guard_refusals_total",
	Help: "Requests refused by a guard, by guard name.",
}, []string{"guard"})

GuardRefusals counts every refusal by a guard, labelled with which one. guard="disk" is the staging free-space guard; guard="quota" is the per-user ceiling. One metric with a label rather than two metrics, so an alert can say "any guard is firing" without being edited every time one is added.

Registry is filex's own registry rather than the default one: a library that registers into prometheus.DefaultRegisterer (a dependency we do not control) cannot then silently add series to our exposition.

Functions

func Handler

func Handler() http.Handler

Handler serves the exposition. Mount it behind the admin gate.

func StorageLabel

func StorageLabel(id int64) string

StorageLabel renders a storage id for a metric label.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL