storagecap

package
v1.147.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 17, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package storagecap is the storage capacity an administrator reads and is alerted on (#2148): how much of each store is used and how much is free, the database against its limit, the object store cluster's disks, a bucket against its budget and each replica's webhook spool volume; the history of each, how fast it is growing, when it would fill at that rate, and leveled alerts on both. #1899 measured usage as metrics; this makes it a page and a notification, with capacity read from the stores where they say it.

Index

Constants

View Source
const (
	KindDatabase  = "database"
	KindSeaweedFS = "seaweedfs"
	KindBucket    = "bucket"
	KindSpool     = "spool"
)

The kinds of store a report and its alerts cover.

View Source
const AlertFamily = "storage"

AlertFamily is the levelalert family the storage alerts are recorded under.

View Source
const (
	DefaultHistoryDays = 180
)

The defaults a deployment starts with before an administrator writes the section.

View Source
const DefaultInterval = 15 * time.Minute

DefaultInterval is how often a store is sampled and its alerts evaluated.

View Source
const LockKey int64 = 4713210402

LockKey is the advisory lock one replica samples and alerts under. The capacity scan holds 4713210401.

View Source
const SettingsSection = "storage"

SettingsSection is the platform_settings section holding the settings. It is a stored key.

Variables

View Source
var (
	DefaultUsedPercent = levelalert.Thresholds{Warning: 75, Critical: 90}
	DefaultDaysLeft    = levelalert.Thresholds{Warning: 30, Critical: 7, Below: true}
)

DefaultUsedPercent and DefaultDaysLeft are the alert thresholds before an administrator sets any: percent of a store used, and days until it fills at its growth rate.

Functions

func BudgetsFrom

func BudgetsFrom(store SettingsStore) func(ctx context.Context) (map[string]int64, error)

BudgetsFrom reads the bucket budgets the settings hold, for the capacity gauges (#1899) to report them over the environment's.

func Findings

func Findings(r Report, settings Settings, pageURL string) []levelalert.Finding

Findings is every store at warning or critical under settings, by percent used and by days until it fills, and marks each store's level on r.

func Unjudged

func Unjudged(r Report) []string

Unjudged is the alert key prefixes of the stores this report could not read: the SeaweedFS cluster and every bucket whose capacity it gives when the cluster read failed, and each spool volume whose replica missed its recent reports without being gone. Their conditions are carried as they stand rather than resolved.

Types

type BucketView

type BucketView struct {
	Bucket   string     `json:"bucket"`
	Purpose  string     `json:"purpose"`
	Bytes    int64      `json:"bytes"`
	Objects  int64      `json:"objects"`
	ListedAt *time.Time `json:"listed_at"`
}

BucketView is what the platform holds in one bucket for one purpose.

type ClusterCapacity

type ClusterCapacity struct {
	Total int64 `json:"total_bytes"`
	Used  int64 `json:"used_bytes"`
	Free  int64 `json:"free_bytes"`
	// Servers is how many volume servers answered; Unreached lists the ones
	// that did not, so a partial reading is labeled as one.
	Servers   int      `json:"servers"`
	Unreached []string `json:"unreached"`
}

ClusterCapacity is the disk the cluster's volume servers report: every disk directory of every server read, summed.

func ReadSeaweedFS

func ReadSeaweedFS(ctx context.Context, client *http.Client, cfg SeaweedFS) (ClusterCapacity, error)

ReadSeaweedFS reads the cluster's disk capacity: the volume servers cfg names, or the ones the master lists, each asked for its disks.

type ClusterView

type ClusterView struct {
	MasterURL string          `json:"master_url"`
	Capacity  ClusterCapacity `json:"capacity"`
	Error     string          `json:"error,omitempty"`
}

ClusterView is the SeaweedFS cluster as last read, or why it was not.

type DatabaseView

type DatabaseView struct {
	Bytes      int64       `json:"bytes"`
	LimitBytes *int64      `json:"limit_bytes"`
	Tables     []TableView `json:"tables"`
}

DatabaseView is the database's size, its limit and its largest relations.

type Deps

type Deps struct {
	DB       *sql.DB
	Settings SettingsStore
	// Spools reads every receiver's spool state; nil without webhooks.
	Spools func(ctx context.Context) ([]whstore.SpoolState, error)
	// HTTP reads the SeaweedFS cluster.
	HTTP *http.Client
	// EnvBudgets are the budgets MCP_PLATFORM_STORAGE_BUDGETS names, which
	// a budget in the settings overrides bucket by bucket.
	EnvBudgets map[string]int64
	// OnSeaweedFS reports whether a bucket is on the SeaweedFS cluster: such
	// a bucket with no budget can grow into the cluster's free space, which
	// is then its free space. Nil names none.
	OnSeaweedFS func(ctx context.Context, bucket string) bool
	// Engine routes the alerts; nil records nothing and sends nothing.
	Engine *levelalert.Engine
	// PageURL is the admin storage page an alert links to; empty omits it.
	PageURL  string
	Interval time.Duration
	Now      func() time.Time
}

Deps is what the service reads and sends through.

type Forecast

type Forecast struct {
	// Rate7d and Rate30d are the growth in bytes per day fitted over the last
	// 7 and 30 days, nil where the history is too short.
	Rate7d  *float64 `json:"rate_7d_bytes_per_day"`
	Rate30d *float64 `json:"rate_30d_bytes_per_day"`
	// RateUsed names the rate the projection uses, "7d" or "30d": the slower
	// of the two that are growing, so a burst does not bring the date in.
	RateUsed string `json:"rate_used,omitempty"`
	// DaysLeft and ExhaustionAt are when the store fills at RateUsed, nil
	// when it is not growing or has no known capacity.
	DaysLeft     *float64   `json:"days_left"`
	ExhaustionAt *time.Time `json:"exhaustion_at"`
}

Forecast is how fast a store is growing and when it would fill.

func Project

func Project(points []Point, now time.Time) Forecast

Project fits the store's growth over its history and, with its capacity, when it fills. points are in time order; the last is the current reading.

type Input

type Input struct {
	Channels           *[]string              `json:"channels,omitempty"`
	DatabaseLimitBytes *int64                 `json:"database_limit_bytes,omitempty"`
	Budgets            *map[string]int64      `json:"budgets,omitempty"`
	SeaweedFS          *SeaweedFS             `json:"seaweedfs,omitempty"`
	HistoryDays        *int                   `json:"history_days,omitempty"`
	UsedPercent        *levelalert.Thresholds `json:"used_percent,omitempty"`
	DaysLeft           *levelalert.Thresholds `json:"days_left,omitempty"`
}

Input is the write shape. An absent field keeps the stored value.

func (Input) Apply

func (in Input) Apply(stored Settings) (Settings, error)

Apply returns stored with in written over it, or the first reason in is invalid. Whether each channel exists is checked by the route.

type Point

type Point struct {
	At    time.Time `json:"at"`
	Used  int64     `json:"used_bytes"`
	Total *int64    `json:"total_bytes"`
}

Point is one sample of a store.

type ReconcileView

type ReconcileView struct {
	Purpose      string    `json:"purpose"`
	Orphaned     int64     `json:"orphaned"`
	Dangling     int64     `json:"dangling"`
	ReconciledAt time.Time `json:"reconciled_at"`
}

ReconcileView is what the last reconcile found for one purpose.

type Report

type Report struct {
	GeneratedAt time.Time       `json:"generated_at"`
	HistoryDays int             `json:"history_days"`
	Stores      []StoreView     `json:"stores"`
	Database    DatabaseView    `json:"database"`
	Buckets     []BucketView    `json:"buckets"`
	Reconciles  []ReconcileView `json:"reconciles"`
	LastScan    *ScanView       `json:"last_scan"`
	SeaweedFS   *ClusterView    `json:"seaweedfs"`
	Spools      []SpoolView     `json:"spools"`
}

Report is what GET /api/v1/admin/storage answers: every store the platform writes to, used and free, with its growth and projection, and the detail behind each.

type ScanView

type ScanView struct {
	DurationSeconds float64   `json:"duration_seconds"`
	Objects         int64     `json:"objects"`
	FinishedAt      time.Time `json:"finished_at"`
}

ScanView is the last full listing of the buckets.

type SeaweedFS

type SeaweedFS struct {
	// MasterURL is the master's HTTP address, e.g. http://seaweedfs-master:9333.
	MasterURL string `json:"master_url"`
	// VolumeURLs, when set, are the volume servers read instead of the ones
	// the master lists: a master inside a cluster lists addresses the
	// platform may not reach.
	VolumeURLs []string `json:"volume_urls"`
}

SeaweedFS names the cluster the platform reads disk capacity from. The S3 gateway does not report it; the master lists the volume servers and each reports its disks.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service reads the stores, keeps their history and evaluates the alerts.

func New

func New(d Deps) *Service

New returns the service, or nil without a database.

func (*Service) History

func (s *Service) History(ctx context.Context, store string, days int) ([]Point, error)

History is a store's samples over the last days, oldest first.

func (*Service) Pass

func (s *Service) Pass(ctx context.Context) error

Pass is one iteration: read every store, record a sample of each, drop the history past the settings' period, and evaluate the alerts.

func (*Service) Report

func (s *Service) Report(ctx context.Context) (Report, error)

Report reads every store now, with each one's growth and projection from its history.

func (*Service) Start

func (s *Service) Start(ctx context.Context)

Start runs the sampling loop. Nil-safe.

func (*Service) Stop

func (s *Service) Stop()

Stop ends the loop and waits for it. Nil-safe; a second call is a no-op.

type Settings

type Settings struct {
	Channels []string `json:"channels"`
	// DatabaseLimitBytes is the size the database may reach: its volume, or
	// the plan's quota. The platform cannot read the database server's disk,
	// so it is the operator's. Zero is unknown.
	DatabaseLimitBytes int64 `json:"database_limit_bytes"`
	// Budgets is the byte budget per bucket, for a store the platform cannot
	// read capacity from (cloud S3). It overrides MCP_PLATFORM_STORAGE_BUDGETS
	// for every bucket it names.
	Budgets     map[string]int64      `json:"budgets"`
	SeaweedFS   SeaweedFS             `json:"seaweedfs"`
	HistoryDays int                   `json:"history_days"`
	UsedPercent levelalert.Thresholds `json:"used_percent"`
	DaysLeft    levelalert.Thresholds `json:"days_left"`
}

Settings is the stored configuration.

func Defaults

func Defaults() Settings

Defaults is the configuration before anything is written.

type SettingsStore

type SettingsStore interface {
	Get(ctx context.Context) (settingsjson.Stored[Settings], error)
	Set(ctx context.Context, v Settings, author string) error
}

SettingsStore reads and writes the section.

func NewSettingsStore

func NewSettingsStore(db *sql.DB) SettingsStore

NewSettingsStore returns the section over db.

type SpoolView

type SpoolView struct {
	Replica   string    `json:"replica"`
	Dir       string    `json:"dir"`
	Bytes     int64     `json:"bytes"`
	MaxBytes  int64     `json:"max_bytes"`
	FsTotal   *int64    `json:"fs_total_bytes"`
	FsFree    *int64    `json:"fs_free_bytes"`
	Reporting bool      `json:"reporting"`
	UpdatedAt time.Time `json:"updated_at"`
}

SpoolView is one receiver's spool and the volume it is on.

type StoreView

type StoreView struct {
	// Store is the key its history is kept under: database, seaweedfs,
	// bucket:<name>, spool:<replica>.
	Store string `json:"store"`
	Kind  string `json:"kind"`
	Name  string `json:"name"`
	// UsedBytes is what the store holds. TotalBytes is its capacity, nil
	// where the platform knows none: a database with no limit set, a bucket
	// with no budget.
	UsedBytes  int64  `json:"used_bytes"`
	TotalBytes *int64 `json:"total_bytes"`
	// CapacityFrom names where a bucket's capacity comes from: "budget", or
	// "seaweedfs" for a bucket on the cluster with no budget, whose capacity
	// is what it holds plus the cluster's free space. Empty for the rest.
	CapacityFrom string   `json:"capacity_from,omitempty"`
	FreeBytes    *int64   `json:"free_bytes"`
	PercentUsed  *float64 `json:"percent_used"`
	Forecast
	// Level is the worse of the store's two alert levels under the current
	// thresholds, empty when neither is crossed.
	Level string `json:"level,omitempty"`
}

StoreView is one store: the database, the SeaweedFS cluster, a bucket, or a replica's spool volume.

type TableView

type TableView struct {
	Name  string `json:"name"`
	Index bool   `json:"index"`
	Bytes int64  `json:"bytes"`
	Rows  int64  `json:"rows"`
}

TableView is one table or index.

type View

type View struct {
	Settings
	UpdatedBy string `json:"updated_by,omitempty"`
	UpdatedAt string `json:"updated_at,omitempty"`
}

View is the read shape: the settings with who wrote them last.

func Read

func Read(ctx context.Context, store SettingsStore) (View, error)

Read returns the stored settings, or the defaults before any are written.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL