Documentation
¶
Overview ¶
Package storagecap is the storage capacity an administrator reads and is alerted on (#2148): how much of each store is used and how much is free, the database against its limit, the object store cluster's disks, a bucket against its budget and each replica's webhook spool volume; the history of each, how fast it is growing, when it would fill at that rate, and leveled alerts on both. #1899 measured usage as metrics; this makes it a page and a notification, with capacity read from the stores where they say it.
Index ¶
- Constants
- Variables
- func BudgetsFrom(store SettingsStore) func(ctx context.Context) (map[string]int64, error)
- func Findings(r Report, settings Settings, pageURL string) []levelalert.Finding
- func Unjudged(r Report) []string
- type BucketView
- type ClusterCapacity
- type ClusterView
- type DatabaseView
- type Deps
- type Forecast
- type Input
- type Point
- type ReconcileView
- type Report
- type ScanView
- type SeaweedFS
- type Service
- type Settings
- type SettingsStore
- type SpoolView
- type StoreView
- type TableView
- type View
Constants ¶
const ( KindDatabase = "database" KindSeaweedFS = "seaweedfs" KindBucket = "bucket" KindSpool = "spool" )
The kinds of store a report and its alerts cover.
const AlertFamily = "storage"
AlertFamily is the levelalert family the storage alerts are recorded under.
const (
DefaultHistoryDays = 180
)
The defaults a deployment starts with before an administrator writes the section.
const DefaultInterval = 15 * time.Minute
DefaultInterval is how often a store is sampled and its alerts evaluated.
const LockKey int64 = 4713210402
LockKey is the advisory lock one replica samples and alerts under. The capacity scan holds 4713210401.
const SettingsSection = "storage"
SettingsSection is the platform_settings section holding the settings. It is a stored key.
Variables ¶
var ( DefaultUsedPercent = levelalert.Thresholds{Warning: 75, Critical: 90} DefaultDaysLeft = levelalert.Thresholds{Warning: 30, Critical: 7, Below: true} )
DefaultUsedPercent and DefaultDaysLeft are the alert thresholds before an administrator sets any: percent of a store used, and days until it fills at its growth rate.
Functions ¶
func BudgetsFrom ¶
BudgetsFrom reads the bucket budgets the settings hold, for the capacity gauges (#1899) to report them over the environment's.
func Findings ¶
func Findings(r Report, settings Settings, pageURL string) []levelalert.Finding
Findings is every store at warning or critical under settings, by percent used and by days until it fills, and marks each store's level on r.
func Unjudged ¶
Unjudged is the alert key prefixes of the stores this report could not read: the SeaweedFS cluster and every bucket whose capacity it gives when the cluster read failed, and each spool volume whose replica missed its recent reports without being gone. Their conditions are carried as they stand rather than resolved.
Types ¶
type BucketView ¶
type BucketView struct {
Bucket string `json:"bucket"`
Purpose string `json:"purpose"`
Bytes int64 `json:"bytes"`
Objects int64 `json:"objects"`
ListedAt *time.Time `json:"listed_at"`
}
BucketView is what the platform holds in one bucket for one purpose.
type ClusterCapacity ¶
type ClusterCapacity struct {
Total int64 `json:"total_bytes"`
Used int64 `json:"used_bytes"`
Free int64 `json:"free_bytes"`
// Servers is how many volume servers answered; Unreached lists the ones
// that did not, so a partial reading is labeled as one.
Servers int `json:"servers"`
Unreached []string `json:"unreached"`
}
ClusterCapacity is the disk the cluster's volume servers report: every disk directory of every server read, summed.
func ReadSeaweedFS ¶
func ReadSeaweedFS(ctx context.Context, client *http.Client, cfg SeaweedFS) (ClusterCapacity, error)
ReadSeaweedFS reads the cluster's disk capacity: the volume servers cfg names, or the ones the master lists, each asked for its disks.
type ClusterView ¶
type ClusterView struct {
MasterURL string `json:"master_url"`
Capacity ClusterCapacity `json:"capacity"`
Error string `json:"error,omitempty"`
}
ClusterView is the SeaweedFS cluster as last read, or why it was not.
type DatabaseView ¶
type DatabaseView struct {
Bytes int64 `json:"bytes"`
LimitBytes *int64 `json:"limit_bytes"`
Tables []TableView `json:"tables"`
}
DatabaseView is the database's size, its limit and its largest relations.
type Deps ¶
type Deps struct {
DB *sql.DB
Settings SettingsStore
// Spools reads every receiver's spool state; nil without webhooks.
Spools func(ctx context.Context) ([]whstore.SpoolState, error)
// HTTP reads the SeaweedFS cluster.
HTTP *http.Client
// EnvBudgets are the budgets MCP_PLATFORM_STORAGE_BUDGETS names, which
// a budget in the settings overrides bucket by bucket.
EnvBudgets map[string]int64
// OnSeaweedFS reports whether a bucket is on the SeaweedFS cluster: such
// a bucket with no budget can grow into the cluster's free space, which
// is then its free space. Nil names none.
OnSeaweedFS func(ctx context.Context, bucket string) bool
// Buckets are the buckets the configuration writes to, asked about
// beside those the usage rows name, so a bucket nothing was written to
// yet is still detected.
Buckets []string
// Engine routes the alerts; nil records nothing and sends nothing.
Engine *levelalert.Engine
// PageURL is the admin storage page an alert links to; empty omits it.
PageURL string
Interval time.Duration
Now func() time.Time
}
Deps is what the service reads and sends through.
type Forecast ¶
type Forecast struct {
// Rate7d and Rate30d are the growth in bytes per day fitted over the last
// 7 and 30 days, nil where the history is too short.
Rate7d *float64 `json:"rate_7d_bytes_per_day"`
Rate30d *float64 `json:"rate_30d_bytes_per_day"`
// HistoryHours is how much history the store has, first sample to last,
// and Rate7dMinHistoryHours and Rate30dMinHistoryHours how much each rate
// needs, so a reader says how far a missing rate is from being measured.
HistoryHours float64 `json:"history_hours"`
Rate7dMinHistoryHours float64 `json:"rate_7d_min_history_hours"`
Rate30dMinHistoryHours float64 `json:"rate_30d_min_history_hours"`
// RateUsed names the rate the projection uses, "7d" or "30d": the slower
// of the two that are growing, so a burst does not bring the date in.
RateUsed string `json:"rate_used,omitempty"`
// DaysLeft and ExhaustionAt are when the store fills at RateUsed, nil
// when it is not growing or has no known capacity.
DaysLeft *float64 `json:"days_left"`
ExhaustionAt *time.Time `json:"exhaustion_at"`
}
Forecast is how fast a store is growing and when it would fill.
type Input ¶
type Input struct {
Channels *[]string `json:"channels,omitempty"`
DatabaseLimitBytes *int64 `json:"database_limit_bytes,omitempty"`
Budgets *map[string]int64 `json:"budgets,omitempty"`
SeaweedFS *SeaweedFS `json:"seaweedfs,omitempty"`
HistoryDays *int `json:"history_days,omitempty"`
UsedPercent *levelalert.Thresholds `json:"used_percent,omitempty"`
DaysLeft *levelalert.Thresholds `json:"days_left,omitempty"`
}
Input is the write shape. An absent field keeps the stored value.
type Point ¶
type Point struct {
At time.Time `json:"at"`
Used int64 `json:"used_bytes"`
Total *int64 `json:"total_bytes"`
}
Point is one sample of a store.
type ReconcileView ¶
type ReconcileView struct {
Purpose string `json:"purpose"`
Orphaned int64 `json:"orphaned"`
Dangling int64 `json:"dangling"`
ReconciledAt time.Time `json:"reconciled_at"`
}
ReconcileView is what the last reconcile found for one purpose.
type Report ¶
type Report struct {
GeneratedAt time.Time `json:"generated_at"`
HistoryDays int `json:"history_days"`
Stores []StoreView `json:"stores"`
Database DatabaseView `json:"database"`
Buckets []BucketView `json:"buckets"`
Reconciles []ReconcileView `json:"reconciles"`
LastScan *ScanView `json:"last_scan"`
SeaweedFS *ClusterView `json:"seaweedfs"`
// SeaweedFSBuckets are the buckets whose endpoint answers as SeaweedFS:
// the settings offer the cluster's address only where one does.
SeaweedFSBuckets []string `json:"seaweedfs_buckets"`
Spools []SpoolView `json:"spools"`
}
Report is what GET /api/v1/admin/storage answers: every store the platform writes to, used and free, with its growth and projection, and the detail behind each.
type ScanView ¶
type ScanView struct {
DurationSeconds float64 `json:"duration_seconds"`
Objects int64 `json:"objects"`
FinishedAt time.Time `json:"finished_at"`
}
ScanView is the last full listing of the buckets.
type SeaweedFS ¶
type SeaweedFS struct {
// MasterURL is the master's HTTP address, e.g. http://seaweedfs-master:9333.
MasterURL string `json:"master_url"`
// VolumeURLs, when set, are the volume servers read instead of the ones
// the master lists: a master inside a cluster lists addresses the
// platform may not reach.
VolumeURLs []string `json:"volume_urls"`
}
SeaweedFS names the cluster the platform reads disk capacity from. The S3 gateway does not report it; the master lists the volume servers and each reports its disks.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service reads the stores, keeps their history and evaluates the alerts.
func (*Service) Pass ¶
Pass is one iteration: read every store, record a sample of each, drop the history past the settings' period, and evaluate the alerts.
func (*Service) Report ¶
Report reads every store now, with each one's growth and projection from its history.
type Settings ¶
type Settings struct {
Channels []string `json:"channels"`
// DatabaseLimitBytes is the size the database may reach: its volume, or
// the plan's quota. The platform cannot read the database server's disk,
// so it is the operator's. Zero is unknown.
DatabaseLimitBytes int64 `json:"database_limit_bytes"`
// Budgets is the byte budget per bucket, for a store the platform cannot
// read capacity from (cloud S3). It overrides MCP_PLATFORM_STORAGE_BUDGETS
// for every bucket it names.
Budgets map[string]int64 `json:"budgets"`
SeaweedFS SeaweedFS `json:"seaweedfs"`
HistoryDays int `json:"history_days"`
UsedPercent levelalert.Thresholds `json:"used_percent"`
DaysLeft levelalert.Thresholds `json:"days_left"`
}
Settings is the stored configuration.
type SettingsStore ¶
type SettingsStore interface {
Get(ctx context.Context) (settingsjson.Stored[Settings], error)
Set(ctx context.Context, v Settings, author string) error
}
SettingsStore reads and writes the section.
func NewSettingsStore ¶
func NewSettingsStore(db *sql.DB) SettingsStore
NewSettingsStore returns the section over db.
type SpoolView ¶
type SpoolView struct {
Replica string `json:"replica"`
Dir string `json:"dir"`
Bytes int64 `json:"bytes"`
MaxBytes int64 `json:"max_bytes"`
FsTotal *int64 `json:"fs_total_bytes"`
FsFree *int64 `json:"fs_free_bytes"`
Reporting bool `json:"reporting"`
UpdatedAt time.Time `json:"updated_at"`
}
SpoolView is one receiver's spool and the volume it is on.
type StoreView ¶
type StoreView struct {
// Store is the key its history is kept under: database, seaweedfs,
// bucket:<name>, spool:<replica>.
Store string `json:"store"`
Kind string `json:"kind"`
Name string `json:"name"`
// UsedBytes is what the store holds. TotalBytes is its capacity, nil
// where the platform knows none: a database with no limit set, a bucket
// with no budget.
UsedBytes int64 `json:"used_bytes"`
TotalBytes *int64 `json:"total_bytes"`
// CapacityFrom names where a bucket's capacity comes from: "budget", or
// "seaweedfs" for a bucket on the cluster with no budget, whose capacity
// is what it holds plus the cluster's free space. Empty for the rest.
CapacityFrom string `json:"capacity_from,omitempty"`
FreeBytes *int64 `json:"free_bytes"`
PercentUsed *float64 `json:"percent_used"`
Forecast
// Level is the worse of the store's two alert levels under the current
// thresholds, empty when neither is crossed.
Level string `json:"level,omitempty"`
}
StoreView is one store: the database, the SeaweedFS cluster, a bucket, or a replica's spool volume.
type TableView ¶
type TableView struct {
Name string `json:"name"`
Index bool `json:"index"`
Bytes int64 `json:"bytes"`
Rows int64 `json:"rows"`
}
TableView is one table or index.