Documentation
¶
Overview ¶
Package appdbwatch samples every application database on an interval (#2406, #2409): its health reduced to numbers, and its schema against what the applied migrations build. One replica samples at a time under an advisory lock; the samples are kept for the connection page's history and the gauges, and the leveled alerts are judged on them.
Index ¶
- Constants
- Variables
- func Assess(r *appdbhealth.Report, s Settings)
- func Bytes(n int64) string
- func DriftOf(ctx context.Context, kit appdbdrift.Kit, store appdbdrift.Migrations, ...) []appdbdrift.Result
- func Duration(secs float64) string
- func Findings(connection string, s Sample, settings Settings, link string) []levelalert.Finding
- type Deps
- type Kit
- type Point
- type Sample
- type Service
- func (s *Service) Gauges(ctx context.Context) []observability.AppDBHealthSample
- func (s *Service) History(ctx context.Context, connection string, days int) ([]Point, error)
- func (s *Service) Latest(ctx context.Context, window time.Duration) ([]Point, error)
- func (s *Service) Pass(ctx context.Context) error
- func (s *Service) SaveSettings(ctx context.Context, v Settings, author string) error
- func (s *Service) Settings(ctx context.Context) (View, error)
- func (s *Service) Start(ctx context.Context)
- func (s *Service) Stop()
- type Settings
- type SettingsStore
- type View
Constants ¶
const ( ConditionTransactionIDAge = "transaction_id_age" ConditionIdleInTransaction = "idle_in_transaction" ConditionConnections = "connections" ConditionInvalidIndexes = "invalid_indexes" ConditionDeadTuples = "dead_tuples" ConditionDrift = "drift" )
The conditions an application database is alerted on, the last part of each alert's key.
const ( DefaultIntervalSeconds = 300 DefaultHistoryDays = 30 DefaultLongRunningSeconds = 60 // MinIntervalSeconds is the shortest interval a database is sampled at. MinIntervalSeconds = 10 // MaxHistoryDays bounds how long samples are kept. MaxHistoryDays = 3650 )
Defaults.
const AlertFamily = "appdb"
AlertFamily is the levelalert family the application database alerts are recorded under.
const DefaultTick = 10 * time.Second
DefaultTick is how often each replica asks whether a pass is due; the pass itself runs once per the settings' interval.
const LockKey int64 = 4713210404
LockKey is the advisory lock one replica samples under. Storage holds 4713210402 and the webhook alerts 4713210403.
const SettingsSection = "appdb_health"
SettingsSection is the platform_settings section the health settings are kept in.
Variables ¶
var ( DefaultTransactionIDAge = levelalert.Thresholds{Warning: 500_000_000, Critical: 1_000_000_000} DefaultIdleInTransaction = levelalert.Thresholds{Warning: 300, Critical: 1800} DefaultConnectionsPercent = levelalert.Thresholds{Warning: 80, Critical: 95} DefaultInvalidIndexes = levelalert.Thresholds{Warning: 1} DefaultDeadTuplePercent = levelalert.Thresholds{Warning: 20, Critical: 50} DefaultDriftObjects = levelalert.Thresholds{Warning: 1} )
The default thresholds of each condition, a warning and a critical level.
Functions ¶
func Assess ¶ added in v1.154.1
func Assess(r *appdbhealth.Report, s Settings)
Assess judges a report against the health settings (#2528): each condition an alert is posted for, at the level its thresholds give, one problem per session, index or table past it, then the advice no alert is posted for. The status is the most severe alert level, or healthy: it is the level the alert engine posts for the same numbers, so the card and the alerts agree. It writes Status and Findings into r.
func DriftOf ¶
func DriftOf(ctx context.Context, kit appdbdrift.Kit, store appdbdrift.Migrations, connection string) []appdbdrift.Result
DriftOf checks every schema the platform keeps migrations for on connection.
Types ¶
type Deps ¶
type Deps struct {
DB *sql.DB
Settings SettingsStore
// Kit is the live postgres toolkit, nil when none is registered.
Kit func() Kit
// Migrations lists the platform's migrations, which name the schemas a
// connection's drift is checked in.
Migrations appdbdrift.Migrations
// Engine routes the alerts; nil records nothing and sends nothing.
Engine *levelalert.Engine
// PageURL is the connection page an alert links to; nil omits it.
PageURL func(connection string) string
Tick time.Duration
Now func() time.Time
}
Deps is what the service reads and sends through.
type Kit ¶
type Kit interface {
appdbhealth.Querier
appdbdrift.Kit
ListConnections() []toolkit.ConnectionDetail
}
Kit is the postgres toolkit as the loop reads it.
type Point ¶
type Point struct {
Connection string `json:"connection"`
SampledAt time.Time `json:"sampled_at"`
Sample Sample `json:"sample"`
}
Point is one stored sample.
type Sample ¶
type Sample struct {
appdbhealth.Summary
// Reachable is false when the database did not answer the server read.
Reachable bool `json:"reachable"`
// DriftObjects is how many objects differ from what the applied
// migrations build, across the schemas that could be checked, and
// DriftUnchecked how many schemas the platform keeps migrations for could
// not be (their Drift result says why).
DriftObjects int `json:"drift_objects"`
DriftUnchecked int `json:"drift_unchecked"`
// Drift is the check of each schema the platform keeps migrations for.
Drift []appdbdrift.Result `json:"drift"`
}
Sample is one connection's reading as the history keeps it.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service samples the databases, keeps their history and evaluates the alerts.
func (*Service) Gauges ¶
func (s *Service) Gauges(ctx context.Context) []observability.AppDBHealthSample
Gauges is what the health gauges report: each connection's newest sample within three intervals. A read that fails reports nothing.
func (*Service) Pass ¶
Pass is one iteration: sample every application database, keep the samples, drop the history past the settings' period, and evaluate the alerts. A pass within most of the interval of the last one is skipped: another replica, or this one, took it.
func (*Service) SaveSettings ¶
SaveSettings validates and writes the settings.
type Settings ¶
type Settings struct {
// Channels are the notification channels the alerts are sent to.
Channels []string `json:"channels"`
// IntervalSeconds is how often each application database is sampled.
IntervalSeconds int `json:"interval_seconds"`
HistoryDays int `json:"history_days"`
// LongRunningSeconds is how long a statement runs before its session is
// listed as long-running.
LongRunningSeconds int `json:"long_running_seconds"`
// TransactionIDAge is the age of the oldest unfrozen transaction id.
TransactionIDAge levelalert.Thresholds `json:"transaction_id_age"`
// IdleInTransactionSeconds is how long the longest session has been idle
// in a transaction.
IdleInTransactionSeconds levelalert.Thresholds `json:"idle_in_transaction_seconds"`
// ConnectionsPercent is the share of max_connections in use.
ConnectionsPercent levelalert.Thresholds `json:"connections_percent"`
InvalidIndexes levelalert.Thresholds `json:"invalid_indexes"`
// DeadTuplePercent is the highest dead tuple share of a table holding
// 1,000 rows or more.
DeadTuplePercent levelalert.Thresholds `json:"dead_tuple_percent"`
// DriftObjects is how many objects differ from what the applied
// migrations build.
DriftObjects levelalert.Thresholds `json:"drift_objects"`
}
Settings is how application databases are sampled and what raises an alert (#2409). A threshold of 0 is off.
type SettingsStore ¶
type SettingsStore interface {
Get(ctx context.Context) (settingsjson.Stored[Settings], error)
Set(ctx context.Context, v Settings, author string) error
}
SettingsStore reads and writes the section.
func NewSettingsStore ¶
func NewSettingsStore(db *sql.DB) SettingsStore
NewSettingsStore returns the section over db.