Documentation
¶
Overview ¶
Package appdbwatch samples every application database on an interval (#2406, #2409): its health reduced to numbers, and its schema against what the applied migrations build. One replica samples at a time under an advisory lock; the samples are kept for the connection page's history and the gauges, and the leveled alerts are judged on them.
Index ¶
- Constants
- Variables
- func DriftOf(ctx context.Context, kit appdbdrift.Kit, store appdbdrift.Migrations, ...) []appdbdrift.Result
- func Findings(connection string, s Sample, settings Settings, link string) []levelalert.Finding
- type Deps
- type Kit
- type Point
- type Sample
- type Service
- func (s *Service) Gauges(ctx context.Context) []observability.AppDBHealthSample
- func (s *Service) History(ctx context.Context, connection string, days int) ([]Point, error)
- func (s *Service) Latest(ctx context.Context, window time.Duration) ([]Point, error)
- func (s *Service) Pass(ctx context.Context) error
- func (s *Service) SaveSettings(ctx context.Context, v Settings, author string) error
- func (s *Service) Settings(ctx context.Context) (View, error)
- func (s *Service) Start(ctx context.Context)
- func (s *Service) Stop()
- type Settings
- type SettingsStore
- type View
Constants ¶
const ( ConditionTransactionIDAge = "transaction_id_age" ConditionIdleInTransaction = "idle_in_transaction" ConditionConnections = "connections" ConditionInvalidIndexes = "invalid_indexes" ConditionDeadTuples = "dead_tuples" ConditionDrift = "drift" )
The conditions an application database is alerted on, the last part of each alert's key.
const ( DefaultIntervalSeconds = 300 DefaultHistoryDays = 30 DefaultLongRunningSeconds = 60 // MinIntervalSeconds is the shortest interval a database is sampled at. MinIntervalSeconds = 10 // MaxHistoryDays bounds how long samples are kept. MaxHistoryDays = 3650 )
Defaults.
const AlertFamily = "appdb"
AlertFamily is the levelalert family the application database alerts are recorded under.
const DefaultTick = 10 * time.Second
DefaultTick is how often each replica asks whether a pass is due; the pass itself runs once per the settings' interval.
const LockKey int64 = 4713210404
LockKey is the advisory lock one replica samples under. Storage holds 4713210402 and the webhook alerts 4713210403.
const SettingsSection = "appdb_health"
SettingsSection is the platform_settings section the health settings are kept in.
Variables ¶
var ( DefaultTransactionIDAge = levelalert.Thresholds{Warning: 500_000_000, Critical: 1_000_000_000} DefaultIdleInTransaction = levelalert.Thresholds{Warning: 300, Critical: 1800} DefaultConnectionsPercent = levelalert.Thresholds{Warning: 80, Critical: 95} DefaultInvalidIndexes = levelalert.Thresholds{Warning: 1} DefaultDeadTuplePercent = levelalert.Thresholds{Warning: 20, Critical: 50} DefaultDriftObjects = levelalert.Thresholds{Warning: 1} )
The default thresholds of each condition, a warning and a critical level.
Functions ¶
func DriftOf ¶
func DriftOf(ctx context.Context, kit appdbdrift.Kit, store appdbdrift.Migrations, connection string) []appdbdrift.Result
DriftOf checks every schema the platform keeps migrations for on connection.
Types ¶
type Deps ¶
type Deps struct {
DB *sql.DB
Settings SettingsStore
// Kit is the live postgres toolkit, nil when none is registered.
Kit func() Kit
// Migrations lists the platform's migrations, which name the schemas a
// connection's drift is checked in.
Migrations appdbdrift.Migrations
// Engine routes the alerts; nil records nothing and sends nothing.
Engine *levelalert.Engine
// PageURL is the connection page an alert links to; nil omits it.
PageURL func(connection string) string
Tick time.Duration
Now func() time.Time
}
Deps is what the service reads and sends through.
type Kit ¶
type Kit interface {
appdbhealth.Querier
appdbdrift.Kit
ListConnections() []toolkit.ConnectionDetail
}
Kit is the postgres toolkit as the loop reads it.
type Point ¶
type Point struct {
Connection string `json:"connection"`
SampledAt time.Time `json:"sampled_at"`
Sample Sample `json:"sample"`
}
Point is one stored sample.
type Sample ¶
type Sample struct {
appdbhealth.Summary
// Reachable is false when the database did not answer the server read.
Reachable bool `json:"reachable"`
// DriftObjects is how many objects differ from what the applied
// migrations build, across the schemas that could be checked, and
// DriftUnchecked how many schemas the platform keeps migrations for could
// not be (their Drift result says why).
DriftObjects int `json:"drift_objects"`
DriftUnchecked int `json:"drift_unchecked"`
// Drift is the check of each schema the platform keeps migrations for.
Drift []appdbdrift.Result `json:"drift"`
}
Sample is one connection's reading as the history keeps it.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service samples the databases, keeps their history and evaluates the alerts.
func (*Service) Gauges ¶
func (s *Service) Gauges(ctx context.Context) []observability.AppDBHealthSample
Gauges is what the health gauges report: each connection's newest sample within three intervals. A read that fails reports nothing.
func (*Service) Pass ¶
Pass is one iteration: sample every application database, keep the samples, drop the history past the settings' period, and evaluate the alerts. A pass within most of the interval of the last one is skipped: another replica, or this one, took it.
func (*Service) SaveSettings ¶
SaveSettings validates and writes the settings.
type Settings ¶
type Settings struct {
// Channels are the notification channels the alerts are sent to.
Channels []string `json:"channels"`
// IntervalSeconds is how often each application database is sampled.
IntervalSeconds int `json:"interval_seconds"`
HistoryDays int `json:"history_days"`
// LongRunningSeconds is how long a statement runs before its session is
// listed as long-running.
LongRunningSeconds int `json:"long_running_seconds"`
// TransactionIDAge is the age of the oldest unfrozen transaction id.
TransactionIDAge levelalert.Thresholds `json:"transaction_id_age"`
// IdleInTransactionSeconds is how long the longest session has been idle
// in a transaction.
IdleInTransactionSeconds levelalert.Thresholds `json:"idle_in_transaction_seconds"`
// ConnectionsPercent is the share of max_connections in use.
ConnectionsPercent levelalert.Thresholds `json:"connections_percent"`
InvalidIndexes levelalert.Thresholds `json:"invalid_indexes"`
// DeadTuplePercent is the highest dead tuple share of a table holding
// 1,000 rows or more.
DeadTuplePercent levelalert.Thresholds `json:"dead_tuple_percent"`
// DriftObjects is how many objects differ from what the applied
// migrations build.
DriftObjects levelalert.Thresholds `json:"drift_objects"`
}
Settings is how application databases are sampled and what raises an alert (#2409). A threshold of 0 is off.
type SettingsStore ¶
type SettingsStore interface {
Get(ctx context.Context) (settingsjson.Stored[Settings], error)
Set(ctx context.Context, v Settings, author string) error
}
SettingsStore reads and writes the section.
func NewSettingsStore ¶
func NewSettingsStore(db *sql.DB) SettingsStore
NewSettingsStore returns the section over db.