sqlstats

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 18, 2022 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var DumpStmtStatsToLogBeforeReset = settings.RegisterBoolSetting(
	settings.TenantWritable,
	"sql.metrics.statement_details.dump_to_logs",
	"dump collected statement statistics to node logs when periodically cleared",
	false,
).WithPublic()

DumpStmtStatsToLogBeforeReset specifies whether we dump the statements statistics to logs before being reset.

View Source
var LogicalPlanCollectionPeriod = settings.RegisterDurationSetting(
	settings.TenantWritable,
	"sql.metrics.statement_details.plan_collection.period",
	"the time until a new logical plan is collected",
	5*time.Minute,
	settings.NonNegativeDuration,
).WithPublic()

LogicalPlanCollectionPeriod specifies the interval between collections of logical plans for each fingerprint.

View Source
var MaxMemReportedSQLStatsStmtFingerprints = settings.RegisterIntSetting(
	settings.TenantWritable,
	"sql.metrics.max_mem_reported_stmt_fingerprints",
	"the maximum number of reported statement fingerprints stored in memory",
	100000,
).WithPublic()

MaxMemReportedSQLStatsStmtFingerprints specifies the maximum of unique statement fingerprints we store in memory.

View Source
var MaxMemReportedSQLStatsTxnFingerprints = settings.RegisterIntSetting(
	settings.TenantWritable,
	"sql.metrics.max_mem_reported_txn_fingerprints",
	"the maximum number of reported transaction fingerprints stored in memory",
	100000,
).WithPublic()

MaxMemReportedSQLStatsTxnFingerprints specifies the maximum of unique transaction fingerprints we store in memory.

View Source
var MaxMemSQLStatsStmtFingerprints = settings.RegisterIntSetting(
	settings.TenantWritable,
	"sql.metrics.max_mem_stmt_fingerprints",
	"the maximum number of statement fingerprints stored in memory",
	100000,
).WithPublic()

MaxMemSQLStatsStmtFingerprints specifies the maximum of unique statement fingerprints we store in memory.

View Source
var MaxMemSQLStatsTxnFingerprints = settings.RegisterIntSetting(
	settings.TenantWritable,
	"sql.metrics.max_mem_txn_fingerprints",
	"the maximum number of transaction fingerprints stored in memory",
	100000,
).WithPublic()

MaxMemSQLStatsTxnFingerprints specifies the maximum of unique transaction fingerprints we store in memory.

View Source
var MaxSQLStatReset = settings.RegisterDurationSetting(
	settings.TenantWritable,
	"diagnostics.forced_sql_stat_reset.interval",
	"interval after which the reported SQL Stats are reset even "+
		"if not collected by telemetry reporter. It has a max value of 24H.",
	time.Hour*2,
	settings.NonNegativeDurationWithMaximum(time.Hour*24),
).WithPublic()

MaxSQLStatReset is the cluster setting that controls at what interval SQL statement statistics must be flushed within.

View Source
var MaxSQLStatsStmtFingerprintsPerExplicitTxn = settings.RegisterIntSetting(
	settings.TenantWritable,
	"sql.metrics.max_stmt_fingerprints_per_explicit_txn",
	"the maximum number of statement fingerprints stored per explicit transaction",
	2000,
)

MaxSQLStatsStmtFingerprintsPerExplicitTxn specifies the maximum of unique statement fingerprints we store for each explicit transaction.

This limit is introduced because when SQL Stats starts to record statement statistics for statements inside an explicit transaction, the transaction fingerprint ID is not known until the transaction is finished. SQL Stats holds those statement statistics in a temporary container until the explicit transaction finishes, then SQL Stats will upsert the statistics held in the temporary container into its in-memory store. However, the temporary container cannot inherit the node-level statement statistics fingerprint limit (that is: sql.metrics.max_mem_stmt_fingerprints). This is because if we count statement fingerprints inside the temporary container towards the total fingerprint count, we would be over-counting the statement fingerprint.

For example: let's suppose we execute the following transaction: * BEGIN; SELECT 1; SELECT 1, 1; COMMIT; This results in 4 statement fingerprints and 1 txn fingerprint. Let's suppose currently our statement fingerprint limit is 6. If we are to execute the same statement again:

  • BEGIN; <- this increments current statement fingerprint count to 5 since we hold statement stats for explicit transaction in a temporary container before we can perform the upsert.
  • SELECT 1; <- this increments the count to 6
  • SELECT 1, 1; <- ERR: this causes the count to exceed our stmt fingerprint limit before we can perform the upsert.

The total amount of memory consumed will still be constrained by the top-level memory monitor created for SQL Stats.

View Source
var SampleLogicalPlans = settings.RegisterBoolSetting(
	settings.TenantWritable,
	"sql.metrics.statement_details.plan_collection.enabled",
	"periodically save a logical plan for each fingerprint",
	true,
).WithPublic()

SampleLogicalPlans specifies whether we periodically sample the logical plan for each fingerprint.

View Source
var StatsCollectionLatencyThreshold = settings.RegisterDurationSetting(
	settings.TenantWritable,
	"sql.metrics.statement_details.threshold",
	"minimum execution time to cause statement statistics to be collected. "+
		"If configured, no transaction stats are collected.",
	0,
).WithPublic()

StatsCollectionLatencyThreshold specifies the minimum amount of time consumed by a SQL statement before it is collected for statistics reporting.

View Source
var StmtStatsEnable = settings.RegisterBoolSetting(
	settings.TenantWritable,
	"sql.metrics.statement_details.enabled", "collect per-statement query statistics", true,
).WithPublic()

StmtStatsEnable determines whether to collect per-statement statistics.

View Source
var TxnStatsEnable = settings.RegisterBoolSetting(
	settings.TenantWritable,
	"sql.metrics.transaction_details.enabled", "collect per-application transaction statistics", true,
).WithPublic()

TxnStatsEnable determines whether to collect per-application transaction statistics.

View Source
var TxnStatsNumStmtFingerprintIDsToRecord = settings.RegisterIntSetting(
	settings.TenantWritable,
	"sql.metrics.transaction_details.max_statement_ids",
	"max number of statement fingerprint IDs to store for transaction statistics",
	1000,
	settings.PositiveInt,
)

TxnStatsNumStmtFingerprintIDsToRecord limits the number of statementFingerprintIDs stored in transactions statistics for a single transaction. This defaults to 1000, and currently is non-configurable (hidden setting).

Functions

This section is empty.

Types

type AggregatedTransactionVisitor

type AggregatedTransactionVisitor func(appName string, statistics *roachpb.TxnStats) error

AggregatedTransactionVisitor is the callback invoked when iterate through transaction statistics collected at the application level using IterateAggregatedTransactionStats(). If an error is encountered when calling the visitor, the iteration is aborted.

type ApplicationStats

type ApplicationStats interface {
	Reader
	Writer

	// MergeApplicationStatementStats merges the other application's statement
	// statistics into the current ApplicationStats. It returns how many number
	// of statistics were being discarded due to memory constraint. If the
	// transformer is non-nil, then it is applied to other's statement statistics
	// before other's statement statistics are merged into the current
	// ApplicationStats.
	MergeApplicationStatementStats(
		ctx context.Context,
		other ApplicationStats,
		transformer func(statistics *roachpb.CollectedStatementStatistics),
	) uint64

	// MergeApplicationTransactionStats merges the other application's transaction
	// statistics into the current ApplicationStats. It returns how many number
	// of statistics were being discarded due to memory constraint.
	MergeApplicationTransactionStats(
		ctx context.Context,
		other ApplicationStats,
	) uint64

	// NewApplicationStatsWithInheritedOptions returns a new ApplicationStats
	// interface that inherits all memory limits of the existing
	NewApplicationStatsWithInheritedOptions() ApplicationStats

	// Free frees the current ApplicationStats and zeros out the memory counts
	// and fingerprint counts.
	Free(context.Context)
}

ApplicationStats is an interface to read from or write to the statistics belongs to an application.

type IteratorOptions

type IteratorOptions struct {
	// SortedAppNames determines whether or not the application names will be
	// sorted when iterating through statistics.
	SortedAppNames bool

	// SortedKey determines whether the key of the statistics will be sorted
	// when iterating through statistics;
	SortedKey bool
}

IteratorOptions provides the ability to the caller to change how it iterates the statements and transactions. TODO(azhng): introduce StartTime and EndTime field so we can implement

virtual indexes on crdb_internal.{statement,transaction}_statistics
using the iterators.

type Provider

type Provider interface {
	Storage

	Start(ctx context.Context, stopper *stop.Stopper)
}

Provider is a wrapper around sqlstats subsystem for external consumption.

type Reader

type Reader interface {
	// IterateStatementStats iterates through all the collected statement statistics
	// by using StatementVisitor. Caller can specify iteration behavior, such
	// as ordering, through IteratorOptions argument. StatementVisitor can return
	// error, if an error is returned after the execution of the visitor, the
	// iteration is aborted.
	IterateStatementStats(context.Context, *IteratorOptions, StatementVisitor) error

	// IterateTransactionStats iterates through all the collected transaction
	// statistics by using TransactionVisitor. It behaves similarly to
	// IterateStatementStats.
	IterateTransactionStats(context.Context, *IteratorOptions, TransactionVisitor) error

	// IterateAggregatedTransactionStats iterates through all the collected app-level
	// transactions statistics. It behaves similarly to IterateStatementStats.
	IterateAggregatedTransactionStats(context.Context, *IteratorOptions, AggregatedTransactionVisitor) error
}

Reader provides methods to retrieve transaction/statement statistics from the Storage.

type RecordedStmtStats

type RecordedStmtStats struct {
	AutoRetryCount  int
	RowsAffected    int
	ParseLatency    float64
	PlanLatency     float64
	RunLatency      float64
	ServiceLatency  float64
	OverheadLatency float64
	BytesRead       int64
	RowsRead        int64
	RowsWritten     int64
	Nodes           []int64
	StatementType   tree.StatementType
	Plan            *roachpb.ExplainTreePlanNode
	PlanGist        string
	StatementError  error
}

RecordedStmtStats stores the statistics of a statement to be recorded.

type RecordedTxnStats

type RecordedTxnStats struct {
	TransactionTimeSec      float64
	Committed               bool
	ImplicitTxn             bool
	RetryCount              int64
	StatementFingerprintIDs []roachpb.StmtFingerprintID
	ServiceLatency          time.Duration
	RetryLatency            time.Duration
	CommitLatency           time.Duration
	RowsAffected            int
	CollectedExecStats      bool
	ExecStats               execstats.QueryLevelStats
	RowsRead                int64
	RowsWritten             int64
	BytesRead               int64
}

RecordedTxnStats stores the statistics of a transaction to be recorded.

type StatementVisitor

type StatementVisitor func(context.Context, *roachpb.CollectedStatementStatistics) error

StatementVisitor is the callback that is invoked when caller iterate through all statement statistics using IterateStatementStats(). If an error is encountered when calling the visitor, the iteration is aborted.

type StatsCollector

type StatsCollector interface {
	Writer

	// PhaseTimes returns the sessionphase.Times that this StatsCollector is
	// currently tracking.
	PhaseTimes() *sessionphase.Times

	// PreviousPhaseTimes returns the sessionphase.Times that this StatsCollector
	// was previously tracking before being Reset.
	PreviousPhaseTimes() *sessionphase.Times

	// Reset resets the StatsCollector with a new ApplicationStats and a new copy
	// of the sessionphase.Times.
	Reset(ApplicationStats, *sessionphase.Times)

	// StartExplicitTransaction informs StatsCollector that all subsequent
	// statements will be executed in the context an explicit transaction.
	StartExplicitTransaction()

	// EndExplicitTransaction informs the StatsCollector that the explicit txn has
	// finished execution. (Either COMMITTED or ABORTED). This means the txn's
	// fingerprint ID is now available. StatsCollector will now go back to update
	// the transaction fingerprint ID field of all the statement statistics for that
	// txn.
	EndExplicitTransaction(ctx context.Context, transactionFingerprintID roachpb.TransactionFingerprintID,
	)
}

StatsCollector is an interface that collects statistics for transactions and statements for the entire lifetime of a session.

type Storage

type Storage interface {
	Reader

	// GetLastReset returns the last time when the sqlstats is being reset.
	GetLastReset() time.Time

	// GetApplicationStats returns an ApplicationStats instance for the given
	// application name.
	GetApplicationStats(appName string) ApplicationStats

	// Reset resets all the statistics stored in-memory in the current Storage.
	Reset(context.Context) error
}

Storage provides clients with interface to perform read and write operations to sql statistics.

type TestingKnobs

type TestingKnobs struct {
	// OnStmtStatsFlushFinished is a callback that is triggered when stmt stats
	// finishes flushing.
	OnStmtStatsFlushFinished func()

	// OnTxnStatsFlushFinished is a callback that is triggered when txn stats
	// finishes flushing.
	OnTxnStatsFlushFinished func()

	// StubTimeNow allows tests to override the timeutil.Now() function used
	// by the flush operation to calculate aggregated_ts timestamp.
	StubTimeNow func() time.Time

	// AOSTClause overrides the AS OF SYSTEM TIME clause in queries used in
	// persistedsqlstats.
	AOSTClause string
}

TestingKnobs provides hooks and knobs for unit tests.

func (*TestingKnobs) GetAOSTClause

func (knobs *TestingKnobs) GetAOSTClause() string

GetAOSTClause returns the appropriate AS OF SYSTEM TIME clause to be used when reading from statements and transactions system tables.

func (*TestingKnobs) ModuleTestingKnobs

func (*TestingKnobs) ModuleTestingKnobs()

ModuleTestingKnobs implements base.ModuleTestingKnobs interface.

type TransactionVisitor

type TransactionVisitor func(context.Context, *roachpb.CollectedTransactionStatistics) error

TransactionVisitor is the callback that is invoked when caller iterate through all transaction statistics using IterateTransactionStats(). If an error is encountered when calling the visitor, the iteration is aborted.

type Writer

type Writer interface {
	// RecordStatement records statistics for a statement.
	RecordStatement(ctx context.Context, key roachpb.StatementStatisticsKey, value RecordedStmtStats) (roachpb.StmtFingerprintID, error)

	// RecordStatementExecStats records execution statistics for a statement.
	// This is sampled and not recorded for every single statement.
	RecordStatementExecStats(key roachpb.StatementStatisticsKey, stats execstats.QueryLevelStats) error

	// ShouldSaveLogicalPlanDesc returns whether we should save the logical plan
	// description for a given combination of statement metadata.
	ShouldSaveLogicalPlanDesc(fingerprint string, implicitTxn bool, database string) bool

	// RecordTransaction records statistics for a transaction.
	RecordTransaction(ctx context.Context, key roachpb.TransactionFingerprintID, value RecordedTxnStats) error
}

Writer is the interface that provides methods to record statement and transaction stats.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL