Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DumpStmtStatsToLogBeforeReset = settings.RegisterBoolSetting( settings.TenantWritable, "sql.metrics.statement_details.dump_to_logs", "dump collected statement statistics to node logs when periodically cleared", false, ).WithPublic()
DumpStmtStatsToLogBeforeReset specifies whether we dump the statements statistics to logs before being reset.
var LogicalPlanCollectionPeriod = settings.RegisterDurationSetting( settings.TenantWritable, "sql.metrics.statement_details.plan_collection.period", "the time until a new logical plan is collected", 5*time.Minute, settings.NonNegativeDuration, ).WithPublic()
LogicalPlanCollectionPeriod specifies the interval between collections of logical plans for each fingerprint.
var MaxMemReportedSQLStatsStmtFingerprints = settings.RegisterIntSetting( settings.TenantWritable, "sql.metrics.max_mem_reported_stmt_fingerprints", "the maximum number of reported statement fingerprints stored in memory", 100000, ).WithPublic()
MaxMemReportedSQLStatsStmtFingerprints specifies the maximum of unique statement fingerprints we store in memory.
var MaxMemReportedSQLStatsTxnFingerprints = settings.RegisterIntSetting( settings.TenantWritable, "sql.metrics.max_mem_reported_txn_fingerprints", "the maximum number of reported transaction fingerprints stored in memory", 100000, ).WithPublic()
MaxMemReportedSQLStatsTxnFingerprints specifies the maximum of unique transaction fingerprints we store in memory.
var MaxMemSQLStatsStmtFingerprints = settings.RegisterIntSetting( settings.TenantWritable, "sql.metrics.max_mem_stmt_fingerprints", "the maximum number of statement fingerprints stored in memory", 100000, ).WithPublic()
MaxMemSQLStatsStmtFingerprints specifies the maximum of unique statement fingerprints we store in memory.
var MaxMemSQLStatsTxnFingerprints = settings.RegisterIntSetting( settings.TenantWritable, "sql.metrics.max_mem_txn_fingerprints", "the maximum number of transaction fingerprints stored in memory", 100000, ).WithPublic()
MaxMemSQLStatsTxnFingerprints specifies the maximum of unique transaction fingerprints we store in memory.
var MaxSQLStatReset = settings.RegisterDurationSetting( settings.TenantWritable, "diagnostics.forced_sql_stat_reset.interval", "interval after which the reported SQL Stats are reset even "+ "if not collected by telemetry reporter. It has a max value of 24H.", time.Hour*2, settings.NonNegativeDurationWithMaximum(time.Hour*24), ).WithPublic()
MaxSQLStatReset is the cluster setting that controls at what interval SQL statement statistics must be flushed within.
var MaxSQLStatsStmtFingerprintsPerExplicitTxn = settings.RegisterIntSetting( settings.TenantWritable, "sql.metrics.max_stmt_fingerprints_per_explicit_txn", "the maximum number of statement fingerprints stored per explicit transaction", 2000, )
MaxSQLStatsStmtFingerprintsPerExplicitTxn specifies the maximum of unique statement fingerprints we store for each explicit transaction.
This limit is introduced because when SQL Stats starts to record statement statistics for statements inside an explicit transaction, the transaction fingerprint ID is not known until the transaction is finished. SQL Stats holds those statement statistics in a temporary container until the explicit transaction finishes, then SQL Stats will upsert the statistics held in the temporary container into its in-memory store. However, the temporary container cannot inherit the node-level statement statistics fingerprint limit (that is: sql.metrics.max_mem_stmt_fingerprints). This is because if we count statement fingerprints inside the temporary container towards the total fingerprint count, we would be over-counting the statement fingerprint.
For example: let's suppose we execute the following transaction: * BEGIN; SELECT 1; SELECT 1, 1; COMMIT; This results in 4 statement fingerprints and 1 txn fingerprint. Let's suppose currently our statement fingerprint limit is 6. If we are to execute the same statement again:
- BEGIN; <- this increments current statement fingerprint count to 5 since we hold statement stats for explicit transaction in a temporary container before we can perform the upsert.
- SELECT 1; <- this increments the count to 6
- SELECT 1, 1; <- ERR: this causes the count to exceed our stmt fingerprint limit before we can perform the upsert.
The total amount of memory consumed will still be constrained by the top-level memory monitor created for SQL Stats.
var SampleLogicalPlans = settings.RegisterBoolSetting( settings.TenantWritable, "sql.metrics.statement_details.plan_collection.enabled", "periodically save a logical plan for each fingerprint", true, ).WithPublic()
SampleLogicalPlans specifies whether we periodically sample the logical plan for each fingerprint.
var StatsCollectionLatencyThreshold = settings.RegisterDurationSetting( settings.TenantWritable, "sql.metrics.statement_details.threshold", "minimum execution time to cause statement statistics to be collected. "+ "If configured, no transaction stats are collected.", 0, ).WithPublic()
StatsCollectionLatencyThreshold specifies the minimum amount of time consumed by a SQL statement before it is collected for statistics reporting.
var StmtStatsEnable = settings.RegisterBoolSetting( settings.TenantWritable, "sql.metrics.statement_details.enabled", "collect per-statement query statistics", true, ).WithPublic()
StmtStatsEnable determines whether to collect per-statement statistics.
var TxnStatsEnable = settings.RegisterBoolSetting( settings.TenantWritable, "sql.metrics.transaction_details.enabled", "collect per-application transaction statistics", true, ).WithPublic()
TxnStatsEnable determines whether to collect per-application transaction statistics.
var TxnStatsNumStmtFingerprintIDsToRecord = settings.RegisterIntSetting( settings.TenantWritable, "sql.metrics.transaction_details.max_statement_ids", "max number of statement fingerprint IDs to store for transaction statistics", 1000, settings.PositiveInt, )
TxnStatsNumStmtFingerprintIDsToRecord limits the number of statementFingerprintIDs stored in transactions statistics for a single transaction. This defaults to 1000, and currently is non-configurable (hidden setting).
Functions ¶
This section is empty.
Types ¶
type AggregatedTransactionVisitor ¶
AggregatedTransactionVisitor is the callback invoked when iterate through transaction statistics collected at the application level using IterateAggregatedTransactionStats(). If an error is encountered when calling the visitor, the iteration is aborted.
type ApplicationStats ¶
type ApplicationStats interface {
Reader
Writer
// MergeApplicationStatementStats merges the other application's statement
// statistics into the current ApplicationStats. It returns how many number
// of statistics were being discarded due to memory constraint. If the
// transformer is non-nil, then it is applied to other's statement statistics
// before other's statement statistics are merged into the current
// ApplicationStats.
MergeApplicationStatementStats(
ctx context.Context,
other ApplicationStats,
transformer func(statistics *roachpb.CollectedStatementStatistics),
) uint64
// MergeApplicationTransactionStats merges the other application's transaction
// statistics into the current ApplicationStats. It returns how many number
// of statistics were being discarded due to memory constraint.
MergeApplicationTransactionStats(
ctx context.Context,
other ApplicationStats,
) uint64
// NewApplicationStatsWithInheritedOptions returns a new ApplicationStats
// interface that inherits all memory limits of the existing
NewApplicationStatsWithInheritedOptions() ApplicationStats
// Free frees the current ApplicationStats and zeros out the memory counts
// and fingerprint counts.
Free(context.Context)
}
ApplicationStats is an interface to read from or write to the statistics belongs to an application.
type IteratorOptions ¶
type IteratorOptions struct {
// SortedAppNames determines whether or not the application names will be
// sorted when iterating through statistics.
SortedAppNames bool
// SortedKey determines whether the key of the statistics will be sorted
// when iterating through statistics;
SortedKey bool
}
IteratorOptions provides the ability to the caller to change how it iterates the statements and transactions. TODO(azhng): introduce StartTime and EndTime field so we can implement
virtual indexes on crdb_internal.{statement,transaction}_statistics
using the iterators.
type Reader ¶
type Reader interface {
// IterateStatementStats iterates through all the collected statement statistics
// by using StatementVisitor. Caller can specify iteration behavior, such
// as ordering, through IteratorOptions argument. StatementVisitor can return
// error, if an error is returned after the execution of the visitor, the
// iteration is aborted.
IterateStatementStats(context.Context, *IteratorOptions, StatementVisitor) error
// IterateTransactionStats iterates through all the collected transaction
// statistics by using TransactionVisitor. It behaves similarly to
// IterateStatementStats.
IterateTransactionStats(context.Context, *IteratorOptions, TransactionVisitor) error
// IterateAggregatedTransactionStats iterates through all the collected app-level
// transactions statistics. It behaves similarly to IterateStatementStats.
IterateAggregatedTransactionStats(context.Context, *IteratorOptions, AggregatedTransactionVisitor) error
}
Reader provides methods to retrieve transaction/statement statistics from the Storage.
type RecordedStmtStats ¶
type RecordedStmtStats struct {
AutoRetryCount int
RowsAffected int
ParseLatency float64
PlanLatency float64
RunLatency float64
ServiceLatency float64
OverheadLatency float64
BytesRead int64
RowsRead int64
RowsWritten int64
Nodes []int64
StatementType tree.StatementType
Plan *roachpb.ExplainTreePlanNode
PlanGist string
StatementError error
}
RecordedStmtStats stores the statistics of a statement to be recorded.
type RecordedTxnStats ¶
type RecordedTxnStats struct {
TransactionTimeSec float64
Committed bool
ImplicitTxn bool
RetryCount int64
StatementFingerprintIDs []roachpb.StmtFingerprintID
ServiceLatency time.Duration
RetryLatency time.Duration
CommitLatency time.Duration
RowsAffected int
CollectedExecStats bool
ExecStats execstats.QueryLevelStats
RowsRead int64
RowsWritten int64
BytesRead int64
}
RecordedTxnStats stores the statistics of a transaction to be recorded.
type StatementVisitor ¶
type StatementVisitor func(context.Context, *roachpb.CollectedStatementStatistics) error
StatementVisitor is the callback that is invoked when caller iterate through all statement statistics using IterateStatementStats(). If an error is encountered when calling the visitor, the iteration is aborted.
type StatsCollector ¶
type StatsCollector interface {
Writer
// PhaseTimes returns the sessionphase.Times that this StatsCollector is
// currently tracking.
PhaseTimes() *sessionphase.Times
// PreviousPhaseTimes returns the sessionphase.Times that this StatsCollector
// was previously tracking before being Reset.
PreviousPhaseTimes() *sessionphase.Times
// Reset resets the StatsCollector with a new ApplicationStats and a new copy
// of the sessionphase.Times.
Reset(ApplicationStats, *sessionphase.Times)
// StartExplicitTransaction informs StatsCollector that all subsequent
// statements will be executed in the context an explicit transaction.
StartExplicitTransaction()
// EndExplicitTransaction informs the StatsCollector that the explicit txn has
// finished execution. (Either COMMITTED or ABORTED). This means the txn's
// fingerprint ID is now available. StatsCollector will now go back to update
// the transaction fingerprint ID field of all the statement statistics for that
// txn.
EndExplicitTransaction(ctx context.Context, transactionFingerprintID roachpb.TransactionFingerprintID,
)
}
StatsCollector is an interface that collects statistics for transactions and statements for the entire lifetime of a session.
type Storage ¶
type Storage interface {
Reader
// GetLastReset returns the last time when the sqlstats is being reset.
GetLastReset() time.Time
// GetApplicationStats returns an ApplicationStats instance for the given
// application name.
GetApplicationStats(appName string) ApplicationStats
// Reset resets all the statistics stored in-memory in the current Storage.
Reset(context.Context) error
}
Storage provides clients with interface to perform read and write operations to sql statistics.
type TestingKnobs ¶
type TestingKnobs struct {
// OnStmtStatsFlushFinished is a callback that is triggered when stmt stats
// finishes flushing.
OnStmtStatsFlushFinished func()
// OnTxnStatsFlushFinished is a callback that is triggered when txn stats
// finishes flushing.
OnTxnStatsFlushFinished func()
// StubTimeNow allows tests to override the timeutil.Now() function used
// by the flush operation to calculate aggregated_ts timestamp.
StubTimeNow func() time.Time
// AOSTClause overrides the AS OF SYSTEM TIME clause in queries used in
// persistedsqlstats.
AOSTClause string
}
TestingKnobs provides hooks and knobs for unit tests.
func (*TestingKnobs) GetAOSTClause ¶
func (knobs *TestingKnobs) GetAOSTClause() string
GetAOSTClause returns the appropriate AS OF SYSTEM TIME clause to be used when reading from statements and transactions system tables.
func (*TestingKnobs) ModuleTestingKnobs ¶
func (*TestingKnobs) ModuleTestingKnobs()
ModuleTestingKnobs implements base.ModuleTestingKnobs interface.
type TransactionVisitor ¶
type TransactionVisitor func(context.Context, *roachpb.CollectedTransactionStatistics) error
TransactionVisitor is the callback that is invoked when caller iterate through all transaction statistics using IterateTransactionStats(). If an error is encountered when calling the visitor, the iteration is aborted.
type Writer ¶
type Writer interface {
// RecordStatement records statistics for a statement.
RecordStatement(ctx context.Context, key roachpb.StatementStatisticsKey, value RecordedStmtStats) (roachpb.StmtFingerprintID, error)
// RecordStatementExecStats records execution statistics for a statement.
// This is sampled and not recorded for every single statement.
RecordStatementExecStats(key roachpb.StatementStatisticsKey, stats execstats.QueryLevelStats) error
// ShouldSaveLogicalPlanDesc returns whether we should save the logical plan
// description for a given combination of statement metadata.
ShouldSaveLogicalPlanDesc(fingerprint string, implicitTxn bool, database string) bool
// RecordTransaction records statistics for a transaction.
RecordTransaction(ctx context.Context, key roachpb.TransactionFingerprintID, value RecordedTxnStats) error
}
Writer is the interface that provides methods to record statement and transaction stats.