handle

package
v1.1.0-beta.0...-262327f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 11, 2023 License: Apache-2.0 Imports: 48 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// TiDBGlobalStats represents the global-stats for a partitioned table.
	TiDBGlobalStats = "global"
	// MaxPartitionMergeBatchSize indicates the max batch size for a worker to merge partition stats
	MaxPartitionMergeBatchSize = 256
)
View Source 
const (
	// StatsMetaHistorySourceAnalyze indicates stats history meta source from analyze
	StatsMetaHistorySourceAnalyze = "analyze"
	// StatsMetaHistorySourceLoadStats indicates stats history meta source from load stats
	StatsMetaHistorySourceLoadStats = "load stats"
	// StatsMetaHistorySourceFlushStats indicates stats history meta source from flush stats
	StatsMetaHistorySourceFlushStats = "flush stats"
	// StatsMetaHistorySourceExtendedStats indicates stats history meta source from extended stats
	StatsMetaHistorySourceExtendedStats = "extended stats"
	// StatsMetaHistorySourceSchemaChange indicates stats history meta source from schema change
	StatsMetaHistorySourceSchemaChange = "schema change"
)
View Source 
const (
	// DumpAll indicates dump all the delta info in to kv.
	DumpAll dumpMode = true
	// DumpDelta indicates dump part of the delta info in to kv.
	DumpDelta dumpMode = false
)
View Source 
const (
	// StatsOwnerKey is the stats owner path that is saved to etcd.
	StatsOwnerKey = "/tidb/stats/owner"
	// StatsPrompt is the prompt for stats owner manager.
	StatsPrompt = "stats"
)

Variables

View Source 
var AutoAnalyzeMinCnt int64 = 1000

AutoAnalyzeMinCnt means if the count of table is less than this value, we needn't do auto analyze.

View Source 
var (
	// DumpStatsDeltaRatio is the lower bound of `Modify Count / Table Count` for stats delta to be dumped.
	DumpStatsDeltaRatio = 1 / 10000.0
)

Functions

func DurationToTS 

func DurationToTS(d time.Duration) uint64

DurationToTS converts duration to timestamp.

func JSONTableToBlocks 

func JSONTableToBlocks(jsTable *JSONTable, blockSize int) ([][]byte, error)

JSONTableToBlocks convert JSONTable to json, then compresses it to blocks by gzip.

func MergeGlobalStatsTopNByConcurrency 

func MergeGlobalStatsTopNByConcurrency(gp *gp.Pool, mergeConcurrency, mergeBatchSize int, wrapper *statistics.StatsWrapper,
	timeZone *time.Location, version int, n uint32, isIndex bool, killed *uint32) (*statistics.TopN,
	[]statistics.TopNMeta, []*statistics.Histogram, error)

MergeGlobalStatsTopNByConcurrency merge partition topN by concurrency To merge global stats topn by concurrency, we will separate the partition topn in concurrency part and deal it with different worker. mergeConcurrency is used to control the total concurrency of the running worker, and mergeBatchSize is sued to control the partition size for each worker to solve it

func NeedAnalyzeTable 

func NeedAnalyzeTable(tbl *statistics.Table, _ time.Duration, autoAnalyzeRatio float64) (bool, string)

NeedAnalyzeTable checks if we need to analyze the table:

  1. If the table has never been analyzed, we need to analyze it when it has not been modified for a while.
  2. If the table had been analyzed before, we need to analyze it when "tbl.ModifyCount/tbl.Count > autoAnalyzeRatio" and the current time is between `start` and `end`.

func SaveTableStatsToStorage 

func SaveTableStatsToStorage(sctx sessionctx.Context, results *statistics.AnalyzeResults, analyzeSnapshot bool, source string) (err error)

SaveTableStatsToStorage saves the stats of a table to storage.

func TableAnalyzed 

func TableAnalyzed(tbl *statistics.Table) bool

TableAnalyzed checks if the table is analyzed.

func TableStatsFromJSON 

func TableStatsFromJSON(tableInfo *model.TableInfo, physicalID int64, jsonTbl *JSONTable) (*statistics.Table, error)

TableStatsFromJSON loads statistic from JSONTable and return the Table of statistic.

Types

type GlobalIndexID 

type GlobalIndexID struct {
	TableID int64
	IndexID int64
}

GlobalIndexID is the key type for indexUsageMap.

type GlobalStats 

type GlobalStats struct {
	Hg                    []*statistics.Histogram
	Cms                   []*statistics.CMSketch
	TopN                  []*statistics.TopN
	Fms                   []*statistics.FMSketch
	MissingPartitionStats []string
	Num                   int
	Count                 int64
	ModifyCount           int64
}

GlobalStats is used to store the statistics contained in the global-level stats which is generated by the merge of partition-level stats. It will both store the column stats and index stats. In the column statistics, the variable `num` is equal to the number of columns in the partition table. In the index statistics, the variable `num` is always equal to one.

type Handle 

type Handle struct {
	InitStatsDone chan struct{}

	// StatsLoad is used to load stats concurrently
	StatsLoad StatsLoad
	// contains filtered or unexported fields
}

Handle can update stats info periodically.

func NewHandle 

func NewHandle(ctx, initStatsCtx sessionctx.Context, lease time.Duration, pool sessionPool, tracker sessionctx.SysProcTracker, autoAnalyzeProcIDGetter func() uint64) (*Handle, error)

NewHandle creates a Handle for update stats.

func (*Handle) AddLockedTables 

func (h *Handle) AddLockedTables(tids []int64, pids []int64, tables []*ast.TableName) (string, error)

AddLockedTables add locked tables id to store. - tids: table ids of which will be locked. - pids: partition ids of which will be locked. - tables: table names of which will be locked. Return the message of skipped tables and error.

func (*Handle) AppendNeededItem 

func (h *Handle) AppendNeededItem(task *NeededItemTask, timeout time.Duration) error

AppendNeededItem appends needed columns/indices to ch, it is only used for test

func (*Handle) BuildExtendedStats 

func (h *Handle) BuildExtendedStats(tableID int64, cols []*model.ColumnInfo, collectors []*statistics.SampleCollector) (*statistics.ExtendedStatsColl, error)

BuildExtendedStats build extended stats for column groups if needed based on the column samples.

func (*Handle) CheckAnalyzeVersion 

func (h *Handle) CheckAnalyzeVersion(tblInfo *model.TableInfo, physicalIDs []int64, version *int) bool

CheckAnalyzeVersion checks whether all the statistics versions of this table's columns and indexes are the same.

func (*Handle) CheckHistoricalStatsEnable 

func (h *Handle) CheckHistoricalStatsEnable() (enable bool, err error)

CheckHistoricalStatsEnable is used to check whether TiDBEnableHistoricalStats is enabled.

func (*Handle) Clear 

func (h *Handle) Clear()

Clear the statsCache, only for test.

func (*Handle) ClearOutdatedHistoryStats 

func (h *Handle) ClearOutdatedHistoryStats() error

ClearOutdatedHistoryStats clear outdated historical stats

func (*Handle) Close 

func (h *Handle) Close()

Close stops the background

func (*Handle) CollectColumnsInExtendedStats 

func (h *Handle) CollectColumnsInExtendedStats(tableID int64) ([]int64, error)

CollectColumnsInExtendedStats returns IDs of the columns involved in extended stats.

func (*Handle) CurrentPruneMode 

func (h *Handle) CurrentPruneMode() variable.PartitionPruneMode

CurrentPruneMode indicates whether tbl support runtime prune for table and first partition id.

func (*Handle) DDLEventCh 

func (h *Handle) DDLEventCh() chan *util.Event

DDLEventCh returns ddl events channel in handle.

func (*Handle) DeleteAnalyzeJobs 

func (h *Handle) DeleteAnalyzeJobs(updateTime time.Time) error

DeleteAnalyzeJobs deletes the analyze jobs whose update time is earlier than updateTime.

func (*Handle) DeleteTableStatsFromKV 

func (h *Handle) DeleteTableStatsFromKV(statsIDs []int64) (err error)

DeleteTableStatsFromKV deletes table statistics from kv. A statsID refers to statistic of a table or a partition.

func (*Handle) DumpColStatsUsageToKV 

func (h *Handle) DumpColStatsUsageToKV() error

DumpColStatsUsageToKV sweeps the whole list, updates the column stats usage map and dumps it to KV.

func (*Handle) DumpHistoricalStatsBySnapshot 

func (h *Handle) DumpHistoricalStatsBySnapshot(
	dbName string,
	tableInfo *model.TableInfo,
	snapshot uint64,
) (
	jt *JSONTable,
	fallbackTbls []string,
	err error,
)

DumpHistoricalStatsBySnapshot dumped json tables from mysql.stats_meta_history and mysql.stats_history. As implemented in getTableHistoricalStatsToJSONWithFallback, if historical stats are nonexistent, it will fall back to the latest stats, and these table names (and partition names) will be returned in fallbackTbls.

func (*Handle) DumpIndexUsageToKV 

func (h *Handle) DumpIndexUsageToKV() error

DumpIndexUsageToKV will dump in-memory index usage information to KV.

func (*Handle) DumpStatsDeltaToKV 

func (h *Handle) DumpStatsDeltaToKV(mode dumpMode) error

DumpStatsDeltaToKV sweeps the whole list and updates the global map, then we dumps every table that held in map to KV. If the mode is `DumpDelta`, it will only dump that delta info that `Modify Count / Table Count` greater than a ratio.

func (*Handle) DumpStatsToJSON 

func (h *Handle) DumpStatsToJSON(dbName string, tableInfo *model.TableInfo,
	historyStatsExec sqlexec.RestrictedSQLExecutor, dumpPartitionStats bool) (*JSONTable, error)

DumpStatsToJSON dumps statistic to json.

func (*Handle) DumpStatsToJSONBySnapshot 

func (h *Handle) DumpStatsToJSONBySnapshot(dbName string, tableInfo *model.TableInfo, snapshot uint64, dumpPartitionStats bool) (*JSONTable, error)

DumpStatsToJSONBySnapshot dumps statistic to json.

func (*Handle) FlushStats 

func (h *Handle) FlushStats()

FlushStats flushes the cached stats update into store.

func (*Handle) GCIndexUsage 

func (h *Handle) GCIndexUsage() error

GCIndexUsage will delete the usage information of those indexes that do not exist.

func (*Handle) GCStats 

func (h *Handle) GCStats(is infoschema.InfoSchema, ddlLease time.Duration) error

GCStats will garbage collect the useless stats info. For dropped tables, we will first update their version so that other tidb could know that table is deleted.

func (*Handle) GetLockedTables 

func (h *Handle) GetLockedTables(tableIDs ...int64) (map[int64]struct{}, error)

GetLockedTables returns the locked status of the given tables. Note: This function query locked tables from store, so please try to batch the query.

func (*Handle) GetMemConsumed 

func (h *Handle) GetMemConsumed() (size int64)

GetMemConsumed returns the mem size of statscache consumed

func (*Handle) GetPartitionStats 

func (h *Handle) GetPartitionStats(tblInfo *model.TableInfo, pid int64, opts ...cache.TableStatsOpt) *statistics.Table

GetPartitionStats retrieves the partition stats from cache.

func (*Handle) GetPredicateColumns 

func (h *Handle) GetPredicateColumns(tableID int64) ([]int64, error)

GetPredicateColumns returns IDs of predicate columns, which are the columns whose stats are used(needed) when generating query plans.

func (*Handle) GetTableLockedAndClearForTest 

func (h *Handle) GetTableLockedAndClearForTest() (map[int64]struct{}, error)

GetTableLockedAndClearForTest for unit test only.

func (*Handle) GetTableStats 

func (h *Handle) GetTableStats(tblInfo *model.TableInfo, opts ...cache.TableStatsOpt) *statistics.Table

GetTableStats retrieves the statistics table from cache, and the cache will be updated by a goroutine.

func (*Handle) HandleAutoAnalyze 

func (h *Handle) HandleAutoAnalyze(is infoschema.InfoSchema) (analyzed bool)

HandleAutoAnalyze analyzes the newly created table or index.

func (*Handle) HandleDDLEvent 

func (h *Handle) HandleDDLEvent(t *util.Event) error

HandleDDLEvent begins to process a ddl task.

func (*Handle) HandleOneTask 

func (h *Handle) HandleOneTask(lastTask *NeededItemTask, readerCtx *StatsReaderContext, ctx sqlexec.RestrictedSQLExecutor, exit chan struct{}) (task *NeededItemTask, err error)

HandleOneTask handles last task if not nil, else handle a new task from chan, and return current task if fail somewhere.

func (*Handle) InitStats 

func (h *Handle) InitStats(is infoschema.InfoSchema) (err error)

InitStats initiates the stats cache. Index/PK stats are fully loaded. Column stats are not loaded, i.e., we only load scalars such as NDV, NullCount, Correlation and don't load CMSketch/Histogram/TopN.

func (*Handle) InitStatsLite 

func (h *Handle) InitStatsLite(is infoschema.InfoSchema) (err error)

InitStatsLite initiates the stats cache. The function is liter and faster than InitStats. Column/index stats are not loaded, i.e., we only load scalars such as NDV, NullCount, Correlation and don't load CMSketch/Histogram/TopN.

func (*Handle) InsertAnalyzeJob 

func (h *Handle) InsertAnalyzeJob(job *statistics.AnalyzeJob, instance string, procID uint64) error

InsertAnalyzeJob inserts analyze job into mysql.analyze_jobs and gets job ID for further updating job.

func (*Handle) InsertExtendedStats 

func (h *Handle) InsertExtendedStats(statsName string, colIDs []int64, tp int, tableID int64, ifNotExists bool) (err error)

InsertExtendedStats inserts a record into mysql.stats_extended and update version in mysql.stats_meta.

func (*Handle) LastUpdateVersion 

func (h *Handle) LastUpdateVersion() uint64

LastUpdateVersion gets the last update version.

func (*Handle) Lease 

func (h *Handle) Lease() time.Duration

Lease returns the stats lease.

func (*Handle) LoadColumnStatsUsage 

func (h *Handle) LoadColumnStatsUsage(loc *time.Location) (map[model.TableItemID]colStatsTimeInfo, error)

LoadColumnStatsUsage loads column stats usage information from disk.

func (*Handle) LoadNeededHistograms 

func (h *Handle) LoadNeededHistograms() (err error)

LoadNeededHistograms will load histograms for those needed columns/indices.

func (*Handle) LoadStatsFromJSON 

func (h *Handle) LoadStatsFromJSON(is infoschema.InfoSchema, jsonTbl *JSONTable) error

LoadStatsFromJSON will load statistic from JSONTable, and save it to the storage.

func (*Handle) MarkExtendedStatsDeleted 

func (h *Handle) MarkExtendedStatsDeleted(statsName string, tableID int64, ifExists bool) (err error)

MarkExtendedStatsDeleted update the status of mysql.stats_extended to be `deleted` and the version of mysql.stats_meta.

func (*Handle) MergePartitionStats2GlobalStatsByTableID 

func (h *Handle) MergePartitionStats2GlobalStatsByTableID(sc sessionctx.Context,
	opts map[ast.AnalyzeOptionType]uint64, is infoschema.InfoSchema,
	physicalID int64, isIndex int, histIDs []int64,
	tablePartitionStats map[int64]*statistics.Table) (globalStats *GlobalStats, err error)

MergePartitionStats2GlobalStatsByTableID merge the partition-level stats to global-level stats based on the tableID.

func (*Handle) NewSessionIndexUsageCollector 

func (h *Handle) NewSessionIndexUsageCollector() *SessionIndexUsageCollector

NewSessionIndexUsageCollector will add a new SessionIndexUsageCollector into linked list headed by idxUsageListHead. idxUsageListHead always points to an empty SessionIndexUsageCollector as a sentinel node. So we let idxUsageListHead.next points to new item. It's helpful to sweepIdxUsageList.

func (*Handle) NewSessionStatsCollector 

func (h *Handle) NewSessionStatsCollector() *SessionStatsCollector

NewSessionStatsCollector allocates a stats collector for a session.

func (*Handle) RecordHistoricalStatsToStorage 

func (h *Handle) RecordHistoricalStatsToStorage(dbName string, tableInfo *model.TableInfo, physicalID int64, isPartition bool) (uint64, error)

RecordHistoricalStatsToStorage records the given table's stats data to mysql.stats_history

func (*Handle) RefreshVars 

func (h *Handle) RefreshVars() error

RefreshVars uses to pull PartitionPruneMethod vars from kv storage.

func (*Handle) ReloadExtendedStatistics 

func (h *Handle) ReloadExtendedStatistics() error

ReloadExtendedStatistics drops the cache for extended statistics and reload data from mysql.stats_extended.

func (*Handle) RemoveLockedTables 

func (h *Handle) RemoveLockedTables(tids []int64, pids []int64, tables []*ast.TableName) (string, error)

RemoveLockedTables remove tables from table locked array. - tids: table ids of which will be unlocked. - pids: partition ids of which will be unlocked. - tables: table names of which will be unlocked. Return the message of skipped tables and error.

func (*Handle) SaveExtendedStatsToStorage 

func (h *Handle) SaveExtendedStatsToStorage(tableID int64, extStats *statistics.ExtendedStatsColl, isLoad bool) (err error)

SaveExtendedStatsToStorage writes extended stats of a table into mysql.stats_extended.

func (*Handle) SaveMetaToStorage 

func (h *Handle) SaveMetaToStorage(tableID, count, modifyCount int64, source string) (err error)

SaveMetaToStorage will save stats_meta to storage.

func (*Handle) SaveStatsToStorage 

func (h *Handle) SaveStatsToStorage(tableID int64, count, modifyCount int64, isIndex int, hg *statistics.Histogram,
	cms *statistics.CMSketch, topN *statistics.TopN, statsVersion int, isAnalyzed int64, updateAnalyzeTime bool, source string) (err error)

SaveStatsToStorage saves the stats to storage. If count is negative, both count and modify count would not be used and not be written to the table. Unless, corresponding fields in the stats_meta table will be updated. TODO: refactor to reduce the number of parameters

func (*Handle) SaveTableStatsToStorage 

func (h *Handle) SaveTableStatsToStorage(results *statistics.AnalyzeResults, analyzeSnapshot bool, source string) (err error)

SaveTableStatsToStorage saves the stats of a table to storage.

func (*Handle) SendLoadRequests 

func (h *Handle) SendLoadRequests(sc *stmtctx.StatementContext, neededHistItems []model.TableItemID, timeout time.Duration) error

SendLoadRequests send neededColumns requests

func (*Handle) SetLease 

func (h *Handle) SetLease(lease time.Duration)

SetLease sets the stats lease.

func (*Handle) SetStatsCacheCapacity 

func (h *Handle) SetStatsCacheCapacity(c int64)

SetStatsCacheCapacity sets capacity

func (*Handle) StatsMetaCountAndModifyCount 

func (h *Handle) StatsMetaCountAndModifyCount(tableID int64) (count, modifyCount int64, err error)

StatsMetaCountAndModifyCount reads count and modify_count for the given table from mysql.stats_meta.

func (*Handle) SubLoadWorker 

func (h *Handle) SubLoadWorker(ctx sessionctx.Context, exit chan struct{}, exitWg *util.WaitGroupEnhancedWrapper)

SubLoadWorker loads hist data for each column

func (*Handle) SyncWaitStatsLoad 

func (*Handle) SyncWaitStatsLoad(sc *stmtctx.StatementContext) error

SyncWaitStatsLoad sync waits loading of neededColumns and return false if timeout

func (*Handle) TableStatsFromStorage 

func (h *Handle) TableStatsFromStorage(tableInfo *model.TableInfo, physicalID int64, loadAll bool, snapshot uint64) (_ *statistics.Table, err error)

TableStatsFromStorage loads table stats info from storage.

func (*Handle) Update 

func (h *Handle) Update(is infoschema.InfoSchema, opts ...cache.TableStatsOpt) error

Update reads stats meta from store and updates the stats map.

func (*Handle) UpdateSessionVar 

func (h *Handle) UpdateSessionVar() error

UpdateSessionVar updates the necessary session variables for the stats reader.

func (*Handle) UpdateStatsHealthyMetrics 

func (h *Handle) UpdateStatsHealthyMetrics()

UpdateStatsHealthyMetrics updates stats healthy distribution metrics according to stats cache.

type IndexUsageInformation 

type IndexUsageInformation struct {
	LastUsedAt   string
	QueryCount   int64
	RowsSelected int64
}

IndexUsageInformation is the data struct to store index usage information.

type JSONTable 

type JSONTable struct {
	Columns           map[string]*jsonColumn `json:"columns"`
	Indices           map[string]*jsonColumn `json:"indices"`
	Partitions        map[string]*JSONTable  `json:"partitions"`
	DatabaseName      string                 `json:"database_name"`
	TableName         string                 `json:"table_name"`
	ExtStats          []*jsonExtendedStats   `json:"ext_stats"`
	Count             int64                  `json:"count"`
	ModifyCount       int64                  `json:"modify_count"`
	Version           uint64                 `json:"version"`
	IsHistoricalStats bool                   `json:"is_historical_stats"`
}

JSONTable is used for dumping statistics.

func BlocksToJSONTable 

func BlocksToJSONTable(blocks [][]byte) (*JSONTable, error)

BlocksToJSONTable convert gzip-compressed blocks to JSONTable

func GenJSONTableFromStats 

func GenJSONTableFromStats(dbName string, tableInfo *model.TableInfo, tbl *statistics.Table) (*JSONTable, error)

GenJSONTableFromStats generate jsonTable from tableInfo and stats

type NeededItemTask 

type NeededItemTask struct {
	ToTimeout   time.Time
	ResultCh    chan stmtctx.StatsLoadResult
	TableItemID model.TableItemID
}

NeededItemTask represents one needed column/indices with expire time.

type SessionIndexUsageCollector 

type SessionIndexUsageCollector struct {
	sync.Mutex
	// contains filtered or unexported fields
}

SessionIndexUsageCollector is a list item that holds the index usage mapper. If you want to write or read mapper, you must lock it.

func (*SessionIndexUsageCollector) Delete 

func (s *SessionIndexUsageCollector) Delete()

Delete will set s.deleted to true which means it can be deleted from linked list.

func (*SessionIndexUsageCollector) Update 

func (s *SessionIndexUsageCollector) Update(tableID int64, indexID int64, value *IndexUsageInformation)

Update updates the mapper in SessionIndexUsageCollector.

type SessionStatsCollector 

type SessionStatsCollector struct {
	sync.Mutex
	// contains filtered or unexported fields
}

SessionStatsCollector is a list item that holds the delta mapper. If you want to write or read mapper, you must lock it.

func NewSessionStatsCollector 

func NewSessionStatsCollector() *SessionStatsCollector

NewSessionStatsCollector initializes a new SessionStatsCollector.

func (*SessionStatsCollector) ClearForTest 

func (s *SessionStatsCollector) ClearForTest()

ClearForTest clears the mapper for test.

func (*SessionStatsCollector) Delete 

func (s *SessionStatsCollector) Delete()

Delete only sets the deleted flag true, it will be deleted from list when DumpStatsDeltaToKV is called.

func (*SessionStatsCollector) Update 

func (s *SessionStatsCollector) Update(id int64, delta int64, count int64, colSize *map[int64]int64)

Update will updates the delta and count for one table id.

func (*SessionStatsCollector) UpdateColStatsUsage 

func (s *SessionStatsCollector) UpdateColStatsUsage(colMap colStatsUsageMap)

UpdateColStatsUsage updates the last time when the column stats are used(needed).

type StatsLoad 

type StatsLoad struct {
	NeededItemsCh  chan *NeededItemTask
	TimeoutItemsCh chan *NeededItemTask
	WorkingColMap  map[model.TableItemID][]chan stmtctx.StatsLoadResult
	SubCtxs        []sessionctx.Context
	sync.Mutex
}

StatsLoad is used to load stats concurrently

type StatsReaderContext 

type StatsReaderContext struct {
	// contains filtered or unexported fields
}

StatsReaderContext exported for testing

Source Files

View all Source files

Directories

Path Synopsis
internal
internal/lfu
internal/mapcache
internal/metrics
internal/testutil

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.