source

package
v0.1.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2024 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GenerateJSONFile 

func GenerateJSONFile(columns []string, data [][]interface{}) (string, int, error)

Types

type DatabendIngesterStatsData added in v0.1.3 

type DatabendIngesterStatsData struct {
	BytesPerSecond float64
	RowsPerSecondd float64
}

type DatabendSourceStatsRecorder added in v0.1.3 

type DatabendSourceStatsRecorder struct {
	// contains filtered or unexported fields
}

func NewDatabendIntesterStatsRecorder added in v0.1.3 

func NewDatabendIntesterStatsRecorder() *DatabendSourceStatsRecorder

func (*DatabendSourceStatsRecorder) RecordMetric added in v0.1.3 

func (stats *DatabendSourceStatsRecorder) RecordMetric(rows int)

func (*DatabendSourceStatsRecorder) Stats added in v0.1.3 

func (stats *DatabendSourceStatsRecorder) Stats(statsWindow time.Duration) DatabendIngesterStatsData

type Source 

type Source struct {
	// contains filtered or unexported fields
}

func NewSource 

func NewSource(cfg *config.Config) (*Source, error)

func (*Source) AdjustBatchSizeAccordingToSourceDbTable added in v0.1.7 

func (s *Source) AdjustBatchSizeAccordingToSourceDbTable() int64

AdjustBatchSizeAccordingToSourceDbTable has a concept called s, s = (maxKey - minKey) / sourceTableRowCount if s == 1 it means the data is uniform in the table, if s is much bigger than 1, it means the data is not uniform in the table

func (*Source) DeleteAfterSync 

func (s *Source) DeleteAfterSync() error

func (*Source) GetAllSourceReadRowsCount added in v0.1.5 

func (s *Source) GetAllSourceReadRowsCount() (int, error)

func (*Source) GetDatabasesAccordingToSourceDbRegex added in v0.1.5 

func (s *Source) GetDatabasesAccordingToSourceDbRegex(sourceDatabasePattern string) ([]string, error)

func (*Source) GetDbTablesAccordingToSourceDbTables added in v0.1.5 

func (s *Source) GetDbTablesAccordingToSourceDbTables() (map[string][]string, error)

func (*Source) GetMinMaxSplitKey added in v0.0.6 

func (s *Source) GetMinMaxSplitKey() (int64, int64, error)

func (*Source) GetMinMaxTimeSplitKey added in v0.0.6 

func (s *Source) GetMinMaxTimeSplitKey() (string, string, error)

func (*Source) GetSourceReadRowsCount 

func (s *Source) GetSourceReadRowsCount(table, db string) (int, error)

func (*Source) GetTablesAccordingToSourceTableRegex added in v0.1.5 

func (s *Source) GetTablesAccordingToSourceTableRegex(sourceTablePattern string, databases []string) (map[string][]string, error)

func (*Source) QueryTableData 

func (s *Source) QueryTableData(threadNum int, conditionSql string) ([][]interface{}, []string, error)

func (*Source) SlimCondition added in v0.0.4 

func (s *Source) SlimCondition(minSplitKey, maxSplitKey int64) [][]int64

func (*Source) SplitCondition 

func (s *Source) SplitCondition(minSplitKey, maxSplitKey int64) []string

func (*Source) SplitConditionAccordingMaxGoRoutine added in v0.0.4 

func (s *Source) SplitConditionAccordingMaxGoRoutine(minSplitKey, maxSplitKey, allMax int64) <-chan string

func (*Source) SplitConditionAccordingToTimeSplitKey added in v0.0.6 

func (s *Source) SplitConditionAccordingToTimeSplitKey(minTimeSplitKey, maxTimeSplitKey string) ([]string, error)

func (*Source) SplitTimeConditionsByMaxThread added in v0.0.6 

func (s *Source) SplitTimeConditionsByMaxThread(conditions []string, maxThread int) [][]string

type Sourcer 

type Sourcer interface {
	QueryTableData(threadNum int, conditionSql string) ([][]interface{}, []string, error)
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.