Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
View Source
var ExtractedChan = make(chan map[string]interface{}) // Unbuffered channel for extracted records; processing goroutines will read from this channel
View Source
var ProcessingWG sync.WaitGroup // WaitGroup to track processing goroutines
View Source
var ResultChan = make(chan map[string]interface{}, 100) // Buffered channel to prevent blocking on writes when processing is slower than extraction
View Source
var TransformMetrics = &TransformationMetrics{}
Functions ¶
func ExtractRecords ¶ added in v0.5.0
func ExtractRecords(sourceFunc func(*models.StreamConfig) error)
ExtractRecords begins streaming records from source (sending to ExtractedChan) and start goroutines to extract records (sending to ResultChan)
Types ¶
type TransformationMetrics ¶ added in v0.7.1
type TransformationMetrics struct {
Processed uint64 `json:"processed"`
Skipped uint64 `json:"skipped"`
SkippedBookmark uint64 `json:"skipped_bookmark"` // skipped by bookmark
// contains filtered or unexported fields
}
TransformationMetrics tracks record transformation statistics
Source Files
Click to show internal directories.
Click to hide internal directories.