Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var SourceTablesRegistry = map[string][]*SourceTable{}/* 117 elements not displayed */
Functions ¶
func IsStreamingAsset ¶ added in v0.11.681
IsStreamingAsset reports whether an ingestr asset runs as a continuous, never-terminating stream rather than a one-shot batch load. This is true for a CDC asset in stream mode (cdc: true + stream: true) and for a message-broker source that opts into continuous ingestion with the stream parameter.
It is the single classifier the run command and the filter chain use to route streaming assets away from the batch DAG.
Types ¶
type BasicOperator ¶
type BasicOperator struct {
// contains filtered or unexported fields
}
func NewBasicOperator ¶
func NewBasicOperator(conn config.ConnectionGetter, j jinja.RendererInterface) (*BasicOperator, error)
func (*BasicOperator) Run ¶
func (o *BasicOperator) Run(ctx context.Context, ti scheduler.TaskInstance) error
type IngestrCheckOperator ¶
type IngestrCheckOperator struct {
// contains filtered or unexported fields
}
func NewColumnCheckOperator ¶
func NewColumnCheckOperator(configs *map[pipeline.AssetType]executor.Config) *IngestrCheckOperator
func (IngestrCheckOperator) Run ¶
func (i IngestrCheckOperator) Run(ctx context.Context, ti scheduler.TaskInstance) error
type IngestrCustomCheckOperator ¶
type IngestrCustomCheckOperator struct {
// contains filtered or unexported fields
}
func NewCustomCheckOperator ¶
func NewCustomCheckOperator(configs *map[pipeline.AssetType]executor.Config) *IngestrCustomCheckOperator
func (IngestrCustomCheckOperator) Run ¶
func (i IngestrCustomCheckOperator) Run(ctx context.Context, ti scheduler.TaskInstance) error
type SeedOperator ¶ added in v0.11.128
type SeedOperator struct {
// contains filtered or unexported fields
}
func NewSeedOperator ¶ added in v0.11.128
func NewSeedOperator(conn config.ConnectionGetter, j jinja.RendererInterface) (*SeedOperator, error)
func (*SeedOperator) Run ¶ added in v0.11.128
func (o *SeedOperator) Run(ctx context.Context, ti scheduler.TaskInstance) error
type Source ¶ added in v0.11.421
type Source struct {
Name string `json:"name"`
Tables []*SourceTable `json:"tables"`
}
Source represents an ingestr source with its available tables.
func GetAllSources ¶ added in v0.11.421
func GetAllSources() []*Source
GetAllSources returns all available ingestr sources and their tables. Sources are sorted by name for consistent output.
func GetSourceTables ¶ added in v0.11.421
GetSourceTables returns the available tables for a specific ingestr source.
type SourceTable ¶ added in v0.11.421
type SourceTable struct {
Name string `json:"name"`
PrimaryKey string `json:"primary_key,omitempty"`
IncKey string `json:"incremental_key,omitempty"`
IncStrategy string `json:"incremental_strategy,omitempty"`
}
SourceTable represents a table available from an ingestr source with its metadata.
Source Files