etl

package

v1.0.0 Latest Latest Go to latest Published: Mar 25, 2026 License: GPL-3.0 Imports: 2 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/hsqbyte/protospec

Links

Open Source Insights

Documentation ¶

Overview ¶

Package etl provides protocol data ETL pipeline.

Index ¶

type Pipeline
- func NewPipeline(name string) *Pipeline
type SinkType
type SourceType
type TransformOp

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Pipeline ¶

type Pipeline struct {
	Name       string        `json:"name"`
	Source     SourceType    `json:"source"`
	SourceURI  string        `json:"source_uri"`
	Transforms []TransformOp `json:"transforms"`
	Sink       SinkType      `json:"sink"`
	SinkURI    string        `json:"sink_uri"`
}

Pipeline represents an ETL pipeline configuration.

func NewPipeline ¶

func NewPipeline(name string) *Pipeline

NewPipeline creates a new ETL pipeline.

func (*Pipeline) AddTransform ¶

func (p *Pipeline) AddTransform(op TransformOp) *Pipeline

AddTransform adds a transform step.

func (*Pipeline) Describe ¶

func (p *Pipeline) Describe() string

Describe returns a human-readable pipeline description.

func (*Pipeline) Run ¶

func (p *Pipeline) Run() error

Run executes the ETL pipeline (stub).

func (*Pipeline) SetSink ¶

func (p *Pipeline) SetSink(t SinkType, uri string) *Pipeline

SetSink sets the pipeline sink.

func (*Pipeline) SetSource ¶

func (p *Pipeline) SetSource(t SourceType, uri string) *Pipeline

SetSource sets the pipeline source.

type SinkType ¶

type SinkType string

SinkType represents an ETL data sink type.

const (
	SinkFile          SinkType = "file"
	SinkDatabase      SinkType = "database"
	SinkElasticsearch SinkType = "elasticsearch"
	SinkS3            SinkType = "s3"
)

type SourceType ¶

type SourceType string

SourceType represents an ETL data source type.

const (
	SourcePCAP      SourceType = "pcap"
	SourceInterface SourceType = "interface"
	SourceKafka     SourceType = "kafka"
	SourceFile      SourceType = "file"
)

type TransformOp ¶

type TransformOp struct {
	Type   string `json:"type"` // decode, filter, aggregate, format
	Config string `json:"config"`
}

TransformOp represents a transform operation.

Source Files ¶

View all Source files

etl.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL