Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewDataMover ¶
func NewDataMover(sourceParallelism, destinationParallelism int, source Source, destination Destination, sourceErrorHandler, destErrorHandler func(error) bool) (*dataMover, error)
NewDataMover is the constructor for a data mover. Use this to instantiate a new configured data mover.
Types ¶
type Destination ¶
type Destination interface {
// Initialize can be used to initialize a source connector. I.E. db connections, http clients, etc.
Initialize() error
// Persist will be called for each batch of data returned by the source connector. This is where the logic should be
// implemented to persist data from the source connector to the destination.
Persist(data []map[string]interface{}) error
}
Destination is the interface that a destination connector must implement
type DestinationHandler ¶
type DestinationHandler struct {
// contains filtered or unexported fields
}
DestinationHandler is the data mover's destination dispatcher job handler
func (DestinationHandler) HandleJob ¶
func (h DestinationHandler) HandleJob(job parallelism.Job)
HandleJob is the data mover's destination dispatcher job handler implementation. This calls the configured destination connector's Persist() method with the data passed in on the job, from the source connector.
type DestinationJob ¶
type DestinationJob struct {
Data []map[string]interface{}
}
DestinationJob is the destination dispatcher's job struct, it has a field for the data from the source connector.
func (DestinationJob) GetData ¶
func (j DestinationJob) GetData() interface{}
GetData is the destination dispatcher's job GetData implementation, it returns the Data from the job.
type Source ¶
type Source interface {
// Initialize can be used to initialize a source connector. I.E. db connections, http clients, etc.
Initialize() error
// GetData will be called until it returns an empty array. When sourceParallelism is greater than 1, GetData will
// be called in parallel. The mover makes no attempt at tracking what data has been fetched already, nor does it
// make any attempts at handling concurrency. That sort of tracking is 100% up to the implementation.
GetData() ([]map[string]interface{}, error) // GetData
}
Source is the interface that a source connector must implement.
type SourceHandler ¶
type SourceHandler struct {
// contains filtered or unexported fields
}
SourceHandler is the data mover's source dispatcher job handler
func (SourceHandler) HandleJob ¶
func (h SourceHandler) HandleJob(job parallelism.Job)
HandleJob is the data mover's source dispatcher job handler implementation. This calls the configured source connector's GetData() function and submits the resulting data to the data mover's destination dispatcher to be handled by the destination connector
Source Files