Documentation
¶
Index ¶
- func BuildJetStreamWatermarkProgressors(ctx context.Context, vertexInstance *v1alpha1.VertexInstance) (fetchWatermark fetch.Fetcher, publishWatermark map[string]publish.Publisher)
- func GetFetchKeyspace(v *dfv1.Vertex) string
- func GetPublishKeySpace(v *dfv1.Vertex) string
- type FetchWMWatchers
- type GenericFetch
- type GenericPublish
- type NoOpWMProgressor
- func (n NoOpWMProgressor) GetHeadWatermark() processor.Watermark
- func (n NoOpWMProgressor) GetLatestWatermark() processor.Watermark
- func (n NoOpWMProgressor) GetWatermark(_ isb.Offset) processor.Watermark
- func (n NoOpWMProgressor) PublishWatermark(_ processor.Watermark, _ isb.Offset)
- func (n NoOpWMProgressor) StopPublisher()
- type Progressor
- type PublishWMStores
- func BuildJetStreamWatermarkProgressorsForSource(ctx context.Context, vertexInstance *v1alpha1.VertexInstance) (fetchWatermark fetch.Fetcher, publishWatermark map[string]publish.Publisher, ...)
- func BuildPublishWMStores(hbStore store.WatermarkKVStorer, otStore store.WatermarkKVStorer) PublishWMStores
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildJetStreamWatermarkProgressors ¶
func BuildJetStreamWatermarkProgressors(ctx context.Context, vertexInstance *v1alpha1.VertexInstance) (fetchWatermark fetch.Fetcher, publishWatermark map[string]publish.Publisher)
BuildJetStreamWatermarkProgressors is used to populate fetchWatermark, and a map of publishWatermark with edge name as the key. These are used as watermark progressors in the pipeline, and is attached to each edge of the vertex. Fetcher has one-to-one relationship , whereas we have multiple publishers as the vertex can read only from one edge, and it can write to many. The function is used only when watermarking is enabled on the pipeline.
func GetFetchKeyspace ¶
GetFetchKeyspace gets the fetch keyspace name fromEdge the vertex.
func GetPublishKeySpace ¶
GetPublishKeySpace gets the publish keyspace name fromEdge the vertex
Types ¶
type FetchWMWatchers ¶
type FetchWMWatchers struct {
HBWatch store.WatermarkKVWatcher
OTWatch store.WatermarkKVWatcher
}
FetchWMWatchers has the watcher information required for fetching watermarks.
func BuildFetchWMWatchers ¶
func BuildFetchWMWatchers(hbWatch store.WatermarkKVWatcher, otWatch store.WatermarkKVWatcher) FetchWMWatchers
BuildFetchWMWatchers builds the FetchWMWatchers
type GenericFetch ¶
type GenericFetch struct {
// contains filtered or unexported fields
}
GenericFetch is a generic fetcher which can be used for most use cases.
func NewGenericFetch ¶
func NewGenericFetch(ctx context.Context, vertexName string, fetchWM FetchWMWatchers) *GenericFetch
NewGenericFetch returns GenericFetch. vertexName is the vertex currently processing. fetchWM is a struct for retrieving both the heartbeat and the offset watermark timeline (Vn-1 vertex).
func (*GenericFetch) GetHeadWatermark ¶ added in v0.5.4
func (g *GenericFetch) GetHeadWatermark() processor.Watermark
GetHeadWatermark returns the head watermark based on the head offset.
func (*GenericFetch) GetWatermark ¶
func (g *GenericFetch) GetWatermark(offset isb.Offset) processor.Watermark
GetWatermark returns the watermark for the offset.
type GenericPublish ¶
type GenericPublish struct {
// contains filtered or unexported fields
}
GenericPublish is a generic publisher which will work for most cases.
func NewGenericPublish ¶
func NewGenericPublish(ctx context.Context, processorName string, publishWM PublishWMStores) *GenericPublish
NewGenericPublish returns GenericPublish. processorName is the unique processor (pod) that is running on this vertex. publishKeyspace is obsolete, and will be removed in subsequent iterations. publishWM is a struct for storing both the heartbeat and the offset watermark timeline stores for the Vn vertex.
func (*GenericPublish) GetLatestWatermark ¶
func (g *GenericPublish) GetLatestWatermark() processor.Watermark
GetLatestWatermark gets the latest watermakr for the generic publisher.
func (*GenericPublish) PublishWatermark ¶
func (g *GenericPublish) PublishWatermark(watermark processor.Watermark, offset isb.Offset)
PublishWatermark publishes for the generic publisher.
func (*GenericPublish) StopPublisher ¶
func (g *GenericPublish) StopPublisher()
StopPublisher stops the generic publisher.
type NoOpWMProgressor ¶
type NoOpWMProgressor struct {
}
NoOpWMProgressor is a no-op watermark progressor. As the name suggests, it does not do anything, no watermark is progressed.
func NewNoOpWMProgressor ¶
func NewNoOpWMProgressor() *NoOpWMProgressor
NewNoOpWMProgressor returns NoOpWMProgressor.
func (NoOpWMProgressor) GetHeadWatermark ¶ added in v0.5.4
func (n NoOpWMProgressor) GetHeadWatermark() processor.Watermark
GetHeadWatermark returns the default head watermark.
func (NoOpWMProgressor) GetLatestWatermark ¶
func (n NoOpWMProgressor) GetLatestWatermark() processor.Watermark
GetLatestWatermark returns the default watermark as the latest watermark.
func (NoOpWMProgressor) GetWatermark ¶
func (n NoOpWMProgressor) GetWatermark(_ isb.Offset) processor.Watermark
GetWatermark returns the default watermark.
func (NoOpWMProgressor) PublishWatermark ¶
func (n NoOpWMProgressor) PublishWatermark(_ processor.Watermark, _ isb.Offset)
PublishWatermark does a no-op publish.
func (NoOpWMProgressor) StopPublisher ¶
func (n NoOpWMProgressor) StopPublisher()
StopPublisher stops the no-op publisher.
type Progressor ¶
Progressor interface defines how the watermark can be progressed. FIXME: delete this
type PublishWMStores ¶
type PublishWMStores struct {
HBStore store.WatermarkKVStorer
OTStore store.WatermarkKVStorer
}
PublishWMStores stores the store information for publishing the watermark.
func BuildJetStreamWatermarkProgressorsForSource ¶
func BuildJetStreamWatermarkProgressorsForSource(ctx context.Context, vertexInstance *v1alpha1.VertexInstance) (fetchWatermark fetch.Fetcher, publishWatermark map[string]publish.Publisher, publishWM PublishWMStores)
BuildJetStreamWatermarkProgressorsForSource is an extension of BuildJetStreamWatermarkProgressors to also return the publish stores. This is for letting source implement as many publishers that it requires to progress the watermark monotonically for each individual processing entity. Eg, watermark progresses independently and monotonically for each partition in a Kafka topic.
func BuildPublishWMStores ¶
func BuildPublishWMStores(hbStore store.WatermarkKVStorer, otStore store.WatermarkKVStorer) PublishWMStores
BuildPublishWMStores builds the PublishWMStores.
Source Files