queue

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 24, 2025 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package queue implements the data transmission queue.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ClearChunkedSyncFailureInjector added in v0.9.0

func ClearChunkedSyncFailureInjector()

ClearChunkedSyncFailureInjector removes the registered failure injector.

func RegisterChunkedSyncFailureInjector added in v0.9.0

func RegisterChunkedSyncFailureInjector(injector ChunkedSyncFailureInjector)

RegisterChunkedSyncFailureInjector registers a failure injector for testing.

Types

type BatchPublisher added in v0.5.0

type BatchPublisher interface {
	bus.Publisher
	Close() (map[string]*common.Error, error)
}

BatchPublisher is the interface for publishing data in batch.

type ChunkedSyncClient added in v0.9.0

type ChunkedSyncClient interface {
	// SyncStreamingParts sends streaming parts using chunked transfer.
	SyncStreamingParts(ctx context.Context, parts []StreamingPartData) (*SyncResult, error)
	// Close releases resources associated with the client.
	Close() error
}

ChunkedSyncClient provides methods for chunked data synchronization with streaming support.

type ChunkedSyncFailureInjector added in v0.9.0

type ChunkedSyncFailureInjector interface {
	// BeforeSync returns whether the current SyncStreamingParts invocation should
	// short-circuit. The returned slice provides the failed parts that should be
	// reported back to the caller when a failure is injected.
	BeforeSync(parts []StreamingPartData) (bool, []FailedPart, error)
}

ChunkedSyncFailureInjector allows tests to deterministically inject failures into chunked sync operations. It is only intended for test code.

func GetChunkedSyncFailureInjector added in v0.9.0

func GetChunkedSyncFailureInjector() ChunkedSyncFailureInjector

GetChunkedSyncFailureInjector returns the currently registered failure injector.

type ChunkedSyncHandler added in v0.9.0

type ChunkedSyncHandler interface {
	// HandleFileChunk processes incoming file chunks as they arrive.
	// This method is called for each chunk of file data to enable streaming processing.
	HandleFileChunk(ctx *ChunkedSyncPartContext, chunk []byte) error

	// CreatePartHandler creates a new part handler for the given context.
	CreatePartHandler(ctx *ChunkedSyncPartContext) (PartHandler, error)
}

ChunkedSyncHandler handles incoming chunked sync requests on the server side.

type ChunkedSyncPartContext added in v0.9.0

type ChunkedSyncPartContext struct {
	Handler               PartHandler
	Group                 string
	FileName              string
	PartType              string
	ID                    uint64
	CompressedSizeBytes   uint64
	UncompressedSizeBytes uint64
	TotalCount            uint64
	BlocksCount           uint64
	MinTimestamp          int64
	MaxTimestamp          int64
	MinKey                int64
	MaxKey                int64
	ShardID               uint32
}

ChunkedSyncPartContext represents the context for a chunked sync operation.

func (*ChunkedSyncPartContext) Close added in v0.9.0

func (c *ChunkedSyncPartContext) Close() error

Close releases resources associated with the context.

type Client added in v0.5.0

type Client interface {
	run.Unit
	bus.Publisher
	bus.Broadcaster
	NewBatchPublisher(timeout time.Duration) BatchPublisher
	NewChunkedSyncClient(node string, chunkSize uint32) (ChunkedSyncClient, error)
	Register(bus.Topic, schema.EventHandler)
	OnAddOrUpdate(md schema.Metadata)
	GracefulStop()
	HealthyNodes() []string
}

Client is the interface for publishing data to the queue.

type FailedPart added in v0.9.0

type FailedPart struct {
	PartID string
	Error  string
}

FailedPart contains information about a part that failed to sync.

type FileData added in v0.9.0

type FileData struct {
	FileName string
	Data     []byte
}

FileData represents file data with name and content.

type FileInfo added in v0.9.0

type FileInfo struct {
	Reader fs.SeqReader
	Name   string
}

FileInfo represents information about an individual file within a part.

type PartHandler added in v0.9.0

type PartHandler interface {
	NewPartType(ctx *ChunkedSyncPartContext) error
	FinishSync() error
	Close() error
}

PartHandler handles individual parts during sync operations.

type Queue

type Queue interface {
	Client
	Server
	run.Service
}

Queue builds a data transmission tunnel between subscribers and publishers.

func Local added in v0.5.0

func Local() Queue

Local return a new local Queue.

type Server added in v0.5.0

type Server interface {
	run.Unit
	bus.Subscriber
	RegisterChunkedSyncHandler(topic bus.Topic, handler ChunkedSyncHandler)
	GetPort() *uint32
}

Server is the interface for receiving data from the queue.

type StreamingPartData added in v0.9.0

type StreamingPartData struct {
	Group                 string
	Topic                 string
	PartType              string
	Files                 []FileInfo
	ID                    uint64
	CompressedSizeBytes   uint64
	UncompressedSizeBytes uint64
	TotalCount            uint64
	BlocksCount           uint64
	MinTimestamp          int64
	MaxTimestamp          int64
	MinKey                int64
	MaxKey                int64
	ShardID               uint32
}

StreamingPartData represents streaming part data for syncing.

type SyncMetadata added in v0.9.0

type SyncMetadata = clusterv1.SyncMetadata

SyncMetadata is an alias for clusterv1.SyncMetadata.

type SyncResult added in v0.9.0

type SyncResult struct {
	SessionID   string
	FailedParts []FailedPart
	TotalBytes  uint64
	DurationMs  int64
	ChunksCount uint32
	PartsCount  uint32
	Success     bool
}

SyncResult represents the result of a sync operation.

Directories

Path Synopsis
Package pub implements the queue client.
Package pub implements the queue client.
Package sub implements the queue server.
Package sub implements the queue server.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL