Documentation
¶
Overview ¶
Package queue implements the data transmission queue.
Index ¶
- Variables
- func ClearChunkedSyncFailureInjector()
- func RegisterChunkedSyncFailureInjector(injector ChunkedSyncFailureInjector)
- type BatchPublisher
- type ChunkedSyncClient
- type ChunkedSyncFailureInjector
- type ChunkedSyncHandler
- type ChunkedSyncPartContext
- type Client
- type FailedPart
- type FileData
- type FileInfo
- type PartHandler
- type Queue
- type Server
- type StreamingPartData
- type SyncMetadata
- type SyncResult
Constants ¶
This section is empty.
Variables ¶
var ErrNotImplemented = errors.New("not implemented")
ErrNotImplemented is returned by Client implementations that have no peers to dial — notably the standalone local pipeline. Callers (e.g. the cluster barrier fan-out) treat this sentinel as "no peer to probe; skip" rather than as a hard failure, so the standalone path degrades to in-process self-probing without a special-cased branch in the caller.
var ErrServerBusy = errors.New("receiver under memory pressure")
ErrServerBusy signals the receiver is under memory pressure and the sender should back off and retry the whole part.
Functions ¶
func ClearChunkedSyncFailureInjector ¶ added in v0.9.0
func ClearChunkedSyncFailureInjector()
ClearChunkedSyncFailureInjector removes the registered failure injector.
func RegisterChunkedSyncFailureInjector ¶ added in v0.9.0
func RegisterChunkedSyncFailureInjector(injector ChunkedSyncFailureInjector)
RegisterChunkedSyncFailureInjector registers a failure injector for testing.
Types ¶
type BatchPublisher ¶ added in v0.5.0
BatchPublisher is the interface for publishing data in batch.
type ChunkedSyncClient ¶ added in v0.9.0
type ChunkedSyncClient interface {
// SyncStreamingParts sends streaming parts using chunked transfer.
SyncStreamingParts(ctx context.Context, parts []StreamingPartData) (*SyncResult, error)
// Close releases resources associated with the client.
Close() error
}
ChunkedSyncClient provides methods for chunked data synchronization with streaming support.
type ChunkedSyncFailureInjector ¶ added in v0.9.0
type ChunkedSyncFailureInjector interface {
// BeforeSync returns whether the current SyncStreamingParts invocation should
// short-circuit. The returned slice provides the failed parts that should be
// reported back to the caller when a failure is injected.
BeforeSync(parts []StreamingPartData) (bool, []FailedPart, error)
}
ChunkedSyncFailureInjector allows tests to deterministically inject failures into chunked sync operations. It is only intended for test code.
func GetChunkedSyncFailureInjector ¶ added in v0.9.0
func GetChunkedSyncFailureInjector() ChunkedSyncFailureInjector
GetChunkedSyncFailureInjector returns the currently registered failure injector.
type ChunkedSyncHandler ¶ added in v0.9.0
type ChunkedSyncHandler interface {
// HandleFileChunk processes incoming file chunks as they arrive.
// This method is called for each chunk of file data to enable streaming processing.
HandleFileChunk(ctx *ChunkedSyncPartContext, chunk []byte) error
// CreatePartHandler creates a new part handler for the given context.
CreatePartHandler(ctx *ChunkedSyncPartContext) (PartHandler, error)
}
ChunkedSyncHandler handles incoming chunked sync requests on the server side.
type ChunkedSyncPartContext ¶ added in v0.9.0
type ChunkedSyncPartContext struct {
Handler PartHandler
// Context carries the gRPC stream context for receive-side cancellation
// (memory wait upper bound / disconnect).
Context context.Context
Group string
FileName string
PartType string
ID uint64
CompressedSizeBytes uint64
UncompressedSizeBytes uint64
TotalCount uint64
BlocksCount uint64
MinTimestamp int64
MaxTimestamp int64
MinKey int64
MaxKey int64
ShardID uint32
}
ChunkedSyncPartContext represents the context for a chunked sync operation.
func (*ChunkedSyncPartContext) Close ¶ added in v0.9.0
func (c *ChunkedSyncPartContext) Close() error
Close releases resources associated with the context.
func (*ChunkedSyncPartContext) RetrieveContext ¶ added in v0.11.0
func (c *ChunkedSyncPartContext) RetrieveContext() context.Context
RetrieveContext returns the part's stream context, or context.Background() if none was set. Used by receive handlers to bound memory-pressure waits.
type Client ¶ added in v0.5.0
type Client interface {
run.Unit
bus.Publisher
bus.Broadcaster
route.TableProvider
// NewBatchPublisher creates a batch publisher whose timeout starts when Close seals the batch.
NewBatchPublisher(timeout time.Duration) BatchPublisher
NewChunkedSyncClient(node string, chunkSize uint32) (ChunkedSyncClient, error)
// SetSelfNode stamps the publisher's own node identity onto the wire
// (SenderNode / SenderRole / SenderTier on every SendRequest) and onto
// the banyandb_lifecycle_migration_* metric labels. Pass "" for any
// dimension the caller doesn't know (e.g. the lifecycle has no Role
// enum entry; the tier is untracked when the co-located data node's
// tier isn't known at boot time).
SetSelfNode(name, role, tier string)
// NewNodeSchemaStatusClient returns a client for the cluster.v1
// NodeSchemaStatusService against the named node, sharing the underlying
// *grpc.ClientConn from this queue client's existing connection pool.
// The standalone local pipeline returns ErrNotImplemented so callers can
// treat it as "no peer to probe" without a type-switch.
NewNodeSchemaStatusClient(node string) (clusterv1.NodeSchemaStatusServiceClient, error)
Register(bus.Topic, schema.EventHandler)
OnAddOrUpdate(md schema.Metadata)
GracefulStop()
HealthyNodes() []string
}
Client is the interface for publishing data to the queue.
type FailedPart ¶ added in v0.9.0
FailedPart contains information about a part that failed to sync.
type FileInfo ¶ added in v0.9.0
FileInfo represents information about an individual file within a part.
type PartHandler ¶ added in v0.9.0
type PartHandler interface {
NewPartType(ctx *ChunkedSyncPartContext) error
FinishSync() error
Close() error
}
PartHandler handles individual parts during sync operations.
type Server ¶ added in v0.5.0
type Server interface {
run.Unit
bus.Subscriber
RegisterChunkedSyncHandler(topic bus.Topic, handler ChunkedSyncHandler)
GetPort() *uint32
SetRouteProviders(providers map[string]route.TableProvider)
// SetMetadataRepo enables fodc.v1.GroupLifecycleService — liaison-only
// by design; data-node processes must not call it.
SetMetadataRepo(repo metadata.Repo)
// SetNodeSchemaStatusRepo enables cluster.v1.NodeSchemaStatusService —
// per-node by design (reports the local schema cache); liaison and
// data-node processes both call it.
SetNodeSchemaStatusRepo(svc metadata.Service)
}
Server is the interface for receiving data from the queue.
type StreamingPartData ¶ added in v0.9.0
type StreamingPartData struct {
Group string
Topic string
PartType string
Files []FileInfo
ID uint64
CompressedSizeBytes uint64
UncompressedSizeBytes uint64
TotalCount uint64
BlocksCount uint64
MinTimestamp int64
MaxTimestamp int64
MinKey int64
MaxKey int64
ShardID uint32
}
StreamingPartData represents streaming part data for syncing.
type SyncMetadata ¶ added in v0.9.0
type SyncMetadata = clusterv1.SyncMetadata
SyncMetadata is an alias for clusterv1.SyncMetadata.
type SyncResult ¶ added in v0.9.0
type SyncResult struct {
SessionID string
FailedParts []FailedPart
TotalBytes uint64
DurationMs int64
ChunksCount uint32
PartsCount uint32
Success bool
}
SyncResult represents the result of a sync operation.