managedwriter

package
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2021 License: Apache-2.0 Imports: 24 Imported by: 0

README

BQ Storage: ManagedWriter

The code in this package is a slightly modified fork from https://github.com/googleapis/google-cloud-go/tree/a2af4de215a42848368ec3081263d34782032caa/bigquery/storage/managedwriter.

NOTE: eventually we want to phase out our own managed writer fork, here is what we definitely require from that package in order to do so:

  • be able to define a custom Retryer for all its GRPC purposes, currently it is not possible for any purpose at all;
  • receive some kind of commitment from them that they do seriously consider, making this a production-ready part of the bigquery storage API;
  • make the stats/diagnostic code opt-in;

Documentation

Index

Constants

View Source 
const NoStreamOffset int64 = -1

NoStreamOffset is a sentinel value for signalling we're not tracking stream offset (e.g. a default stream which allows simultaneous append streams).

Variables

This section is empty.

Functions

func StreamTypeToEnum 

func StreamTypeToEnum(t StreamType) storagepb.WriteStream_Type

func TableParentFromStreamName 

func TableParentFromStreamName(streamName string) string

TableParentFromStreamName is a utility function for extracting the parent table prefix from a stream name. When an invalid stream ID is passed, this simply returns the original stream name.

Types

type AppendResult 

type AppendResult struct {
	// contains filtered or unexported fields
}

AppendResult tracks the status of a batch of data rows.

func (*AppendResult) GetResult 

func (ar *AppendResult) GetResult(ctx context.Context) (int64, error)

GetResult returns the optional offset of this row, or the associated error. It blocks until the result is ready.

func (*AppendResult) MarkReady 

func (ar *AppendResult) MarkReady()

func (*AppendResult) Ready 

func (ar *AppendResult) Ready() <-chan struct{}

Ready blocks until the append request has reached a completed state, which may be a successful append or an error.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is a managed BigQuery Storage write client scoped to a single project.

func NewClient 

func NewClient(ctx context.Context, projectID string, maxRetries int, initialRetryDelay time.Duration, maxRetryDeadlineOffset time.Duration, retryDelayMultiplier float64, opts ...option.ClientOption) (c *Client, err error)

NewClient instantiates a new client.

func (*Client) BatchCommitWriteStreams 

func (c *Client) BatchCommitWriteStreams(ctx context.Context, req *storagepb.BatchCommitWriteStreamsRequest, opts ...gax.CallOption) (*storagepb.BatchCommitWriteStreamsResponse, error)

BatchCommitWriteStreams atomically commits a group of PENDING streams that belong to the same parent table.

Streams must be finalized before commit and cannot be committed multiple times. Once a stream is committed, data in the stream becomes available for read operations.

func (*Client) Close 

func (c *Client) Close() error

Close releases resources held by the client.

func (*Client) CreateWriteStream 

func (c *Client) CreateWriteStream(ctx context.Context, req *storagepb.CreateWriteStreamRequest, opts ...gax.CallOption) (*storagepb.WriteStream, error)

CreateWriteStream creates a write stream to the given table. Additionally, every table has a special stream named ‘_default’ to which data can be written. This stream doesn’t need to be created using CreateWriteStream. It is a stream that can be used simultaneously by any number of clients. Data written to this stream is considered committed as soon as an acknowledgement is received.

func (*Client) NewManagedStream 

func (c *Client) NewManagedStream(ctx context.Context, opts ...WriterOption) (*ManagedStream, error)

NewManagedStream establishes a new managed stream for appending data into a table.

Context here is retained for use by the underlying streaming connections the managed stream may create.

type ManagedStream 

type ManagedStream struct {
	// contains filtered or unexported fields
}

ManagedStream is the abstraction over a single write stream.

func (*ManagedStream) AppendRows 

func (ms *ManagedStream) AppendRows(ctx context.Context, data [][]byte, offset int64) (*AppendResult, error)

AppendRows sends the append requests to the service, and returns a single AppendResult for tracking the set of data.

The format of the row data is binary serialized protocol buffer bytes, and and the message must adhere to the format of the schema Descriptor passed in when creating the managed stream.

func (*ManagedStream) Close 

func (ms *ManagedStream) Close() error

Close closes a managed stream.

func (*ManagedStream) Finalize 

func (ms *ManagedStream) Finalize(ctx context.Context) (int64, error)

Finalize is used to mark a stream as complete, and thus ensure no further data can be appended to the stream. You cannot finalize a DefaultStream, as it always exists.

Finalizing does not advance the current offset of a BufferedStream, nor does it commit data in a PendingStream.

func (*ManagedStream) FlushRows 

func (ms *ManagedStream) FlushRows(ctx context.Context, offset int64) (int64, error)

FlushRows advances the offset at which rows in a BufferedStream are visible. Calling this method for other stream types yields an error.

func (*ManagedStream) StreamName 

func (ms *ManagedStream) StreamName() string

StreamName returns the corresponding write stream ID being managed by this writer.

func (*ManagedStream) StreamType 

func (ms *ManagedStream) StreamType() StreamType

StreamType returns the configured type for this stream.

type StreamType 

type StreamType string

StreamType indicates the type of stream this write client is managing. 

var (
	// DefaultStream most closely mimics the legacy bigquery
	// tabledata.insertAll semantics.  Successful inserts are
	// committed immediately, and there's no tracking offsets as
	// all writes go into a "default" stream that always exists
	// for a table.
	DefaultStream StreamType = "DEFAULT"

	// CommittedStream appends data immediately, but creates a
	// discrete stream for the work so that offset tracking can
	// be used to track writes.
	CommittedStream StreamType = "COMMITTED"

	// BufferedStream is a form of checkpointed stream, that allows
	// you to advance the offset of visible rows via Flush operations.
	//
	// NOTE: Buffered Streams are currently in limited preview, and as such
	// methods like FlushRows() may yield errors for non-enrolled projects.
	BufferedStream StreamType = "BUFFERED"

	// PendingStream is a stream in which no data is made visible to
	// readers until the stream is finalized and committed explicitly.
	PendingStream StreamType = "PENDING"
)

type WriterOption 

type WriterOption func(*ManagedStream)

WriterOption are variadic options used to configure a ManagedStream instance.

func WithDataOrigin 

func WithDataOrigin(dataOrigin string) WriterOption

WithDataOrigin is used to attach an origin context to the instrumentation metrics emitted by the library.

func WithDestinationTable 

func WithDestinationTable(destTable string) WriterOption

WithDestinationTable specifies the destination table to which a created stream will append rows. Format of the table: 

projects/{projectid}/datasets/{dataset}/tables/{table}

func WithMaxInflightBytes 

func WithMaxInflightBytes(n int) WriterOption

WithMaxInflightBytes bounds the inflight append request bytes on the write connection.

func WithMaxInflightRequests 

func WithMaxInflightRequests(n int) WriterOption

WithMaxInflightRequests bounds the inflight appends on the write connection.

func WithSchemaDescriptor 

func WithSchemaDescriptor(dp *descriptorpb.DescriptorProto) WriterOption

WithSchemaDescriptor describes the format of the serialized data being sent by AppendRows calls on the stream.

func WithStreamName 

func WithStreamName(name string) WriterOption

WithStreamName allows users to set the stream name this writer will append to explicitly. By default, the managed client will create the stream when instantiated if necessary.

Note: Supplying this option causes other options which affect stream construction such as WithStreamType and WithDestinationTable to be ignored.

func WithTraceID 

func WithTraceID(traceID string) WriterOption

WithTraceID allows instruments requests to the service with a custom trace prefix. This is generally for diagnostic purposes only.

func WithType 

func WithType(st StreamType) WriterOption

WithType sets the stream type for the managed stream.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.