Documentation
¶
Overview ¶
Package indexobj provides tooling for creating index-oriented data objects.
Index ¶
- Variables
- type Builder
- func (b *Builder) AppendColumnIndex(tenantID string, path string, section int64, columnName string, ...) error
- func (b *Builder) AppendIndexPointer(tenantID string, path string, startTs time.Time, endTs time.Time) error
- func (b *Builder) AppendStream(tenantID string, stream streams.Stream) (int64, error)
- func (b *Builder) Flush() (*dataobj.Object, io.Closer, error)
- func (b *Builder) GetEstimatedSize() int
- func (b *Builder) ObserveLogLine(tenantID string, path string, section int64, streamIDInObject int64, ...) error
- func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
- func (b *Builder) Reset()
- func (b *Builder) TimeRanges() []multitenancy.TimeRange
- func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
- type BuilderConfig
Constants ¶
This section is empty.
Variables ¶
var ( ErrBuilderFull = errors.New("builder full") ErrBuilderEmpty = errors.New("builder empty") )
ErrBuilderFull is returned by [Builder.Append] when the buffer is full and needs to flush; call Builder.Flush to flush it.
Functions ¶
This section is empty.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
A Builder constructs a logs-oriented data object from a set of incoming log data. Log data is appended by calling [LogBuilder.Append]. A complete data object is constructed by by calling [LogBuilder.Flush].
Methods on Builder are not goroutine-safe; callers are responsible for synchronization.
func NewBuilder ¶
func NewBuilder(cfg BuilderConfig, scratchStore scratch.Store) (*Builder, error)
NewBuilder creates a new Builder which stores log-oriented data objects.
NewBuilder returns an error if the provided config is invalid.
func (*Builder) AppendColumnIndex ¶
func (b *Builder) AppendColumnIndex(tenantID string, path string, section int64, columnName string, columnIndex int64, valuesBloom []byte) error
Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBuilderFull if the builder is full.
Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.
func (*Builder) AppendIndexPointer ¶
func (*Builder) AppendStream ¶
AppendStream appends a stream to the object's stream section, returning the stream ID within this object.
func (*Builder) Flush ¶
Flush flushes all buffered data to the buffer provided. Calling Flush can result in a no-op if there is no buffered data to flush.
Builder.Reset is called after a successful Flush to discard any pending data and allow new data to be appended.
func (*Builder) GetEstimatedSize ¶
func (*Builder) ObserveLogLine ¶
func (b *Builder) ObserveLogLine(tenantID string, path string, section int64, streamIDInObject int64, streamIDInIndex int64, ts time.Time, uncompressedSize int64) error
Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBuilderFull if the builder is full.
Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.
func (*Builder) RegisterMetrics ¶
func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
RegisterMetrics registers metrics about builder to report to reg. All metrics will have a tenant label set to the tenant ID of the Builder.
If multiple Builders for the same tenant are running in the same process, reg must contain additional labels to differentiate between them.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset discards pending data and resets the builder to an empty state.
func (*Builder) TimeRanges ¶
func (b *Builder) TimeRanges() []multitenancy.TimeRange
TimeRanges returns the time range of the data in the builder, by tenant.
func (*Builder) UnregisterMetrics ¶
func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
UnregisterMetrics unregisters metrics about builder from reg.
type BuilderConfig ¶
type BuilderConfig struct {
// TargetPageSize configures a target size for encoded pages within the data
// object. TargetPageSize accounts for encoding, but not for compression.
TargetPageSize flagext.Bytes `yaml:"target_page_size"`
// MaxPageRows configures a maximum row count for encoded pages within the data
// object. If set to 0 or negative number, the page size will not be limited by a
// row count.
MaxPageRows int `yaml:"max_page_rows"`
// TargetObjectSize configures a target size for data objects.
TargetObjectSize flagext.Bytes `yaml:"target_object_size"`
// TargetSectionSize configures the maximum size of data in a section. Sections
// which support this parameter will place overflow data into new sections of
// the same type.
TargetSectionSize flagext.Bytes `yaml:"target_section_size"`
// BufferSize configures the size of the buffer used to accumulate
// uncompressed logs in memory prior to sorting.
BufferSize flagext.Bytes `yaml:"buffer_size"`
// SectionStripeMergeLimit configures the number of stripes to merge at once when
// flushing stripes into a section. MergeSize must be larger than 1. Lower
// values of MergeSize trade off lower memory overhead for higher time spent
// merging.
SectionStripeMergeLimit int `yaml:"section_stripe_merge_limit"`
}
BuilderConfig configures a Builder.
func (*BuilderConfig) RegisterFlagsWithPrefix ¶
func (cfg *BuilderConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
RegisterFlagsWithPrefix registers flags with the given prefix.
func (*BuilderConfig) Validate ¶
func (cfg *BuilderConfig) Validate() error
Validate validates the BuilderConfig.
Source Files