Documentation
¶
Overview ¶
Package logsobj provides tooling for creating logs-oriented data objects.
Index ¶
- Variables
- type Builder
- func (b *Builder) Append(tenant string, stream logproto.Stream) error
- func (b *Builder) CopyAndSort(obj *dataobj.Object) (*dataobj.Object, io.Closer, error)
- func (b *Builder) Flush() (*dataobj.Object, io.Closer, error)
- func (b *Builder) GetEstimatedSize() int
- func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
- func (b *Builder) Reset()
- func (b *Builder) TimeRanges() []multitenancy.TimeRange
- func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
- type BuilderConfig
Constants ¶
This section is empty.
Variables ¶
var ( ErrBuilderFull = errors.New("builder full") ErrBuilderEmpty = errors.New("builder empty") )
ErrBuilderFull is returned by Builder.Append when the buffer is full and needs to flush; call Builder.Flush to flush it.
Functions ¶
This section is empty.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
A Builder constructs a logs-oriented data object from a set of incoming log data. Log data is appended by calling [LogBuilder.Append]. A complete data object is constructed by by calling [LogBuilder.Flush].
Methods on Builder are not goroutine-safe; callers are responsible for synchronization.
func NewBuilder ¶
func NewBuilder(cfg BuilderConfig, scratchStore scratch.Store) (*Builder, error)
NewBuilder creates a new Builder which stores log-oriented data objects.
NewBuilder returns an error if the provided config is invalid.
func (*Builder) Append ¶
Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBuilderFull if the builder is full.
Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.
func (*Builder) CopyAndSort ¶
CopyAndSort takes an existing dataobj.Object and rewrites the logs sections so the logs are sorted object-wide. The order of the sections is deterministic. For each tenant, first come the streams sections in the order of the old object and second come the new, rewritten logs sections. Tenants are sorted in natural order.
func (*Builder) Flush ¶
Flush flushes all buffered data to the buffer provided. Calling Flush can result in a no-op if there is no buffered data to flush.
Builder.Reset is called after a successful Flush to discard any pending data and allow new data to be appended.
func (*Builder) GetEstimatedSize ¶
func (*Builder) RegisterMetrics ¶
func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
RegisterMetrics registers metrics about builder to report to reg. All metrics will have a tenant label set to the tenant ID of the Builder.
If multiple Builders for the same tenant are running in the same process, reg must contain additional labels to differentiate between them.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset discards pending data and resets the builder to an empty state.
func (*Builder) TimeRanges ¶
func (b *Builder) TimeRanges() []multitenancy.TimeRange
TimeRanges returns the time ranges for each tenant.
func (*Builder) UnregisterMetrics ¶
func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
UnregisterMetrics unregisters metrics about builder from reg.
type BuilderConfig ¶
type BuilderConfig struct {
// TargetPageSize configures a target size for encoded pages within the data
// object. TargetPageSize accounts for encoding, but not for compression.
TargetPageSize flagext.Bytes `yaml:"target_page_size"`
// MaxPageRows configures a maximum row count for encoded pages within the data
// object. If set to 0 or negative number, the page size will not be limited by a
// row count.
MaxPageRows int `yaml:"max_page_rows"`
// TargetObjectSize configures a target size for data objects.
TargetObjectSize flagext.Bytes `yaml:"target_object_size"`
// TargetSectionSize configures the maximum size of data in a section. Sections
// which support this parameter will place overflow data into new sections of
// the same type.
TargetSectionSize flagext.Bytes `yaml:"target_section_size"`
// BufferSize configures the size of the buffer used to accumulate
// uncompressed logs in memory prior to sorting.
BufferSize flagext.Bytes `yaml:"buffer_size"`
// SectionStripeMergeLimit configures the number of stripes to merge at once when
// flushing stripes into a section. MergeSize must be larger than 1. Lower
// values of MergeSize trade off lower memory overhead for higher time spent
// merging.
SectionStripeMergeLimit int `yaml:"section_stripe_merge_limit"`
// DataobjSortOrder defines the order in which the rows of the logs sections are sorted.
// They can either be sorted by [streamID ASC, timestamp DESC] or [timestamp DESC, streamID ASC].
DataobjSortOrder string `yaml:"dataobj_sort_order" doc:"hidden"`
}
BuilderConfig configures a Builder.
func (*BuilderConfig) RegisterFlagsWithPrefix ¶
func (cfg *BuilderConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
RegisterFlagsWithPrefix registers flags with the given prefix.
func (*BuilderConfig) Validate ¶
func (cfg *BuilderConfig) Validate() error
Validate validates the BuilderConfig.
Source Files