Documentation
¶
Overview ¶
Package logsobj provides tooling for creating logs-oriented data objects.
Index ¶
- Variables
- type Builder
- func (b *Builder) Append(tenant string, stream logproto.Stream) error
- func (b *Builder) CopyAndSort(ctx context.Context, obj *dataobj.Object) (*dataobj.Object, io.Closer, error)
- func (b *Builder) Flush() (*dataobj.Object, io.Closer, error)
- func (b *Builder) GetEstimatedSize() int
- func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
- func (b *Builder) Reset()
- func (b *Builder) TimeRanges() []multitenancy.TimeRange
- func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
- type BuilderBaseConfig
- type BuilderConfig
- type BuilderFactory
- type SizedBuilderPool
- type Sorter
Constants ¶
This section is empty.
Variables ¶
var ( ErrBuilderFull = errors.New("builder full") ErrBuilderEmpty = errors.New("builder empty") )
ErrBuilderFull is returned by Builder.Append when the buffer is full and needs to flush; call Builder.Flush to flush it.
Functions ¶
This section is empty.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
A Builder constructs a logs-oriented data object from a set of incoming log data. Log data is appended by calling [LogBuilder.Append]. A complete data object is constructed by by calling [LogBuilder.Flush].
Methods on Builder are not goroutine-safe; callers are responsible for synchronization.
func NewBuilder ¶
func NewBuilder(cfg BuilderConfig, scratchStore scratch.Store) (*Builder, error)
NewBuilder creates a new Builder which stores log-oriented data objects.
NewBuilder returns an error if the provided config is invalid.
func (*Builder) Append ¶
Append buffers a stream to be written to a data object. Append returns an error if the stream labels cannot be parsed or ErrBuilderFull if the builder is full.
Once a Builder is full, call Builder.Flush to flush the buffered data, then call Append again with the same entry.
func (*Builder) CopyAndSort ¶
func (b *Builder) CopyAndSort(ctx context.Context, obj *dataobj.Object) (*dataobj.Object, io.Closer, error)
CopyAndSort takes an existing dataobj.Object and rewrites the logs sections so the logs are sorted object-wide. The order of the sections is deterministic. For each tenant, first come the streams sections in the order of the old object and second come the new, rewritten logs sections. Tenants are sorted in natural order.
func (*Builder) Flush ¶
Flush flushes all buffered data to the buffer provided. Calling Flush can result in a no-op if there is no buffered data to flush.
Builder.Reset is called after a successful Flush to discard any pending data and allow new data to be appended.
func (*Builder) GetEstimatedSize ¶
func (*Builder) RegisterMetrics ¶
func (b *Builder) RegisterMetrics(reg prometheus.Registerer) error
RegisterMetrics registers metrics about builder to report to reg. All metrics will have a tenant label set to the tenant ID of the Builder.
If multiple Builders for the same tenant are running in the same process, reg must contain additional labels to differentiate between them.
func (*Builder) Reset ¶
func (b *Builder) Reset()
Reset discards pending data and resets the builder to an empty state.
func (*Builder) TimeRanges ¶
func (b *Builder) TimeRanges() []multitenancy.TimeRange
TimeRanges returns the time ranges for each tenant.
func (*Builder) UnregisterMetrics ¶
func (b *Builder) UnregisterMetrics(reg prometheus.Registerer)
UnregisterMetrics unregisters metrics about builder from reg.
type BuilderBaseConfig ¶ added in v3.7.0
type BuilderBaseConfig struct {
// TargetPageSize configures a target size for encoded pages within the data
// object. TargetPageSize accounts for encoding, but not for compression.
TargetPageSize flagext.Bytes `yaml:"target_page_size"`
// MaxPageRows configures a maximum row count for encoded pages within the data
// object. If set to 0 or negative number, the page size will not be limited by a
// row count.
MaxPageRows int `yaml:"max_page_rows"`
// TargetObjectSize configures a target size for data objects.
TargetObjectSize flagext.Bytes `yaml:"target_object_size"`
// TargetSectionSize configures the maximum size of data in a section. Sections
// which support this parameter will place overflow data into new sections of
// the same type.
TargetSectionSize flagext.Bytes `yaml:"target_section_size"`
// BufferSize configures the size of the buffer used to accumulate
// uncompressed logs in memory prior to sorting.
BufferSize flagext.Bytes `yaml:"buffer_size"`
// SectionStripeMergeLimit configures the number of stripes to merge at once when
// flushing stripes into a section. MergeSize must be larger than 1. Lower
// values of MergeSize trade off lower memory overhead for higher time spent
// merging.
SectionStripeMergeLimit int `yaml:"section_stripe_merge_limit"`
}
BuilderBaseConfig configures a data object builder.
func (*BuilderBaseConfig) RegisterFlagsWithPrefix ¶ added in v3.7.0
func (cfg *BuilderBaseConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
RegisterFlagsWithPrefix registers flags with the given prefix.
func (*BuilderBaseConfig) Validate ¶ added in v3.7.0
func (cfg *BuilderBaseConfig) Validate() error
Validate validates the BuilderConfig.
type BuilderConfig ¶
type BuilderConfig struct {
BuilderBaseConfig `yaml:",inline"`
// DataobjSortOrder defines the order in which the rows of the logs sections are sorted.
// They can either be sorted by [streamID ASC, timestamp DESC] or [timestamp DESC, streamID ASC].
DataobjSortOrder string `yaml:"dataobj_sort_order" doc:"hidden"`
}
BuilderConfig configures a Builder.
func (*BuilderConfig) RegisterFlagsWithPrefix ¶
func (cfg *BuilderConfig) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet)
RegisterFlagsWithPrefix registers flags with the given prefix.
func (*BuilderConfig) Validate ¶
func (cfg *BuilderConfig) Validate() error
Validate validates the BuilderConfig.
type BuilderFactory ¶ added in v3.7.0
type BuilderFactory struct {
// contains filtered or unexported fields
}
A BuilderFactory is used to create builders.
func NewBuilderFactory ¶ added in v3.7.0
func NewBuilderFactory(cfg BuilderConfig, scratchStore scratch.Store) *BuilderFactory
func (*BuilderFactory) NewBuilder ¶ added in v3.7.0
func (f *BuilderFactory) NewBuilder(r prometheus.Registerer) (*Builder, error)
NewBuilder returns a new builder, or an error. The registerer is optional. No metrics will be registered if the registerer is nil.
type SizedBuilderPool ¶ added in v3.7.0
type SizedBuilderPool struct {
// contains filtered or unexported fields
}
A SizedBuilderPool implements a fixed-size pool of builders.
func NewSizedBuilderPool ¶ added in v3.7.0
func NewSizedBuilderPool(builders []*Builder) *SizedBuilderPool
NewSizedBuilderPool returns a new SizedBuilderPool.
func (*SizedBuilderPool) Get ¶ added in v3.7.0
func (p *SizedBuilderPool) Get() *Builder
Get returns the next builder in the pool. If there are no builders available, because the pool is currently empty, it returns nil instead.
func (*SizedBuilderPool) Put ¶ added in v3.7.0
func (p *SizedBuilderPool) Put(b *Builder)
Put returns the builder to the pool.
func (*SizedBuilderPool) Wait ¶ added in v3.7.0
func (p *SizedBuilderPool) Wait(ctx context.Context) (*Builder, error)
Wait returns the next builder in the pool. If there are no builders available, because the pool is currently empty, it blocks until either a builder becomes available or the context is canceled.
type Sorter ¶ added in v3.7.0
type Sorter struct {
// contains filtered or unexported fields
}
A Sorter sorts data objects.
func NewSorter ¶ added in v3.7.0
func NewSorter(factory *BuilderFactory, r prometheus.Registerer) *Sorter
NewSorter returns a new Sorter.
Source Files