storage

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrIntakeMemoryExhausted = errors.New("intake memory exhausted")

ErrIntakeMemoryExhausted reports that too many concurrent requests are already buffered. It is retryable: the sender keeps its data.

Functions

func SegmentKey

func SegmentKey(dst []byte, req *prompb.WriteRequest, series *prompb.TimeSeries, hash uint64) ([]byte, error)

SegmentKey returns the segment key for series using its effective labels from req, including request-level common labels and the per-series filter.

Types

type Backend added in v0.2.0

type Backend string
const (
	BackendADX        Backend = "adx"
	BackendClickHouse Backend = "clickhouse"
)

func ParseBackend added in v0.2.0

func ParseBackend(raw string) (Backend, error)

func (Backend) String added in v0.2.0

func (b Backend) String() string

func (Backend) Validate added in v0.2.0

func (b Backend) Validate() error

type LocalStore

type LocalStore struct {
	// contains filtered or unexported fields
}

LocalStore provides local storage of time series data. It manages Write Ahead Logs (WALs) for each metric.

func NewLocalStore

func NewLocalStore(opts StoreOpts) *LocalStore

func (*LocalStore) Close

func (s *LocalStore) Close() error

func (*LocalStore) GetWAL

func (s *LocalStore) GetWAL(ctx context.Context, key []byte) (*wal.WAL, error)

func (*LocalStore) Import

func (s *LocalStore) Import(filename string, body io.ReadCloser) (int, error)

func (*LocalStore) ImportWith added in v0.4.0

func (s *LocalStore) ImportWith(filename string, body io.ReadCloser, forward func(data []byte) bool) (int, error)

func (*LocalStore) Index

func (s *LocalStore) Index() *wal.Index

func (*LocalStore) Open

func (s *LocalStore) Open(ctx context.Context) error

func (*LocalStore) PrefixesByAge

func (s *LocalStore) PrefixesByAge() []string

func (*LocalStore) Remove

func (s *LocalStore) Remove(path string) error

func (*LocalStore) SegmentCount added in v0.4.0

func (s *LocalStore) SegmentCount() int64

func (*LocalStore) Size added in v0.2.0

func (s *LocalStore) Size() int64

func (*LocalStore) TracksSegment added in v0.4.0

func (s *LocalStore) TracksSegment(path string) bool

TracksSegment returns true if path is a WAL segment file the store accounts for, either a closed segment pending upload or an active segment still being written. A segment file on disk that is not tracked is orphaned: it is never uploaded or removed until the store is reopened.

func (*LocalStore) WALCount

func (s *LocalStore) WALCount() int

func (*LocalStore) WriteDebug added in v0.2.0

func (s *LocalStore) WriteDebug(w io.Writer) error

WriteDebug writes debug information to the given writer.

func (*LocalStore) WriteNativeLogs

func (s *LocalStore) WriteNativeLogs(ctx context.Context, logs *types.LogBatch) error

func (*LocalStore) WriteOTLPLogs

func (s *LocalStore) WriteOTLPLogs(ctx context.Context, database, table string, logs *otlp.Logs) error

func (*LocalStore) WriteTimeSeries

func (s *LocalStore) WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error

type MemoryLimiter added in v0.4.0

type MemoryLimiter struct {
	// contains filtered or unexported fields
}

MemoryLimiter bounds the total bytes buffered by concurrent intake requests. Disk budgets alone do not protect the process: each transfer buffers its body before admission, so without an aggregate ceiling enough concurrent requests can exhaust memory regardless of how much disk capacity remains.

func NewMemoryLimiter added in v0.4.0

func NewMemoryLimiter(limit int64) *MemoryLimiter

func (*MemoryLimiter) InFlight added in v0.4.0

func (m *MemoryLimiter) InFlight() int64

InFlight reports the bytes currently reserved.

func (*MemoryLimiter) Limit added in v0.4.0

func (m *MemoryLimiter) Limit() int64

Limit reports the ceiling; zero means it is disabled.

func (*MemoryLimiter) Release added in v0.4.0

func (m *MemoryLimiter) Release(n int64)

Release returns previously reserved bytes.

func (*MemoryLimiter) Reserve added in v0.4.0

func (m *MemoryLimiter) Reserve(n int64) bool

Reserve claims n bytes of intake memory. It reports false when the claim would exceed the ceiling, in which case the caller must shed or retry later. Without a ceiling every claim succeeds but is still counted, so usage can be measured before a ceiling is chosen.

type PartialWriteError added in v0.4.0

type PartialWriteError struct {
	Pending []int
	Err     error
}

PartialWriteError reports that a write stopped because storage was full. Pending holds the indices, in order, of the items that were not written; the others were written or shed, so the pending items can be retried without duplicating them.

func (*PartialWriteError) Error added in v0.4.0

func (e *PartialWriteError) Error() string

func (*PartialWriteError) Unwrap added in v0.4.0

func (e *PartialWriteError) Unwrap() error

type Store

type Store interface {
	service.Component

	// WriteTimeSeries writes a batch of time series to the Store.
	WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error

	// WriteOTLPLogs writes a batch of logs to the Store.
	WriteOTLPLogs(ctx context.Context, database, table string, logs *otlp.Logs) error

	// WriteNativeLogs writes a batch of logs to the Store.
	WriteNativeLogs(ctx context.Context, logs *types.LogBatch) error

	// Import imports a file into the LocalStore and returns the number of bytes stored.
	Import(filename string, body io.ReadCloser) (int, error)

	// ImportWith is like Import, but first offers the buffered file to forward, if set, and does not store it when
	// forward reports that it was accepted elsewhere.
	ImportWith(filename string, body io.ReadCloser, forward func(data []byte) bool) (int, error)
}

type StoreOpts

type StoreOpts struct {
	StorageDir             string
	SegmentMaxSize         int64
	SegmentMaxAge          time.Duration
	MaxDiskUsage           int64
	StartupOpenConcurrency int
	Backend                Backend

	LiftedLabels     []string
	LiftedAttributes []string
	LiftedResources  []string
	WALFlushInterval time.Duration
	EnableWALFsync   bool

	// Policy assigns an ingestion priority to each table.  When nil, all tables are queued.
	Policy *ingestpolicy.Policy

	// Realtime overrides segment rotation settings for realtime tables.
	Realtime wal.RotationPolicy

	// RealtimeReservedDiskBytes is disk space reserved for realtime tables.  Queued writes are rejected once disk
	// usage reaches MaxDiskUsage minus this value.
	RealtimeReservedDiskBytes int64

	// Admission owns per-destination capacity accounting.
	Admission *wal.Admission

	// MetricComponent labels exported table budget metrics.
	MetricComponent string

	// MaxIntakeMemory bounds the total bytes buffered across concurrent
	// imports. Zero disables the ceiling.
	MaxIntakeMemory int64
}

type WriteOutcome added in v0.4.0

type WriteOutcome struct {
	Accepted       int64
	Rejected       int64
	RejectedSeries []*prompb.TimeSeries
}

WriteOutcome reports records that were intentionally shed because a table exceeded its budget. It is carried through the error return of the storage and remote-write interfaces, but it is not a failure: the records are deliberately discarded and must not be retried by the producer.

Callers extract it with SplitOutcome rather than treating it as an error. RejectedSeries is only valid until the caller returns those series to their pool.

func SplitOutcome added in v0.4.0

func SplitOutcome(err error) (WriteOutcome, error)

SplitOutcome separates intentional shedding from failures that require a retry. A successful local write must never mask a failure reported by another destination.

A wrapped outcome still contributes its shed counts, but it is also treated as a failure: wrapping signals that the caller attached a condition beyond deliberate shedding, and silently swallowing that would drop data.

func (*WriteOutcome) Error added in v0.4.0

func (o *WriteOutcome) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL