storage

package
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrIntakeMemoryExhausted = errors.New("intake memory exhausted")

ErrIntakeMemoryExhausted reports that too many concurrent requests are already buffered. It is retryable: the sender keeps its data.

Functions

func IntakeNeeded added in v0.4.1

func IntakeNeeded(size int64) int64

IntakeNeeded returns the intake memory an import of a size-byte body is charged at its peak: the buffer that holds it, the smallest intakeStep doubled enough times, plus the half-size buffer it grows from.

func SegmentKey

func SegmentKey(dst []byte, req *prompb.WriteRequest, series *prompb.TimeSeries, hash uint64) ([]byte, error)

SegmentKey returns the segment key for series using its effective labels from req, including request-level common labels and the per-series filter.

Types

type Backend added in v0.2.0

type Backend string
const (
	BackendADX        Backend = "adx"
	BackendClickHouse Backend = "clickhouse"
)

func ParseBackend added in v0.2.0

func ParseBackend(raw string) (Backend, error)

func (Backend) String added in v0.2.0

func (b Backend) String() string

func (Backend) Validate added in v0.2.0

func (b Backend) Validate() error

type LocalStore

type LocalStore struct {
	// contains filtered or unexported fields
}

LocalStore provides local storage of time series data. It manages Write Ahead Logs (WALs) for each metric.

func NewLocalStore

func NewLocalStore(opts StoreOpts) *LocalStore

func (*LocalStore) Close

func (s *LocalStore) Close() error

func (*LocalStore) GetWAL

func (s *LocalStore) GetWAL(ctx context.Context, key []byte) (*wal.WAL, error)

func (*LocalStore) Import

func (s *LocalStore) Import(filename string, body io.ReadCloser) (int, error)

func (*LocalStore) Index

func (s *LocalStore) Index() *wal.Index

func (*LocalStore) Open

func (s *LocalStore) Open(ctx context.Context) error

func (*LocalStore) PrefixesByAge

func (s *LocalStore) PrefixesByAge() []string

func (*LocalStore) Remove

func (s *LocalStore) Remove(path string) error

func (*LocalStore) SegmentCount added in v0.4.0

func (s *LocalStore) SegmentCount() int64

func (*LocalStore) Size added in v0.2.0

func (s *LocalStore) Size() int64

func (*LocalStore) TracksSegment added in v0.4.0

func (s *LocalStore) TracksSegment(path string) bool

TracksSegment returns true if path is a WAL segment file the store accounts for, either a closed segment pending upload or an active segment still being written. A segment file on disk that is not tracked is orphaned: it is never uploaded or removed until the store is reopened.

func (*LocalStore) WALCount

func (s *LocalStore) WALCount() int

func (*LocalStore) WriteDebug added in v0.2.0

func (s *LocalStore) WriteDebug(w io.Writer) error

WriteDebug writes debug information to the given writer.

func (*LocalStore) WriteNativeLogs

func (s *LocalStore) WriteNativeLogs(ctx context.Context, logs *types.LogBatch) error

func (*LocalStore) WriteOTLPLogs

func (s *LocalStore) WriteOTLPLogs(ctx context.Context, database, table string, logs *otlp.Logs) error

func (*LocalStore) WriteTimeSeries

func (s *LocalStore) WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error

type MemoryLimiter added in v0.4.0

type MemoryLimiter struct {
	// contains filtered or unexported fields
}

MemoryLimiter bounds the total bytes buffered by concurrent intake requests. Disk budgets alone do not protect the process: each transfer buffers its body before admission, so without an aggregate ceiling enough concurrent requests can exhaust memory regardless of how much disk capacity remains.

func NewMemoryLimiter added in v0.4.0

func NewMemoryLimiter(limit int64) *MemoryLimiter

func (*MemoryLimiter) InFlight added in v0.4.0

func (m *MemoryLimiter) InFlight() int64

InFlight reports the bytes currently reserved.

func (*MemoryLimiter) Limit added in v0.4.0

func (m *MemoryLimiter) Limit() int64

Limit reports the ceiling; zero means it is disabled.

func (*MemoryLimiter) Release added in v0.4.0

func (m *MemoryLimiter) Release(n int64)

Release returns previously reserved bytes.

func (*MemoryLimiter) Reserve added in v0.4.0

func (m *MemoryLimiter) Reserve(n int64) bool

Reserve claims n bytes of intake memory. It reports false when the claim would exceed the ceiling, in which case the caller must shed or retry later. Without a ceiling every claim succeeds but is still counted, so usage can be measured before a ceiling is chosen.

type PartialWriteError added in v0.4.0

type PartialWriteError struct {
	Pending []int
	Err     error
}

PartialWriteError reports that a write stopped because storage was full. Pending holds the indices, in order, of the items that were not written; the others were written or shed, so the pending items can be retried without duplicating them.

func (*PartialWriteError) Error added in v0.4.0

func (e *PartialWriteError) Error() string

func (*PartialWriteError) Unwrap added in v0.4.0

func (e *PartialWriteError) Unwrap() error

type Store

type Store interface {
	service.Component

	// WriteTimeSeries writes a batch of time series to the Store.
	WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error

	// WriteOTLPLogs writes a batch of logs to the Store.
	WriteOTLPLogs(ctx context.Context, database, table string, logs *otlp.Logs) error

	// WriteNativeLogs writes a batch of logs to the Store.
	WriteNativeLogs(ctx context.Context, logs *types.LogBatch) error

	// Import imports a file into the LocalStore and returns the number of bytes stored.
	Import(filename string, body io.ReadCloser) (int, error)
}

type StoreOpts

type StoreOpts struct {
	StorageDir             string
	SegmentMaxSize         int64
	SegmentMaxAge          time.Duration
	MaxDiskUsage           int64
	StartupOpenConcurrency int
	Backend                Backend

	LiftedLabels     []string
	LiftedAttributes []string
	LiftedResources  []string
	WALFlushInterval time.Duration
	EnableWALFsync   bool

	// Policy assigns an ingestion priority to each table.  When nil, all tables are queued.
	Policy *ingestpolicy.Policy

	// Realtime overrides segment rotation settings for realtime tables.
	Realtime wal.RotationPolicy

	// UnownedSegmentMaxAge, when positive, replaces SegmentMaxAge for queued tables that Owned reports are owned by
	// another node.
	UnownedSegmentMaxAge time.Duration

	// Owned reports whether this node owns the table of a WAL prefix.  When nil, every table is owned.
	Owned func(prefix string) bool

	// RealtimeReservedDiskBytes is disk space reserved for realtime tables.  Queued writes are rejected once disk
	// usage reaches MaxDiskUsage minus this value.
	RealtimeReservedDiskBytes int64

	// Admission owns per-destination capacity accounting.
	Admission *wal.Admission

	// MetricComponent labels exported table budget metrics.
	MetricComponent string

	// MaxIntakeMemory bounds the total bytes buffered across concurrent
	// imports. Zero disables the ceiling.
	MaxIntakeMemory int64
}

type WriteOutcome added in v0.4.0

type WriteOutcome struct {
	Accepted       int64
	Rejected       int64
	RejectedSeries []*prompb.TimeSeries
}

WriteOutcome reports records that were intentionally shed because a table exceeded its budget. It is carried through the error return of the storage and remote-write interfaces, but it is not a failure: the records are deliberately discarded and must not be retried by the producer.

Callers extract it with SplitOutcome rather than treating it as an error. RejectedSeries is only valid until the caller returns those series to their pool.

func SplitOutcome added in v0.4.0

func SplitOutcome(err error) (WriteOutcome, error)

SplitOutcome separates intentional shedding from failures that require a retry. A successful local write must never mask a failure reported by another destination.

A wrapped outcome still contributes its shed counts, but it is also treated as a failure: wrapping signals that the caller attached a condition beyond deliberate shedding, and silently swallowing that would drop data.

func (*WriteOutcome) Error added in v0.4.0

func (o *WriteOutcome) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL