Documentation
¶
Index ¶
- Variables
- func IntakeNeeded(size int64) int64
- func SegmentKey(dst []byte, req *prompb.WriteRequest, series *prompb.TimeSeries, hash uint64) ([]byte, error)
- type Backend
- type LocalStore
- func (s *LocalStore) Close() error
- func (s *LocalStore) GetWAL(ctx context.Context, key []byte) (*wal.WAL, error)
- func (s *LocalStore) Import(filename string, body io.ReadCloser) (int, error)
- func (s *LocalStore) Index() *wal.Index
- func (s *LocalStore) Open(ctx context.Context) error
- func (s *LocalStore) PrefixesByAge() []string
- func (s *LocalStore) Remove(path string) error
- func (s *LocalStore) SegmentCount() int64
- func (s *LocalStore) Size() int64
- func (s *LocalStore) TracksSegment(path string) bool
- func (s *LocalStore) WALCount() int
- func (s *LocalStore) WriteDebug(w io.Writer) error
- func (s *LocalStore) WriteNativeLogs(ctx context.Context, logs *types.LogBatch) error
- func (s *LocalStore) WriteOTLPLogs(ctx context.Context, database, table string, logs *otlp.Logs) error
- func (s *LocalStore) WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error
- type MemoryLimiter
- type PartialWriteError
- type Store
- type StoreOpts
- type WriteOutcome
Constants ¶
This section is empty.
Variables ¶
var ErrIntakeMemoryExhausted = errors.New("intake memory exhausted")
ErrIntakeMemoryExhausted reports that too many concurrent requests are already buffered. It is retryable: the sender keeps its data.
Functions ¶
func IntakeNeeded ¶ added in v0.4.1
IntakeNeeded returns the intake memory an import of a size-byte body is charged at its peak: the buffer that holds it, the smallest intakeStep doubled enough times, plus the half-size buffer it grows from.
func SegmentKey ¶
func SegmentKey(dst []byte, req *prompb.WriteRequest, series *prompb.TimeSeries, hash uint64) ([]byte, error)
SegmentKey returns the segment key for series using its effective labels from req, including request-level common labels and the per-series filter.
Types ¶
type LocalStore ¶
type LocalStore struct {
// contains filtered or unexported fields
}
LocalStore provides local storage of time series data. It manages Write Ahead Logs (WALs) for each metric.
func NewLocalStore ¶
func NewLocalStore(opts StoreOpts) *LocalStore
func (*LocalStore) Close ¶
func (s *LocalStore) Close() error
func (*LocalStore) Import ¶
func (s *LocalStore) Import(filename string, body io.ReadCloser) (int, error)
func (*LocalStore) Index ¶
func (s *LocalStore) Index() *wal.Index
func (*LocalStore) PrefixesByAge ¶
func (s *LocalStore) PrefixesByAge() []string
func (*LocalStore) Remove ¶
func (s *LocalStore) Remove(path string) error
func (*LocalStore) SegmentCount ¶ added in v0.4.0
func (s *LocalStore) SegmentCount() int64
func (*LocalStore) Size ¶ added in v0.2.0
func (s *LocalStore) Size() int64
func (*LocalStore) TracksSegment ¶ added in v0.4.0
func (s *LocalStore) TracksSegment(path string) bool
TracksSegment returns true if path is a WAL segment file the store accounts for, either a closed segment pending upload or an active segment still being written. A segment file on disk that is not tracked is orphaned: it is never uploaded or removed until the store is reopened.
func (*LocalStore) WALCount ¶
func (s *LocalStore) WALCount() int
func (*LocalStore) WriteDebug ¶ added in v0.2.0
func (s *LocalStore) WriteDebug(w io.Writer) error
WriteDebug writes debug information to the given writer.
func (*LocalStore) WriteNativeLogs ¶
func (*LocalStore) WriteOTLPLogs ¶
func (*LocalStore) WriteTimeSeries ¶
func (s *LocalStore) WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error
type MemoryLimiter ¶ added in v0.4.0
type MemoryLimiter struct {
// contains filtered or unexported fields
}
MemoryLimiter bounds the total bytes buffered by concurrent intake requests. Disk budgets alone do not protect the process: each transfer buffers its body before admission, so without an aggregate ceiling enough concurrent requests can exhaust memory regardless of how much disk capacity remains.
func NewMemoryLimiter ¶ added in v0.4.0
func NewMemoryLimiter(limit int64) *MemoryLimiter
func (*MemoryLimiter) InFlight ¶ added in v0.4.0
func (m *MemoryLimiter) InFlight() int64
InFlight reports the bytes currently reserved.
func (*MemoryLimiter) Limit ¶ added in v0.4.0
func (m *MemoryLimiter) Limit() int64
Limit reports the ceiling; zero means it is disabled.
func (*MemoryLimiter) Release ¶ added in v0.4.0
func (m *MemoryLimiter) Release(n int64)
Release returns previously reserved bytes.
func (*MemoryLimiter) Reserve ¶ added in v0.4.0
func (m *MemoryLimiter) Reserve(n int64) bool
Reserve claims n bytes of intake memory. It reports false when the claim would exceed the ceiling, in which case the caller must shed or retry later. Without a ceiling every claim succeeds but is still counted, so usage can be measured before a ceiling is chosen.
type PartialWriteError ¶ added in v0.4.0
PartialWriteError reports that a write stopped because storage was full. Pending holds the indices, in order, of the items that were not written; the others were written or shed, so the pending items can be retried without duplicating them.
func (*PartialWriteError) Error ¶ added in v0.4.0
func (e *PartialWriteError) Error() string
func (*PartialWriteError) Unwrap ¶ added in v0.4.0
func (e *PartialWriteError) Unwrap() error
type Store ¶
type Store interface {
service.Component
// WriteTimeSeries writes a batch of time series to the Store.
WriteTimeSeries(ctx context.Context, req *prompb.WriteRequest) error
// WriteOTLPLogs writes a batch of logs to the Store.
WriteOTLPLogs(ctx context.Context, database, table string, logs *otlp.Logs) error
// WriteNativeLogs writes a batch of logs to the Store.
WriteNativeLogs(ctx context.Context, logs *types.LogBatch) error
// Import imports a file into the LocalStore and returns the number of bytes stored.
Import(filename string, body io.ReadCloser) (int, error)
}
type StoreOpts ¶
type StoreOpts struct {
StorageDir string
SegmentMaxSize int64
SegmentMaxAge time.Duration
MaxDiskUsage int64
StartupOpenConcurrency int
Backend Backend
LiftedLabels []string
LiftedAttributes []string
LiftedResources []string
WALFlushInterval time.Duration
EnableWALFsync bool
// Policy assigns an ingestion priority to each table. When nil, all tables are queued.
Policy *ingestpolicy.Policy
// Realtime overrides segment rotation settings for realtime tables.
Realtime wal.RotationPolicy
// UnownedSegmentMaxAge, when positive, replaces SegmentMaxAge for queued tables that Owned reports are owned by
// another node.
UnownedSegmentMaxAge time.Duration
// Owned reports whether this node owns the table of a WAL prefix. When nil, every table is owned.
Owned func(prefix string) bool
// RealtimeReservedDiskBytes is disk space reserved for realtime tables. Queued writes are rejected once disk
// usage reaches MaxDiskUsage minus this value.
RealtimeReservedDiskBytes int64
// Admission owns per-destination capacity accounting.
Admission *wal.Admission
// MetricComponent labels exported table budget metrics.
MetricComponent string
// MaxIntakeMemory bounds the total bytes buffered across concurrent
// imports. Zero disables the ceiling.
MaxIntakeMemory int64
}
type WriteOutcome ¶ added in v0.4.0
type WriteOutcome struct {
Accepted int64
Rejected int64
RejectedSeries []*prompb.TimeSeries
}
WriteOutcome reports records that were intentionally shed because a table exceeded its budget. It is carried through the error return of the storage and remote-write interfaces, but it is not a failure: the records are deliberately discarded and must not be retried by the producer.
Callers extract it with SplitOutcome rather than treating it as an error. RejectedSeries is only valid until the caller returns those series to their pool.
func SplitOutcome ¶ added in v0.4.0
func SplitOutcome(err error) (WriteOutcome, error)
SplitOutcome separates intentional shedding from failures that require a retry. A successful local write must never mask a failure reported by another destination.
A wrapped outcome still contributes its shed counts, but it is also treated as a failure: wrapping signals that the caller attached a condition beyond deliberate shedding, and silently swallowing that would drop data.
func (*WriteOutcome) Error ¶ added in v0.4.0
func (o *WriteOutcome) Error() string