insertutil

package
v1.43.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 26, 2025 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// MaxLineSizeBytes is the maximum length of a single line for /insert/* handlers
	MaxLineSizeBytes = flagutil.NewBytes("insert.maxLineSizeBytes", 256*1024, "The maximum size of a single line that can be read by /insert/* handlers. Regardless of this flag, entries above the 2 MB limit are ignored, "+
		"see https://docs.victoriametrics.com/victorialogs/faq/#what-length-a-log-record-is-expected-to-have")

	// MaxFieldsPerLine is the maximum number of fields per line for /insert/* handlers
	MaxFieldsPerLine = flag.Int("insert.maxFieldsPerLine", 1000, "The maximum number of log fields per line, which can be read by /insert/* handlers; "+
		"see https://docs.victoriametrics.com/victorialogs/faq/#how-many-fields-a-single-log-entry-may-contain")
)

Functions

func CanWriteData

func CanWriteData() error

CanWriteData returns non-nil error if data cannot be written to the underlying storage.

func ExtractTimestampFromFields

func ExtractTimestampFromFields(timeFields []string, fields []logstorage.Field) (int64, error)

ExtractTimestampFromFields extracts timestamp in nanoseconds from the first field the name from timeFields at fields.

The value for the corresponding timeFields is set to empty string after returning from the function, so it could be ignored during data ingestion.

The current timestamp is returned if fields do not contain a field with timeField name or if the timeField value is empty.

func IsJSONContentType added in v1.38.0

func IsJSONContentType(ct string) bool

IsJSONContentType returns true if ct is JSON content-type.

func SetLogRowsStorage

func SetLogRowsStorage(storage LogRowsStorage)

SetLogRowsStorage sets the storage for writing data to via LogMessageProcessor.

This function must be called before using LogMessageProcessor and CanWriteData from this package.

Types

type BenchmarkLogMessageProcessor

type BenchmarkLogMessageProcessor struct{}

BenchmarkLogMessageProcessor implements LogMessageProcessor for benchmarks.

func (*BenchmarkLogMessageProcessor) AddRow

func (blp *BenchmarkLogMessageProcessor) AddRow(_ int64, _ []logstorage.Field, _ int)

AddRow implements LogMessageProcessor interface.

func (*BenchmarkLogMessageProcessor) MustClose

func (blp *BenchmarkLogMessageProcessor) MustClose()

MustClose implements LogMessageProcessor interface.

type CommonParams

type CommonParams struct {
	TenantID         logstorage.TenantID
	TimeFields       []string
	MsgFields        []string
	StreamFields     []string
	IgnoreFields     []string
	DecolorizeFields []string
	ExtraFields      []logstorage.Field

	// IsTimeFieldSet means whether the TimeFields is set **manually**.
	// The TimeFields has default value `_time`. It's not empty even if the IsTimeFieldSet is false.
	IsTimeFieldSet bool

	Debug           bool
	DebugRequestURI string
	DebugRemoteAddr string
}

CommonParams contains common HTTP parameters used by log ingestion APIs.

See https://docs.victoriametrics.com/victorialogs/data-ingestion/#http-parameters

func GetCommonParams

func GetCommonParams(r *http.Request) (*CommonParams, error)

GetCommonParams returns CommonParams from r.

func GetCommonParamsForSyslog

func GetCommonParamsForSyslog(tenantID logstorage.TenantID, streamFields, ignoreFields, decolorizeFields []string, extraFields []logstorage.Field) *CommonParams

GetCommonParamsForSyslog returns common params needed for parsing syslog messages and storing them to the given tenantID.

func (*CommonParams) NewLogMessageProcessor

func (cp *CommonParams) NewLogMessageProcessor(protocolName string, isStreamMode bool) LogMessageProcessor

NewLogMessageProcessor returns new LogMessageProcessor for the given cp.

MustClose() must be called on the returned LogMessageProcessor when it is no longer needed.

type InsertRowProcessor

type InsertRowProcessor interface {
	// AddInsertRow must add r to the underlying storage.
	AddInsertRow(r *logstorage.InsertRow)
}

InsertRowProcessor is used by native data ingestion protocol parser.

type LineReader

type LineReader struct {
	// Line contains the next line read after the call to NextLine
	//
	// The Line contents is valid until the next call to NextLine.
	Line []byte
	// contains filtered or unexported fields
}

LineReader reads newline-delimited lines from the underlying reader

func NewLineReader

func NewLineReader(name string, r io.Reader) *LineReader

NewLineReader returns LineReader for r.

func (*LineReader) Err

func (lr *LineReader) Err() error

Err returns the last error after NextLine call.

func (*LineReader) NextLine

func (lr *LineReader) NextLine() bool

NextLine reads the next line from the underlying reader.

It returns true if the next line is successfully read into Line. If the line length exceeds MaxLineSizeBytes, then this line is skipped and an empty line is returned instead.

If false is returned, then no more lines left to read from r. Check for Err in this case.

type LogMessageProcessor

type LogMessageProcessor interface {
	// AddRow must add row to the LogMessageProcessor with the given timestamp and fields.
	//
	// If streamFieldsLen >= 0, then the given number of initial fields must be used as log stream fields instead of pre-configured fields.
	//
	// The LogMessageProcessor implementation cannot hold references to fields, since the caller can reuse them.
	AddRow(timestamp int64, fields []logstorage.Field, streamFieldsLen int)

	// MustClose() must flush all the remaining fields and free up resources occupied by LogMessageProcessor.
	MustClose()
}

LogMessageProcessor is an interface for log message processors.

type LogRowsStorage

type LogRowsStorage interface {
	// MustAddRows must add lr to the underlying storage.
	MustAddRows(lr *logstorage.LogRows)

	// CanWriteData must returns non-nil error if logs cannot be added to the underlying storage.
	CanWriteData() error
}

LogRowsStorage is an interface for ingesting logs into the storage.

type TestLogMessageProcessor

type TestLogMessageProcessor struct {
	// contains filtered or unexported fields
}

TestLogMessageProcessor implements LogMessageProcessor for testing.

func (*TestLogMessageProcessor) AddRow

func (tlp *TestLogMessageProcessor) AddRow(timestamp int64, fields []logstorage.Field, streamFieldsLen int)

AddRow adds row with the given timestamp and fields to tlp

func (*TestLogMessageProcessor) MustClose

func (tlp *TestLogMessageProcessor) MustClose()

MustClose closes tlp.

func (*TestLogMessageProcessor) Verify

func (tlp *TestLogMessageProcessor) Verify(timestampsExpected []int64, resultExpected string) error

Verify verifies the number of rows, timestamps and results after AddRow calls.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL