logestimator

package
v0.58.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 8, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type CachedStructuredLogEstimator

type CachedStructuredLogEstimator struct {
	// contains filtered or unexported fields
}

CachedStructuredLogEstimator manages cached log volume estimates, singleflight in-flight deduplication, and cancellation of outdated in-flight estimation queries per task slot.

func NewCachedStructuredLogEstimator

func NewCachedStructuredLogEstimator() *CachedStructuredLogEstimator

NewCachedStructuredLogEstimator creates a new CachedStructuredLogEstimator instance.

func (*CachedStructuredLogEstimator) Close

func (e *CachedStructuredLogEstimator) Close()

Close cancels all active in-flight requests and clears the estimation cache.

func (*CachedStructuredLogEstimator) EstimateWithTaskSlot

func (e *CachedStructuredLogEstimator) EstimateWithTaskSlot(
	ctx context.Context,
	taskSlotKey string,
	container googlecloud.ResourceContainer,
	query *StructuredLogQuery,
	startTime, endTime time.Time,
	provider EstimatorProvider,
) (*EstimateResult, error)

EstimateWithTaskSlot estimates the log volume for the given query, returning cached results when available. If a query for the same input is already in flight, it deduplicates the request and waits for the shared result. If a previous in-flight estimation for taskSlotKey has a different query or time range, it cancels the previous request.

func (*CachedStructuredLogEstimator) EstimateWithTaskSlotNonBlocking

func (e *CachedStructuredLogEstimator) EstimateWithTaskSlotNonBlocking(
	ctx context.Context,
	taskSlotKey string,
	container googlecloud.ResourceContainer,
	query *StructuredLogQuery,
	startTime, endTime time.Time,
	provider EstimatorProvider,
) (*EstimateResult, bool, error)

EstimateWithTaskSlotNonBlocking estimates the log volume for the given query asynchronously without blocking the caller. It returns (res, false, nil) if the result is already cached. It returns (nil, true, nil) if the estimation is currently in-flight (pending). It returns (nil, false, err) if an error occurred during estimation.

type EstimateResult

type EstimateResult struct {
	// MetricCount is the base count obtained from Cloud Monitoring metric.
	MetricCount int64 `json:"metricCount"`

	// EstimatedCount is the estimated total count after applying custom filters.
	EstimatedCount int64 `json:"estimatedCount"`

	// CustomFilterRatio is the ratio r (0.0 to 1.0) applied to MetricCount.
	CustomFilterRatio float64 `json:"customFilterRatio"`

	// IsExact is true if no custom filter ratio estimation was required.
	IsExact bool `json:"isExact"`

	// Preset indicates a rough estimation tier if defined on the query.
	Preset EstimatedCountPreset `json:"preset,omitempty"`
}

EstimateResult represents the volume estimate.

type EstimatedCountPreset

type EstimatedCountPreset int

EstimatedCountPreset represents a predefined rough estimation tier.

const (
	// EstimatedCountPresetNone indicates that standard numeric estimation should be performed.
	EstimatedCountPresetNone EstimatedCountPreset = iota

	// EstimatedCountPresetFew indicates that the query matches only a small number of logs.
	EstimatedCountPresetFew
)

func (EstimatedCountPreset) String

func (p EstimatedCountPreset) String() string

String returns the string representation of the preset.

type EstimatorProvider

type EstimatorProvider func(ctx context.Context, container googlecloud.ResourceContainer) (*StructuredLogEstimator, error)

EstimatorProvider is a factory function to construct a StructuredLogEstimator for a container.

type LogProbeClient

type LogProbeClient interface {
	FetchProbe(ctx context.Context, filter string, pageSize int32) ([]*loggingpb.LogEntry, error)
}

LogProbeClient is an interface for probing Cloud Logging.

type LogSamplingProbeFetcher

type LogSamplingProbeFetcher interface {
	// ProbeLogTimestamps returns timestamps of matching log entries (up to maxEntries, ordered by timestamp desc).
	ProbeLogTimestamps(ctx context.Context, container googlecloud.ResourceContainer, filter string, maxEntries int32) ([]time.Time, error)
}

LogSamplingProbeFetcher is an interface for probing Cloud Logging sample logs.

type LoggingClientProbeFetcher

type LoggingClientProbeFetcher struct {
	Client *logging.Client
}

LoggingClientProbeFetcher implements LogSamplingProbeFetcher using *logging.Client.

func (*LoggingClientProbeFetcher) ProbeLogTimestamps

func (f *LoggingClientProbeFetcher) ProbeLogTimestamps(ctx context.Context, container googlecloud.ResourceContainer, filter string, maxEntries int32) ([]time.Time, error)

ProbeLogTimestamps queries recent matching log entries from Cloud Logging to extract timestamps.

type LoggingMonitoringMatcher

type LoggingMonitoringMatcher interface {
	// SupportMetrics returns true if this filter condition can be natively represented in Cloud Monitoring.
	SupportMetrics() bool

	// ToLoggingQuery returns the filter fragment for Cloud Logging queries.
	ToLoggingQuery() string

	// ToMonitoringFilter returns the filter fragment for Cloud Monitoring metric filters.
	ToMonitoringFilter() string
}

LoggingMonitoringMatcher defines a self-contained filter clause for both Cloud Logging query strings and Cloud Monitoring metric filters.

func Comment

func Comment(comment string) LoggingMonitoringMatcher

Comment creates a LoggingMonitoringMatcher for a standalone comment-only line. The comment text is formatted as "-- <comment>". Since comments do not filter log entries, SupportMetrics returns true and ToMonitoringFilter returns an empty string.

func CustomFilter

func CustomFilter(rawQuery string) LoggingMonitoringMatcher

CustomFilter creates a LoggingMonitoringMatcher for payload/custom filters. Cloud Monitoring does not support payload filters natively, so SupportMetrics returns false.

func LogID

LogID creates a LoggingMonitoringMatcher for log IDs (LOG_ID macro in Logging, metric.labels.log in Monitoring).

func ResourceLabel

func ResourceLabel(labelKey string, vm ValueMatcher) LoggingMonitoringMatcher

ResourceLabel creates a LoggingMonitoringMatcher for a resource.labels.<key> field.

func WithComment

func WithComment(matcher LoggingMonitoringMatcher, comment string) LoggingMonitoringMatcher

WithComment wraps an existing LoggingMonitoringMatcher to append a trailing comment (" -- <comment>") to its Cloud Logging query representation. Cloud Monitoring metric evaluation is delegated to the wrapped matcher without appending the comment.

type MetricLogCountFetcher

type MetricLogCountFetcher interface {
	QueryMetricCount(ctx context.Context, container googlecloud.ResourceContainer, metricFilter string, startTime, endTime time.Time) (int64, error)
}

MetricLogCountFetcher is a function or client capable of querying Cloud Monitoring for log entry counts.

type MonitoringClientFetcher

type MonitoringClientFetcher struct {
	Client *monitoring.MetricClient
}

MonitoringClientFetcher implements MetricLogCountFetcher using *monitoring.MetricClient.

func (*MonitoringClientFetcher) QueryMetricCount

func (f *MonitoringClientFetcher) QueryMetricCount(ctx context.Context, container googlecloud.ResourceContainer, metricFilter string, startTime, endTime time.Time) (int64, error)

QueryMetricCount queries Cloud Monitoring for logging.googleapis.com/log_entry_count.

type StructuredLogEstimator

type StructuredLogEstimator struct {
	MetricFetcher      MetricLogCountFetcher
	ProbeFetcher       LogSamplingProbeFetcher
	CallOptionInjector *googlecloud.CallOptionInjector
	TargetSamples      int32
}

StructuredLogEstimator estimates log volume for StructuredLogQuery.

func NewStructuredLogEstimator

func NewStructuredLogEstimator(metricFetcher MetricLogCountFetcher, probeFetcher LogSamplingProbeFetcher, callOptionInjector *googlecloud.CallOptionInjector) *StructuredLogEstimator

NewStructuredLogEstimator creates a StructuredLogEstimator with defaults.

func NewStructuredLogEstimatorFromClients

func NewStructuredLogEstimatorFromClients(loggingClient *logging.Client, metricClient *monitoring.MetricClient, callOptionInjector *googlecloud.CallOptionInjector) *StructuredLogEstimator

NewStructuredLogEstimatorFromClients constructs a StructuredLogEstimator directly from Google Cloud SDK clients.

func (*StructuredLogEstimator) Estimate

func (e *StructuredLogEstimator) Estimate(ctx context.Context, container googlecloud.ResourceContainer, query *StructuredLogQuery, startTime, endTime time.Time) (*EstimateResult, error)

Estimate estimates the total log volume for the given StructuredLogQuery over the time interval. Cloud Monitoring queries for all resource types are executed concurrently. If custom filters require sampling, a time-window bounded sampling probe is executed.

type StructuredLogQuery

type StructuredLogQuery struct {
	// ResourceTypes specifies the resource.type strings (e.g. ["k8s_container"] or ["gke_cluster", "gke_nodepool"]).
	ResourceTypes []string

	// IgnoreMetricsResourceType specifies the resource.type strings that should NOT be queried via Cloud Monitoring metrics
	// (e.g. resource types like "gke_nodepool" that are valid in Cloud Logging but do not exist as MonitoredResourceDescriptors in Cloud Monitoring).
	IgnoreMetricsResourceType []string

	// Filters are the individual filter matchers for this query.
	Filters []LoggingMonitoringMatcher

	// Incomplete indicates that the query is missing required parameters (e.g. ProjectID or ClusterName)
	// and should not trigger log volume estimation.
	Incomplete bool

	// Preset specifies a predefined rough estimation tier (e.g. EstimatedCountPresetFew)
	// that bypasses Cloud Monitoring and probe queries.
	Preset EstimatedCountPreset
}

StructuredLogQuery represents a decomposed Cloud Logging and Cloud Monitoring query.

func (*StructuredLogQuery) AllFiltersSupportMetrics

func (q *StructuredLogQuery) AllFiltersSupportMetrics() bool

AllFiltersSupportMetrics returns true if all matchers in Filters can be natively evaluated in Cloud Monitoring and no resource types are excluded from metrics querying.

func (*StructuredLogQuery) GenerateBaseLoggingQuery

func (q *StructuredLogQuery) GenerateBaseLoggingQuery() string

GenerateBaseLoggingQuery generates the Cloud Logging filter string containing only matchers supported by Cloud Monitoring.

func (*StructuredLogQuery) GenerateCloudLoggingQuery

func (q *StructuredLogQuery) GenerateCloudLoggingQuery() string

GenerateCloudLoggingQuery generates the Cloud Logging filter string.

func (*StructuredLogQuery) GenerateMonitoringMetricFilters

func (q *StructuredLogQuery) GenerateMonitoringMetricFilters() []string

GenerateMonitoringMetricFilters generates the filter strings for logging.googleapis.com/log_entry_count in Cloud Monitoring (one per resource type).

type ValueMatcher

type ValueMatcher interface {
	// ToLoggingFieldQuery returns the Cloud Logging query fragment for a named field.
	ToLoggingFieldQuery(fieldName string) string

	// ToMonitoringFieldFilter returns the Cloud Monitoring metric filter fragment for a named field.
	ToMonitoringFieldFilter(fieldName string) string

	// ToLoggingLogIDQuery returns the Cloud Logging query fragment for LOG_ID(...).
	ToLoggingLogIDQuery() string

	// ToMonitoringLogIDFilter returns the Cloud Monitoring metric filter fragment for metric.labels.log.
	ToMonitoringLogIDFilter() string
}

ValueMatcher defines how a set of values or patterns expands for both general field names (e.g. resource.labels.key) and LogID macros.

func ContainsAny

func ContainsAny(subs ...string) ValueMatcher

ContainsAny creates a ValueMatcher matching any of the substrings.

func Exact

func Exact(val string) ValueMatcher

Exact creates a ValueMatcher for an exact single value.

func FromSetFilter

func FromSetFilter(filter *gcpqueryutil.SetFilterParseResult, isSubstring bool) ValueMatcher

FromSetFilter converts a KHI SetFilterParseResult into a ValueMatcher.

func NoneOf

func NoneOf(vals ...string) ValueMatcher

NoneOf creates a ValueMatcher that excludes all provided values (NOT).

func NotContainsAny

func NotContainsAny(subs ...string) ValueMatcher

NotContainsAny creates a ValueMatcher excluding all of the substrings.

func OneOf

func OneOf(vals ...string) ValueMatcher

OneOf creates a ValueMatcher that matches any of the provided values (OR).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL