servergroup

package
v0.0.96 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 5, 2026 License: MIT Imports: 30 Imported by: 4

Documentation

Index

Constants

View Source
const (
	// PathPrefixLabel is the name of the label that holds the path prefix for a scrape target.
	PathPrefixLabel = "__path_prefix__"
)

Variables

View Source
var (
	// DefaultConfig is the Default base promxy configuration
	DefaultConfig = Config{
		AntiAffinity:        time.Second * 10,
		Scheme:              "http",
		RemoteReadPath:      "api/v1/read",
		Timeout:             0,
		MaxIdleConns:        20000,
		MaxIdleConnsPerHost: 1000,
		IdleConnTimeout:     5 * time.Minute,
		PreferMax:           false,
		HTTPConfig: HTTPClientConfig{
			DialTimeout: time.Millisecond * 200,
		},
	}
)
View Source
var DiscoveryUpdateInterval = 5 * time.Second

DiscoveryUpdateInterval controls how often each server group's discovery manager coalesces and applies target updates. Prometheus defaults this to 5s to throttle chatty service-discovery mechanisms; promxy keeps that default so production behavior is unchanged. It is exposed as a var mainly so tests can drive it low — the discovery manager only emits its first target set (and thus lets the server group become Ready) on the first tick of this interval, so a 5s value adds ~5s of startup latency to every proxy the tests spin up. Must be > 0 (time.NewTicker panics on zero).

Functions

This section is empty.

Types

type AbsoluteTimeRangeConfig

type AbsoluteTimeRangeConfig struct {
	Start    time.Time `yaml:"start"`
	End      time.Time `yaml:"end"`
	Truncate bool      `yaml:"truncate"`
}

AbsoluteTimeRangeConfig contains absolute times to define a servergroup's time range

func (*AbsoluteTimeRangeConfig) UnmarshalYAML

func (tr *AbsoluteTimeRangeConfig) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Config

type Config struct {
	Ordinal int `yaml:"-"`
	// Name is an optional human-readable identifier for this server group.
	Name string `yaml:"name,omitempty"`
	// RemoteRead directs promxy to load RAW data (meaning matrix selectors such as `foo[1h]`)
	// through the RemoteRead API on prom.
	// Pros:
	//  - StaleNaNs work
	//  - ~2x faster (in my local testing, more so if you are using default JSON marshaler in prom)
	//
	// Cons:
	//  - proto marshaling prom side doesn't stream, so the data being sent
	//      over the wire will be 2x its size in memory on the remote prom host.
	//  - "experimental" API (according to docs) -- meaning this might break
	//      without much (if any) warning
	//
	// Upstream prom added a StaleNan to determine if a given timeseries has gone
	// NaN -- the problem being that for range vectors they filter out all "stale" samples
	// meaning that it isn't possible to get a "raw" dump of data through the query/query_range v1 API
	// The only option that exists in reality is the "remote read" API -- which suffers
	// from the same memory-balooning problems that the HTTP+JSON API originally had.
	// It has **less** of a problem (its 2x memory instead of 14x) so it is a viable option.
	RemoteRead bool `yaml:"remote_read"`
	// RemoteReadPath sets the remote read path for the hosts in this servergroup
	RemoteReadPath string `yaml:"remote_read_path"`

	// NativeHistogram tunes how promxy serves native-histogram queries
	// against this server group. The zero value (no `native_histogram`
	// block in YAML) is "AST-only detection, fail loud if remote_read
	// can't preserve fidelity" — the safe default. See NativeHistogramConfig.
	NativeHistogram NativeHistogramConfig `yaml:"native_histogram,omitempty"`
	// HTTP client config for promxy to use when connecting to the various server_groups
	// this is the same config as prometheus
	HTTPConfig HTTPClientConfig `yaml:"http_client"`
	// Scheme defines how promxy talks to this server group (http, https, etc.)
	Scheme string `yaml:"scheme"`
	// Labels is a set of labels that will be added to all metrics retrieved
	// from this server group
	Labels model.LabelSet `json:"labels"`
	// RelabelConfigs are similar in function and identical in configuration as prometheus'
	// relabel config for scrape jobs. The difference here being that the source labels
	// you can pull from are from the downstream servergroup target and the labels you are
	// relabeling are that of the timeseries being returned. This allows you to mutate the
	// labelsets returned by that target at runtime.
	// To further illustrate the difference we'll look at an example:
	//
	//      relabel_configs:
	//    - source_labels: [__meta_consul_tags]
	//      regex: '.*,prod,.*'
	//      action: keep
	//    - source_labels: [__meta_consul_dc]
	//      regex: '.+'
	//      action: replace
	//      target_label: datacenter
	//
	// If we saw this in a scrape-config we would expect:
	//   (1) the scrape would only target hosts with a prod consul label
	//   (2) it would add a label to all returned series of datacenter with the value set to whatever the value of __meat_consul_dc was.
	//
	// If we saw this same config in promxy (pointing at prometheus hosts instead of some exporter), we'd expect a similar behavior:
	//   (1) only targets with the prod consul label would be included in the servergroup
	//   (2) it would add a label to all returned series of this servergroup of datacenter with the value set to whatever the value of __meat_consul_dc was.
	//
	// So in reality its "the same", the difference is in prometheus these apply to the labels/targets of a scrape job,
	// in promxy they apply to the prometheus hosts in the servergroup - but the behavior is the same.
	RelabelConfigs []*relabel.Config `yaml:"relabel_configs,omitempty"`
	// MetricsRelabelConfigs are similar in spirit to prometheus' relabel config but quite different.
	// As this relabeling is being done within the query-path all relabel actions need to be reversible
	// so that we can alter queries (e.g. matchers) based on the relabel config. This is done by strictly
	// limiting the rewrite capability to just those subset of actions that can be reversed. Similar to
	// prometheus' relabel capability these rules are executed in an order -- so the rules can be compounded
	// to create relatively complex relabel behavior.
	// To showcase the versatility, lets look at an example:
	//
	//    metrics_relabel_configs:
	//      # this will drop the `replica` label; enabling replica deduplication
	//      # similar to thanos -- https://github.com/thanos-io/thanos/blob/master/docs/components/query.md#deduplication
	//      - action: labeldrop
	//        source_label: replica
	//      # this will replace the `job` label with `scrape_job`
	//      - action: replace
	//        source_label: job
	//        target_label: scrape_job
	//      # this will drop the label `job`.
	//      - action: labeldrop
	//        source_label: job
	//      # this will lowecase the `branch` label in-place (as source_label and target_label match)
	//      - action: lowercase
	//        source_label: branch
	//        target_label: branch
	//      # this will uppercase the `instance` label into `instanceUpper`
	//      - action: uppercase
	//        source_label: instance
	//        target_label: instanceUpper
	MetricsRelabelConfigs []*promclient.MetricRelabelConfig `yaml:"metrics_relabel_configs,omitempty"`
	// ServiceDiscoveryConfigs is a set of ServiceDiscoveryConfig options that allow promxy to discover
	// all hosts in the server_group
	ServiceDiscoveryConfigs discovery.Configs `yaml:"-"`
	// PathPrefix to prepend to all queries to hosts in this servergroup
	PathPrefix string `yaml:"path_prefix"`
	// QueryParams are a map of query params to add to all HTTP calls made to this downstream
	// the main use-case for this is to add `nocache=1` to VictoriaMetrics downstreams
	// (see https://github.com/jacksontj/promxy/issues/202)
	QueryParams map[string]string `yaml:"query_params"`
	// TODO cache this as a model.Time after unmarshal
	// AntiAffinity defines how large of a gap in the timeseries will cause promxy
	// to merge series from 2 hosts in a server_group. This required for a couple reasons
	// (1) Promxy cannot make assumptions on downstream clock-drift and
	// (2) two prometheus hosts scraping the same target may have different times
	// #2 is caused by prometheus storing the time of the scrape as the time the scrape **starts**.
	// in practice this is actually quite frequent as there are a variety of situations that
	// cause variable scrape completion time (slow exporter, serial exporter, network latency, etc.)
	// any one of these can cause the resulting data in prometheus to have the same time but in reality
	// come from different points in time. Best practice for this value is to set it to your scrape interval
	AntiAffinity time.Duration `yaml:"anti_affinity,omitempty"`

	// AntiAffinityDynamic enables per-series anti-affinity inference. When
	// true, the merge layer infers each series' buffer from the median
	// inter-sample gap of the longer side and uses AntiAffinity only as the
	// fallback when there are too few samples to estimate. This is the
	// right setting when a single server_group hosts series with mixed
	// scrape intervals (e.g. a 1m-scrape job alongside a 15s-scrape one) —
	// a single static AntiAffinity is too tight for the slow series (gaps
	// look like missing data and get gap-filled, doubling count_over_time)
	// or too wide for the fast one (legitimate fresh samples get deduped).
	// See https://github.com/jacksontj/promxy/issues/734.
	AntiAffinityDynamic bool `yaml:"anti_affinity_dynamic,omitempty"`

	// Timeout, if non-zero, specifies the amount of
	// time to wait for a server's response headers after fully
	// writing the request (including its body, if any). This
	// time does not include the time to read the response body.
	Timeout time.Duration `yaml:"timeout,omitempty"`

	// MaxIdleConns, servergroup maximum number of idle connections to keep open.
	MaxIdleConns int `yaml:"max_idle_conns,omitempty"`

	// MaxIdleConnsPerHost, servergroup maximum number of idle connections to keep open per host.
	MaxIdleConnsPerHost int `yaml:"max_idle_conns_per_host,omitempty"`

	// IdleConnTimeout, time wait to close a idle connections.
	IdleConnTimeout time.Duration `yaml:"idle_conn_timeout,omitempty"`

	// IgnoreError will hide all errors from this given servergroup effectively making
	// the responses from this servergroup "not required" for the result.
	// Note: this allows you to make the tradeoff between availability of queries and consistency of results
	IgnoreError bool `yaml:"ignore_error"`

	// DowngradeError converts all errors to warnings from this given servergroup effectively making
	// the responses from this servergroup "not required" for the result.
	// Note: this allows you to make the tradeoff between availability of queries and consistency of results
	DowngradeError bool `yaml:"downgrade_error"`

	// RelativeTimeRangeConfig defines a relative time range that this servergroup will respond to
	// An example use-case would be if a specific servergroup was long-term storage, it might only
	// have data 3d old and retain 90d of data.
	RelativeTimeRangeConfig *RelativeTimeRangeConfig `yaml:"relative_time_range"`

	// AbsoluteTimeRangeConfig defines an absolute time range that this servergroup will respond to
	// An example use-case would be if a specific servergroup was was "deprecated" and wasn't getting
	// any new data after a specific given point in time
	AbsoluteTimeRangeConfig *AbsoluteTimeRangeConfig `yaml:"absolute_time_range"`

	LabelFilterConfig *promclient.LabelFilterConfig `yaml:"label_filter"`

	// InjectMatchers is a list of label matchers that are injected into every selector
	// of every request sent to this servergroup. This effectively scopes the servergroup
	// to the subset of downstream data matching these matchers -- even for queries that
	// never reference the labels being injected (e.g. with `cluster="A"` configured,
	// `count(up)` is sent downstream as `count(up{cluster="A"})`).
	//
	// Each entry is a single matcher in promql syntax (no enclosing braces), e.g.:
	//
	//    inject_matchers:
	//      - 'cluster="A"'
	//      - 'region=~"us-.*"'
	//
	// Unlike `labels` (which only adds labels to responses) and `label_filter` (which only
	// drops queries that can't match), `inject_matchers` always adds the matchers to the
	// queries themselves. A common use-case is presenting a per-tenant view of a merged
	// downstream (e.g. a single Mimir/Thanos/Prometheus holding many clusters) -- see
	// https://github.com/jacksontj/promxy/issues/698
	// NOTE: this is not a "secure" mechanism; it only mutates the request matchers and
	// relies on the downstream honoring them.
	InjectMatchers []string `yaml:"inject_matchers,omitempty"`

	PreferMax bool `yaml:"prefer_max,omitempty"`

	// HTTPClientHeaders are a map of HTTP headers to add to remote read HTTP calls made to this downstream
	// the main use-case for this is to support the X-Scope-OrgID header required by Mimir and Cortex
	// in multi-tenancy mode
	// (see https://github.com/jacksontj/promxy/issues/643)
	HTTPClientHeaders map[string]string `yaml:"http_headers"`

	// AlignQueryRangeWithStep declares that this backend snaps query_range results
	// to the epoch step grid (k*step), as Mimir/Cortex do by default. When set,
	// promxy re-stamps the returned samples back onto the grid implied by the
	// request start (start + j*step) so they line up with promxy's local
	// evaluation grid. Without it, an unaligned request (start % step != 0) whose
	// off-grid distance exceeds the lookback-delta yields no data for this backend.
	// Leave it OFF for backends that do not step-align (e.g. vanilla Prometheus);
	// their samples already sit on the requested grid. See issue #787.
	AlignQueryRangeWithStep bool `yaml:"align_query_range_with_step,omitempty"`
}

Config is the configuration for a ServerGroup that promxy will talk to. This is where the vast majority of options exist.

func (*Config) GetAntiAffinity

func (c *Config) GetAntiAffinity() model.Time

GetAntiAffinity returns the AntiAffinity time for this servergroup

func (*Config) GetInjectMatchers added in v0.0.94

func (c *Config) GetInjectMatchers() ([]*labels.Matcher, error)

GetInjectMatchers parses the configured InjectMatchers entries into label matchers. It returns nil if no matchers are configured.

func (*Config) GetPreferMax added in v0.0.84

func (c *Config) GetPreferMax() bool

GetPreferMax returns the PreferMax setting for this servergroup

func (*Config) GetScheme

func (c *Config) GetScheme() string

GetScheme returns the scheme for this servergroup

func (*Config) MarshalYAML added in v0.0.82

func (c *Config) MarshalYAML() (interface{}, error)

MarshalYAML implements the yaml.Marshaler interface.

func (*Config) UnmarshalYAML

func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type HTTPClientConfig

type HTTPClientConfig struct {
	DialTimeout time.Duration `yaml:"dial_timeout"`
	// DialNetwork controls the address family used when dialing downstream hosts.
	// It maps to the network passed to net.Dialer: "tcp" (default) does dual-stack
	// dialing, while "tcp4"/"tcp6" force IPv4/IPv6 respectively. An empty value is
	// treated as "tcp".
	DialNetwork string `yaml:"dial_network,omitempty"`
	// FallbackDelay maps to net.Dialer.FallbackDelay and controls how long the
	// dual-stack (RFC 6555 "Happy Eyeballs") dialer waits before spawning a
	// fallback connection to the other address family. A zero value uses Go's
	// default (300ms); a negative value disables the fallback attempt entirely.
	// Note: to be effective it must be shorter than DialTimeout, otherwise the
	// dial times out before the fallback is ever attempted.
	FallbackDelay time.Duration                `yaml:"fallback_delay,omitempty"`
	HTTPConfig    config_util.HTTPClientConfig `yaml:",inline"`
	SigV4Config   *sigv4.SigV4Config           `yaml:"sigv4,omitempty"`
}

HTTPClientConfig extends prometheus' HTTPClientConfig

func (*HTTPClientConfig) DialNetworkOrDefault added in v0.0.96

func (c *HTTPClientConfig) DialNetworkOrDefault() string

DialNetworkOrDefault returns the configured dial network, defaulting to "tcp".

type NativeHistogramConfig added in v0.0.94

type NativeHistogramConfig struct {
	// MetadataRefresh controls the metric-name → type cache. When zero
	// (the default), detection is pure AST: a Call to one of the
	// histogram-only PromQL functions (histogram_avg, histogram_count,
	// histogram_sum, histogram_stddev, histogram_stdvar,
	// histogram_fraction). This misses queries that touch histogram
	// metrics without invoking those functions — e.g. `rate(my_hist[5m])`.
	//
	// Set MetadataRefresh to a positive duration to enable the cache:
	// promxy periodically calls /api/v1/metadata on this server group,
	// extracts the histogram-typed metric names, and consults the union
	// of all groups' caches when classifying queries. The cache is
	// name-keyed (typically <2% of the total metric-name space) so
	// memory is bounded by histogram-name count, not series cardinality.
	MetadataRefresh time.Duration `yaml:"metadata_refresh,omitempty"`

	// AllowLossy controls what happens when a histogram-bearing query
	// targets this server group and remote_read isn't configured.
	//
	//   false (default): the query errors out before fan-out. Wrong data
	//     is worse than no data — the operator should configure
	//     remote_read (or explicitly opt into lossy fallback) rather
	//     than silently serve histograms over the lossy JSON path.
	//
	//   true: the query proceeds via the HTTP API even though the
	//     response will be lossy for histogram samples. Use when you
	//     accept the fidelity loss (e.g. dashboards that only consume
	//     histogram_quantile output and don't care about sparse spans).
	AllowLossy bool `yaml:"allow_lossy,omitempty"`
}

NativeHistogramConfig groups the per-server-group knobs that control how promxy handles native-histogram queries. Native histograms only round-trip with full fidelity over remote_read — the HTTP-API JSON path collapses sparse spans and drops empty buckets. Promxy detects histogram-bearing queries from the PromQL AST (and optionally from a metric-name metadata cache) and skips HTTP-API pushdown for them so the embedded engine evaluates locally, fetching raw data via remote_read.

The zero value is a safe default: AST-only detection, and any histogram query that targets a server group without remote_read errors out rather than silently returning lossy data.

type RelativeTimeRangeConfig

type RelativeTimeRangeConfig struct {
	Start    *time.Duration `yaml:"start"`
	End      *time.Duration `yaml:"end"`
	Truncate bool           `yaml:"truncate"`
}

RelativeTimeRangeConfig configures durations relative from "now" to define a servergroup's time range

func (*RelativeTimeRangeConfig) UnmarshalYAML

func (tr *RelativeTimeRangeConfig) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type ServerGroup

type ServerGroup struct {
	Ready chan struct{}

	// TODO: lock/atomics on cfg and client
	Cfg *Config

	OriginalURLs []string
	// contains filtered or unexported fields
}

ServerGroup encapsulates a set of prometheus downstreams to query/aggregate

func NewServerGroup added in v0.0.77

func NewServerGroup() (*ServerGroup, error)

New creates a new servergroup

func (*ServerGroup) ApplyConfig

func (s *ServerGroup) ApplyConfig(cfg *Config) error

ApplyConfig applies new configuration to the ServerGroup TODO: move config + client into state object to be swapped with atomics

func (*ServerGroup) Cancel

func (s *ServerGroup) Cancel()

Cancel stops backround processes (e.g. discovery manager)

func (*ServerGroup) GetValue

func (s *ServerGroup) GetValue(ctx context.Context, start, end time.Time, matchers []*labels.Matcher) storage.SeriesSet

GetValue loads the raw data for a given set of matchers in the time range

func (*ServerGroup) IsHistogramMetric added in v0.0.94

func (s *ServerGroup) IsHistogramMetric(name string) bool

IsHistogramMetric reports whether the given metric name is known to be a histogram metric in this server group's most recent metadata snapshot. Returns false when the metadata cache is disabled (the default), when no fetch has succeeded yet, or when the name simply isn't a histogram.

Used by the proxy-level histogram routing decision to disable HTTP-API pushdown for queries that touch histogram metrics, even when the query doesn't invoke one of the histogram-only PromQL functions.

func (*ServerGroup) LabelNames

func (s *ServerGroup) LabelNames(ctx context.Context, matchers []string, startTime time.Time, endTime time.Time) ([]string, v1.Warnings, error)

LabelNames returns all the unique label names present in the block in sorted order.

func (*ServerGroup) LabelValues

func (s *ServerGroup) LabelValues(ctx context.Context, label string, matchers []string, startTime time.Time, endTime time.Time) (model.LabelValues, v1.Warnings, error)

LabelValues performs a query for the values of the given label.

func (*ServerGroup) Metadata added in v0.0.76

func (s *ServerGroup) Metadata(ctx context.Context, metric, limit string) (map[string][]v1.Metadata, error)

Metadata returns metadata about metrics currently scraped by the metric name.

func (*ServerGroup) Query

func (s *ServerGroup) Query(ctx context.Context, query string, ts time.Time) storage.SeriesSet

Query performs a query for the given time.

func (*ServerGroup) QueryExemplars added in v0.0.94

func (s *ServerGroup) QueryExemplars(ctx context.Context, query string, startTime, endTime time.Time) ([]v1.ExemplarQueryResult, error)

QueryExemplars performs a query for exemplars by the given query and time range.

func (*ServerGroup) QueryRange

func (s *ServerGroup) QueryRange(ctx context.Context, query string, r v1.Range) storage.SeriesSet

QueryRange performs a query for the given range.

func (*ServerGroup) RoundTrip added in v0.0.76

func (s *ServerGroup) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip allows us to intercept and mutate downstream HTTP requests at the transport level

func (*ServerGroup) Series

func (s *ServerGroup) Series(ctx context.Context, matches []string, startTime, endTime time.Time) ([]model.LabelSet, v1.Warnings, error)

Series finds series by label matchers.

func (*ServerGroup) State

func (s *ServerGroup) State() *ServerGroupState

State returns the current ServerGroupState

func (*ServerGroup) Sync

func (s *ServerGroup) Sync()

Sync updates the targets from our discovery manager

type ServerGroupState

type ServerGroupState struct {
	// Targets is the list of target URLs for this discovery round
	Targets []string
	// contains filtered or unexported fields
}

ServerGroupState encapsulates the state of a serverGroup from service discovery

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL