foreman_exporter

foreman_exporter

Foreman Prometheus Exporter

This Prometheus exporter exposes foreman metrics.

Usage

usage: foreman_exporter --url=URL --username=USERNAME --password=PASSWORD [<flags>]


Flags:
  -h, --[no-]help                Show context-sensitive help (also try --help-long and --help-man).
      --[no-]web.disable-exporter-metrics  
                                 Exclude metrics about the exporter itself (process_*, go_*).
      --web.telemetry-path="/metrics"  
                                 Path under which to expose metrics.
      --web.prefix-path=""       Prefix path for all http requests.
      --[no-]web.systemd-socket  Use systemd socket activation listeners instead of port listeners (Linux only).
      --web.listen-address=:11111 ...  
                                 Addresses on which to expose metrics and web interface. Repeatable for multiple addresses.
      --web.config.file=""       [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication. See: https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md
      --url=URL                  Foreman url. ($FOREMAN_URL)
      --username=USERNAME        Foreman username. ($FOREMAN_USERNAME)
      --password=PASSWORD        Foreman password ($FOREMAN_PASSWORD)
      --[no-]skip-tls-verify     Foreman skip TLS verify. ($FOREMAN_SKIP_TLS_VERIFY)
      --concurrency=4            Max concurrent foreman client http request.
      --foreman.max-conns-per-host=0  
                                 Idle connections kept in the pool for the foreman host. Defaults to the concurrency (minimum 4).
      --retry-max=3              Max retries for foreman client http requests (honors the Retry-After header on rate-limit responses).
      --foreman.retry-max-wait=60s  
                                 Cap on the Retry-After delay honored on rate-limit responses (0 to honor it as-is).
      --foreman.rate-limit=0     Max foreman requests per second, retries included (0 to disable). Set it just under the server-side quota: a quota of N requests per minute is N/60 here.
      --foreman.rate-limit-burst=0  
                                 Token bucket depth for --foreman.rate-limit. Defaults to one second worth of requests.
      --limit=0                  Foreman client host limit search.
      --search=""                Foreman client host search filter.
      --timeout-offset=0.5s      Offset to subtract from Prometheus-supplied timeout.
      --[no-]collector.lock-concurrent-requests  
                                 Lock concurrent requests on collectors.
      --collector=host ...       Collector to enabled (repeatable), choices: [host, hostfact].
      --collector.host.labels-include=COLLECTOR.HOST.LABELS-INCLUDE  
                                 Host labels to include (regex).
      --collector.host.labels-exclude=COLLECTOR.HOST.LABELS-EXCLUDE  
                                 Host labels to exclude (regex).
      --collector.host.timeout=30s  
                                 Host default timeout if no request header 'X-Prometheus-Scrape-Timeout-Seconds'
      --[no-]collector.host.cache.enabled  
                                 Enable host cache, if global 'cache.enabled' is false.
      --[no-]collector.host.cache.compression  
                                 Enable host zstd cache compression for kvstore values, if global 'cache.compression' is false.
      --collector.host.cache.ttl-expires=COLLECTOR.HOST.CACHE.TTL-EXPIRES  
                                 Host cache expiration time, if omitted, inherit from 'cache.ttl-expires'.
      --collector.hostfact.search=COLLECTOR.HOSTFACT.SEARCH  
                                 Search host fact query filter.
      --collector.hostfact.include=COLLECTOR.HOSTFACT.INCLUDE  
                                 Host fact to include (regex).
      --collector.hostfact.exclude=COLLECTOR.HOSTFACT.EXCLUDE  
                                 Host fact to exclude (regex).
      --collector.hostfact.timeout=30s  
                                 Host fact default timeout if no request header 'X-Prometheus-Scrape-Timeout-Seconds'.
      --[no-]collector.hostfact.cache.enabled  
                                 Enable host fact cache, if global 'cache.enabled' is false.
      --[no-]collector.hostfact.cache.compression  
                                 Enable host fact zstd cache compression for kvstore values, if global 'cache.compression' is false.
      --collector.hostfact.cache.ttl-expires=COLLECTOR.HOSTFACT.CACHE.TTL-EXPIRES  
                                 Host fact cache expiration time, if omitted, inherit from global 'cache.ttl-expires'.
      --[no-]collector.hostfact.cache.update-on-partial  
                                 Update the host fact cache from a partial scrape (some hosts failed). Partial results are always exported; this only controls whether they are cached.
      --[no-]cache.enabled       Enable cache for all collectors.
      --cache.ttl-expires=1h     Cache Expiration time for all collectors.
      --[no-]cache.compression   Enable zstd cache compression for all collectors in kvstore.
      --[no-]ring.enabled        Enable the ring to deduplicate exported foreman metrics.
      --ring.instance-id=RING.INSTANCE-ID  
                                 Instance ID to register in the ring.
      --ring.instance-addr=RING.INSTANCE-ADDR  
                                 IP address to advertise in the ring. Default is auto-detected.
      --ring.instance-port=7946  Port to advertise in the ring.
      --ring.instance-interface-names=RING.INSTANCE-INTERFACE-NAMES  
                                 List of network interface names to look up when finding the instance IP address.
      --ring.join-members=RING.JOIN-MEMBERS  
                                 Other cluster members to join.
      --log.level=info           Only log messages with the given severity or above. One of: [debug, info, warn, error]
      --log.format=logfmt        Output format of log messages. One of: [logfmt, json]
      --[no-]version             Show application version.

Metrics Exposed

Exporter metrics

This endpoint return metrics about exporter itself and foreman client requests.

# HELP foreman_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which foreman_exporter was built, and the goos and goarch for the build.
# TYPE foreman_exporter_build_info gauge
foreman_exporter_build_info{branch="feat/handle_scrape_timeout",goarch="amd64",goos="linux",goversion="go1.21.1",revision="7059cdd4062a29a53cc43225c23061c3b9750aac",tags="unknown",version="0.0.5-2-g7059cdd-dirty"} 1
# HELP foreman_exporter_client_in_flight_requests A gauge of all in-flight requests for the foreman client.
# TYPE foreman_exporter_client_in_flight_requests gauge
foreman_exporter_client_in_flight_requests 0
# HELP foreman_exporter_client_request_duration_seconds A histogram of all request latencies from the foreman client.
# TYPE foreman_exporter_client_request_duration_seconds histogram
foreman_exporter_client_request_duration_seconds_bucket{le="0.005"} 0
foreman_exporter_client_request_duration_seconds_bucket{le="0.01"} 0
foreman_exporter_client_request_duration_seconds_bucket{le="0.025"} 0
foreman_exporter_client_request_duration_seconds_bucket{le="0.05"} 0
foreman_exporter_client_request_duration_seconds_bucket{le="0.1"} 0
foreman_exporter_client_request_duration_seconds_bucket{le="0.25"} 14
foreman_exporter_client_request_duration_seconds_bucket{le="0.5"} 72
foreman_exporter_client_request_duration_seconds_bucket{le="1"} 73
foreman_exporter_client_request_duration_seconds_bucket{le="2.5"} 74
foreman_exporter_client_request_duration_seconds_bucket{le="5"} 74
foreman_exporter_client_request_duration_seconds_bucket{le="10"} 74
foreman_exporter_client_request_duration_seconds_bucket{le="+Inf"} 74
foreman_exporter_client_request_duration_seconds_sum 26.012748153
foreman_exporter_client_request_duration_seconds_count 74
# HELP foreman_exporter_client_requests_total A counter for all requests from the foreman client.
# TYPE foreman_exporter_client_requests_total counter
foreman_exporter_client_requests_total{code="200",method="get"} 74
# HELP foreman_exporter_client_retry_after_seconds A histogram of Retry-After delays honored from foreman rate-limit responses.
# TYPE foreman_exporter_client_retry_after_seconds histogram
foreman_exporter_client_retry_after_seconds_sum{status="429"} 4
foreman_exporter_client_retry_after_seconds_count{status="429"} 2
# HELP foreman_exporter_client_rate_limit_requests_per_second The client-side rate limit currently applied to foreman requests, 0 when disabled.
# TYPE foreman_exporter_client_rate_limit_requests_per_second gauge
foreman_exporter_client_rate_limit_requests_per_second 16
# HELP foreman_exporter_client_rate_limit_delayed_requests_total A counter of foreman client requests that were held back by the client-side rate limiter.
# TYPE foreman_exporter_client_rate_limit_delayed_requests_total counter
foreman_exporter_client_rate_limit_delayed_requests_total 7861
# HELP foreman_exporter_client_rate_limit_wait_seconds_total Cumulative time foreman client requests spent waiting on the client-side rate limiter.
# TYPE foreman_exporter_client_rate_limit_wait_seconds_total counter
foreman_exporter_client_rate_limit_wait_seconds_total 18952.3

--foreman.rate-limit is expressed in requests per second, while server-side quotas are usually stated per minute: a quota of 1000 requests per minute is --foreman.rate-limit=16. It paces retries too, and the time spent waiting for a token is deliberately excluded from foreman_exporter_client_request_duration_seconds and from foreman_exporter_client_in_flight_requests, so those two keep measuring foreman rather than the exporter's own throttling.

Foreman hosts status

Enabled by default.

This collector return metrics to a dedicated endpoint /host-metrics.

# HELP foreman_exporter_host_status_info Foreman host status
# TYPE foreman_exporter_host_status_info gauge
foreman_exporter_host_status_info{build_status="Installed",configuration_status="Active",global_status="OK",name="server.example.com",organization="example"} 1

If the memory cache is enabled and the cache has expired it is possible to use it even if foreman api is not available (network outage, service restart, slow response...). This could prevent hole in metrics scrapping and alerts flapping. To use it, just pass the uri param expired-cache=true in scrape config or curl cmd.

curl http://localhost:11111/host-metrics?expired-cache=true

If the memory cache is enabled, it is possible to force cache regeneration with the param cache=false.

curl http://localhost:11111/host-metrics?cache=false

The following metrics have been added:

# HELP foreman_exporter_host_scrape_timeout 1 if timeout occurs, 0 otherwise
# TYPE foreman_exporter_host_scrape_timeout gauge
foreman_exporter_host_scrape_timeout 1
# HELP foreman_exporter_host_use_expired_cache 1 if using expired cache, 0 otherwise
# TYPE foreman_exporter_host_use_expired_cache gauge
foreman_exporter_host_use_expired_cache 1

Foreman hosts facts

Enable this collector with the flag --collector=hostfact.

This collector return metrics to a dedicated endpoint /host-facts-metrics.

Foreman hosts facts could render big metrics labels and must be used with the following flags to reduce the number of labels (labels cardinality):

• --collector.hostfact.search=: a foreman query to filter http facts response
• --collector.hostfact.include=: a regex to filter facts to include as labels
• --collector.hostfact.exclude=: a regex to filter facts to exclude as labels

As foreman host facts collector metrics could return many metrics (depending of foreman hosts number) and labels doesn't change a lot, a memory cache could be enabled.

# HELP foreman_exporter_host_facts_info Foreman host facts
# TYPE foreman_exporter_host_facts_info gauge
foreman_exporter_host_facts_info{name="server.example.com", operatingsystem="RedHat",operatingsystemmajrelease="9",operatingsystemrelease="9.2"} 1

If the memory cache is enabled and the cache has expired it is possible to use it even if foreman api is not available (network outage, service restart, slow response...). This could prevent hole in metrics scrapping and alerts flapping. To use it, just pass the uri param expired-cache=true in scrape config or curl cmd.

curl http://localhost:11111/host-facts-metrics?expired-cache=true

If the memory cache is enabled, it is possible to force cache regeneration with the param cache=false.

curl http://localhost:11111/host-facts-metrics?cache=false

The following metrics have been added:

# HELP foreman_exporter_host_facts_scrape_timeout 1 if timeout occurs, 0 otherwise
# TYPE foreman_exporter_host_facts_scrape_timeout gauge
foreman_exporter_host_facts_scrape_timeout 1
# HELP foreman_exporter_host_facts_use_expired_cache 1 if using expired cache, 0 otherwise
# TYPE foreman_exporter_host_facts_use_expired_cache gauge
foreman_exporter_host_facts_use_expired_cache 1

Partial results

A collector scrape can fail for some hosts and succeed for the rest: a few hosts rate-limited by foreman, or one page of the host list that did not come back. Those partial results are exported, rather than the whole scrape being discarded because of a single failure.

Two consequences for anything alerting on these collectors:

• foreman_exporter_host_scrape_error and foreman_exporter_host_facts_scrape_error going to 1 no longer implies that the series disappeared. They mean some hosts are missing, not all. Alert on the series count too if you need to catch a total outage.
• The host list is paginated on the subtotal foreman reports for the search, so a short read is detected: collecting fewer hosts than announced also raises foreman_exporter_host_scrape_error, and the hosts that were collected are still exported.

The cache is deliberately more conservative than the exported metrics. The host collector never caches an incomplete list, and the host fact collector only caches a partial scrape when --collector.hostfact.cache.update-on-partial is set — otherwise a single bad run would degrade the cache for a whole TTL.

# HELP foreman_exporter_host_facts_scrape_error 1 if there was an error, 0 otherwise
# TYPE foreman_exporter_host_facts_scrape_error gauge
foreman_exporter_host_facts_scrape_error 0
# HELP foreman_exporter_host_facts_scrape_duration_seconds Duration of the last completed host facts collector scrape of foreman.
# TYPE foreman_exporter_host_facts_scrape_duration_seconds gauge
foreman_exporter_host_facts_scrape_duration_seconds 527.4

*_scrape_duration_seconds is updated on every outcome, including a scrape that was slow and failed, which is usually the one worth measuring.

HA with memberlist

This exporter could be run in cluster mode with memberlist.

To enable cluster mode, use the following flags:

      --ring.instance-id=RING.INSTANCE-ID  
                                 Instance ID to register in the ring.
      --ring.instance-addr=RING.INSTANCE-ADDR  
                                 IP address to advertise in the ring. Default is auto-detected.
      --ring.instance-port=7946  Port to advertise in the ring.
      --ring.instance-interface-names=RING.INSTANCE-INTERFACE-NAMES  
                                 List of network interface names to look up when finding the instance IP address.
      --ring.join-members=RING.JOIN-MEMBERS  
                                 Other cluster members to join.

One instance of the ring is elected to be the leader and this is the only one which will make request to foreman and export metrics.

If the leader instance goes down, another one will be elected and will start to export metrics.

Each instance exposes the role it currently holds, resolved at scrape time:

# HELP foreman_exporter_node_role Node role, 0 = unknown, 1 = leader, 2 = follower
# TYPE foreman_exporter_node_role gauge
foreman_exporter_node_role 1

A ring that cannot be resolved reports 0 (and increments foreman_exporter_ring_leader_lookup_errors_total) rather than defaulting to follower, so a cluster left without a leader is visible. count(foreman_exporter_node_role == 1) alerts on 0 (no leader) and on > 1 (split ring).

With this config, it is easy to configure a prometheus agent to scrape the exporter metrics and avoid duplication.

If the foreman host facts collector metrics is enabled with the cache option, the cache is stored in the memberlist kvstore and replicated to all ring instances.

      --[no-]cache.enabled       Enable cache
      --cache.ttl-expires=1h     Cache Expiration time
      --[no-]cache.compression   Enable zstd compression for kvstore values

TLS and basic authentication

Foreman Exporter supports TLS and basic authentication. This enables better control of the various HTTP endpoints.

To use TLS and/or basic authentication, you need to pass a configuration file using the --web.config.file parameter. The format of the file is described in the exporter-toolkit repository.

Sources

• Foreman api
• Hashicorp Memberlist
• Grafana Distributed systems kit
• Grafana Mimir Override exporter