cluster-etcd-operator

cluster-etcd-operator

cluster-etcd-operator (CEO) is an operator that handles the scaling of etcd during cluster bootstrap and regular operation. The operator also manages provisioning etcd dependencies such as TLS certificates.

Developing the CEO

See HACKING.md.

Frequently Asked Questions

See FAQ.md.

Security Response

If you've found a security issue that you'd like to disclose confidentially please contact Red Hat's Product Security team. Details at https://access.redhat.com/security/team/contact

Telemetry queries

This operator sends some telemetry data from each OpenShift cluster, this data is used to help us engineers observe the clusters and how they behave. For full list of data that is being sent, see this file.

To query this data make sure you have access to telemetry, primary to https://infogw-proxy.api.openshift.com/. If you do not, follow https://help.datahub.redhat.com/docs/interacting-with-telemetry-data this guide.

Example queries

1. Number of etcd alerts firing.

This gives a good insight if we compared with existing alerts which alerts do not fire and could be adjusted. Or which fire too often and we should look into why.

count by (alertname) (alerts{alertname=~"etcd.+"})

1. Number of etcd slow requests by gRPC service and method.

Again this is good to adjust the new etcdGRPCRequestsSlow alert, if one fires too often on clusters, it should be looked into.

count by (grpc_method, grpc_service) (alerts{alertname="etcdGRPCRequestsSlow"})

1. Median fsync disk duration by provider.

This gives an insight which provider has slowest disks. (None is often metal.)

sum without (_id) (quantile by (_id)(0.5, instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile{quantile="0.99"}) + on(_id) group_left(provider) (topk by (_id) (1, id_provider*0)))

1. Median network peer latency by OpenShift version.

sum without (_id) (quantile by (_id)(0.5, instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile{quantile="0.99"}) + on(_id) group_left(version) (topk by (_id) (1, id_version*0)))

1. Top 100 clusters by db size.

topk(100, instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum)

1. Median disk commit duration by provider.

(None is often metal.)

sum without (_id) (quantile by (_id)(0.5, instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile{quantile="0.99"}) + on(_id) group_left(provider) (topk by (_id) (1, id_provider*0)))

Tests

This repository is compatible with the OpenShift Tests Extension (OTE) framework.

Building the test binary

make build

Running test suites and tests

# Run a specific test suite or test
./cluster-etcd-operator-tests-ext run-suite openshift/cluster-etcd-operator/all
./cluster-etcd-operator-tests-ext run-test "test-name"

# Run with JUnit output
./cluster-etcd-operator-tests-ext run-suite openshift/cluster-etcd-operator/all --junit-path=/tmp/junit-results/junit.xml
./cluster-etcd-operator-tests-ext run-test "test-name" --junit-path=/tmp/junit-results/junit.xml

Listing available tests and suites

# List all test suites
./cluster-etcd-operator-tests-ext list suites

# List tests in a suite
./cluster-etcd-operator-tests-ext list tests --suite=openshift/cluster-etcd-operator/all

#for concurrency
./cluster-etcd-operator-tests-ext run-suite openshift/cluster-etcd-operator/all -c 1

For more information about the OTE framework, see the openshift-tests-extension documentation.