numaresources-operator

NUMA Resources Operator

Operator to allow to expose the per-NUMA-zone compute resources, using the RTE - resource topology exporter. The operator also takes care of deploying the Node Resource Topology API on which the resource topology exporter depends to provide the data. The operator provides minimal support to deploy secondary schedulers.

current limitations

• the NUMA-aware scheduling stack does not yet support the "container" topology manager scope
• the NUMA-aware scheduling stack does not yet support alignment of memory resources

Please check the issues section for the known issues and limitations of the NUMA resources operator.

running the e2e suite against your cluster

The NUMA resources operator comes with a growing e2e suite to validate components of the stack (operator proper, RTE) as well as the NUMA aware scheduling as a whole. Pre-built container images including the suites are available. There is no support for these e2e tests images, and they are recommended to be used only for development/CI purposes.

The e2e suite can optionally setup/teardown the numaresources stack, but the suite expects a pre-configured kubeletconfig. Please find here an example of recommended kubeletconfig.

The e2e suite can set a default kubeletconfig, but this is not recommended. The recommended flow is to pre-configure the cluster with a kubeletconfig object. Should you decide to use the default kubeletconfig, please omit the -e E2E_NROP_INSTALL_SKIP_KC=true from all the podman command lines below.

The e2e suite assumes the cluster has the numaresources operator installed, but with no configuration. To install the numaresources operator, you can use the vehicle which best suits your use case (OLM, make deploy...).

To actually run the tests and the tests only, assuming a pre-configured numaresources stack

podman run -ti -v $KUBECONFIG:/kubeconfig:z -e KUBECONFIG=/kubeconfig -e E2E_NROP_INSTALL_SKIP_KC=true quay.io/openshift-kni/numaresources-operator-tests:4.11.999-snapshot

To setup the stack from scratch and then run the tests (you may want to do that with ephemeral CI clusters)

podman run -ti -v $KUBECONFIG:/kubeconfig:z -e KUBECONFIG=/kubeconfig -e E2E_NROP_INSTALL_SKIP_KC=true quay.io/openshift-kni/numaresources-operator-tests:4.11.999-snapshot --setup

To setup the stack, run the tests and then restore the pristine cluster state:

podman run -ti -v $KUBECONFIG:/kubeconfig:z -e KUBECONFIG=/kubeconfig -e E2E_NROP_INSTALL_SKIP_KC=true quay.io/openshift-kni/numaresources-operator-tests:4.11.999-snapshot --setup --teardown

avoiding dependencies on other images

The E2E suite depends on few extra images. These images are very stable, lightweight and little concern most of times:

• quay.io/openshift-kni/numacell-device-plugin:test-ci
• gcr.io/google_containers/pause-amd64:3.0

However, in some cases it may be unpractical to depend on third party images. The E2E test image can act as replacement for all its dependencies, providing either the same code or replacements suitables for its use case. TO replace the dependencies, you need to set up some environment variables:

export E2E_IMAGE_URL=quay.io/openshift-kni/numaresources-operator-tests:4.11.999-snapshot
podman run -ti \
	-v $KUBECONFIG:/kubeconfig:z \
	-e KUBECONFIG=/kubeconfig \
	-e E2E_NROP_INSTALL_SKIP_KC=true \
	-e E2E_NUMACELL_DEVICE_PLUGIN_URL=${E2E_IMAGE_URL} \
	-e E2E_PAUSE_IMAGE_URL=${E2E_IMAGE_URL} \
	${E2E_IMAGE_URL}