device-injector

Device Injector Plugin

This sample plugin can be used to adjust selected container properties prior to the creation of a container. The desired adjustments happen based on pod annotations. The possible adjustments include

• injection of Linux device nodes
• injection of mounts
• injection of CDI devices
• injection of network devices
• adjustments to sysctl settings
• adjustments to Linux memory policy

Device Annotations

Devices are annotated using the devices.noderesource.dev annotation key prefix. The key devices.noderesource.dev/container.$CONTAINER_NAME annotates devices to be injected into $CONTAINER_NAME. The keys devices.noderesource.dev and devices.noderesource.dev/pod annotate devices to be injected into containers without any other, container-specific device annotations. Only one of these latter two annotations will be ever taken into account. If both are present, devices.noderesource.dev/pod is used and devices.noderesource.dev is silently ignored, otherwise devices.noderesource.dev, in the absence of additional suffix text, is processed as shorthand for the devices.noderesource.dev/pod annotation.

For compatibility with older versions of this plugin, the prefix devices.nri.io is also supported and follows identical rules. When equivalently-specific annotations are present with both the devices.noderesource.dev prefix and the devices.nri.io prefix, the annotation with the devices.noderesource.dev prefix is used and devices.nri.io is silently ignored. In all cases, the most-specific annotation is preferred regardless of prefix.

The order of precedence is as follows:

1. devices.noderesource.dev/container.$CONTAINER_NAME
2. devices.nri.io/container.$CONTAINER_NAME
3. devices.noderesource.dev/pod
4. devices.nri.io/pod
5. devices.noderesource.dev
6. devices.nri.io

The annotation value syntax for device injection is

- path: /dev/dev0
  type: {c|b}
  major: 1
  minor: 3
  file_mode: <permission mode>
  uid: <user ID>
  gid: <group ID>
- path: /dev/dev1
  ...

file_mode, uid and gid can be omitted, the rest are mandatory.

CDI Device Annotations

CDI devices are annotated in a similar manner to devices, but using the cdi-devices.noderesource.dev annotation key prefix. As with devices, the cdi-devices.nri.io annotation key prefix is also supported.

The annotation value for CDI devices is the list of CDI device names to inject.

For instance, the following annotation

metadata:
  name: bash
  annotations:
    cdi-devices.noderesource.dev/container.c0: |
      - vendor0.com/device=null
    cdi-devices.noderesource.dev/container.c1: |
      - vendor0.com/device=zero
    cdi-devices.noderesource.dev/container.c2: |
      - vendor0.com/device=dev0
      - vendor1.com/device=dev0
      - vendor1.com/device=dev1
    cdi-devices.noderesource.dev/container.mgmt: |
      - vendor0.com/device=all

requests the injection of the CDI device vendor0.com/device=null to container c0, the injection of the CDI device vendor0.com/device=zero to container c1, the injection of the CDI devices vendor0.com/device=dev0, vendor1.com/device=dev0 and vendor1.com/device=dev1 to container c2, and the injection of the CDI device vendor0.com/device=all to container mgmt.

Mount Annotations

Mounts are annotated in a similar manner to devices, but using the mounts.noderesource.dev annotation key prefix. As with devices, the mounts.nri.io annotation key prefix is also supported.

The annotation value syntax for mount injection is

  - source: <mount source0>
    destination: <mount destination0>
    type: <mount type0>
    options:
      - option0
        option1
        ...
  - source: <mount source1>
    ...

Network Device Annotations

Network devices are annotated in a similar manner to devices, but using the network-devices.noderesource.dev annotation key prefix. As with devices, the network-devices.nri.io annotation key prefix is also supported.

The annotation value for network devices is the list of host network interfaces to inject, together with their container interface names

For instance, the following annotation

metadata:
  name: net-dev-test
  annotations:
    network-devices.noderesource.dev/container.c0: |
      - hostIf: ens2.100
        Name: netdev0
    network-devices.noderesource.dev/container.c1: |
      - hostIf: ens2.101
        Name: netdev1

requests the injection of the host network interface ens2.100 into container c0 as the network interface netdev0, and the host network interface ens2.101 into container c1 as the network interface netdev1.

Sysctl Annotations

Sysctl settings are annotated in a similar manner to devices, but using the sysctl.noderesource.dev annotation key prefix.

The annotation value for sysctls is a map of sysctl keys to adjusted values. For instance, the following annotation

metadata:
  name: sysctl-test
  annotations:
    sysctl.noderesource.dev/container.c0: |
      net.core.somaxconn: "256"

will adjust the maximum number of queued TCP connections for a socket to 256.

WARNING: Note that many of the sysctl settings are effective for the full pod instead of a single container. Also many of the settings are not namespaced at all. Trying to adjust such a setting will result in an error and a failure to create the container.

Linux Memory Policy Annotations

Memory policies are annotated in a similar manner to devices, but using the memory-policy.noderesource.dev annotation key prefix. As with devices, the memory-policy.nri.io annotation key prefix is also supported.

The annotation value syntax for memory policy adjustment is

  mode: <policy mode, for instance MPOL_INTERLEAVE>
  nodes: <list of NUMA nodes as a string, for instance "2,3">
  flags: <list of policy flags as strings, for instance [ MPOL_F_STATIC_NODES ] >

The supported modes are:

• MPOL_DEFAULT
• MPOL_PREFERRED
• MPOL_BIND
• MPOL_INTERLEAVE
• MPOL_LOCAL
• MPOL_PREFERRED_MANY
• MPOL_WEIGHTED_INTERLEAVE

The supported flags are:

• MPOL_F_STATIC_NODES
• MPOL_F_RELATIVE_NODES
• MPOL_F_NUMA_BALANCING

See man set_mempolicy(2) for a description of what the effects are of these settings.

Deployment

The NRI repository contains minimal kustomize overlays for this plugin at contrib/kustomize/device-injector.

Deploy the latest release with:

kubectl apply -k https://github.com/containerd/nri/contrib/kustomize/device-injector

Deploy a specific release with:

RELEASE_TAG=v0.10.0
kubectl apply -k "github.com/containerd/nri/contrib/kustomize/device-injector?ref=${RELEASE_TAG}"

Deploy the latest development build from tip of the main branch with:

kubectl apply -k https://github.com/containerd/nri/contrib/kustomize/device-injector/unstable

Testing

You can test this plugin using a kubernetes cluster/node with a container runtime that has NRI support enabled. Start the plugin on the target node (device-injector -idx 10), create a pod with some annotated devices or mounts, then verify that those get injected to the containers according to the annotations. See the sample pod spec for an example.