cluster-operator

StorageOS cluster-operator

The StorageOS Cluster Operator deploys and configures a StorageOS cluster on Kubernetes.

For quick installation of cluster operator, use the cluster operator helm chart.

Pre-requisites

• Kubernetes 1.9+
• Kubernetes must be configured to allow (configured by default in 1.10+):
  • Privileged mode containers (enabled by default)
  • Feature gate: MountPropagation=true. This can be done by appending --feature-gates MountPropagation=true to the kube-apiserver and kubelet services.

Refer to the StorageOS prerequisites docs for more information.

Setup/Development:

1. Install operator-sdk.
2. Run operator-sdk generate k8s if there's a change in api type.
3. Build operator container with operator-sdk build storageos/cluster-operator:<tag>
4. Apply the manifests in deploy/ to install the operator
   • Apply service_account.yaml and role_binding.yaml to create a service account and to grant all the permissions.
   • Apply crds/*_storageoscluster_crd.yaml to define the custom resources.
   • Apply operator.yaml to install the operator. Change the container image in this file when installing a new operator.
   • Apply crds/*_storageoscluster_cr.yaml to create a StorageOSCluster custom resource.

NOTE: Installing StorageOS on Minikube is not currently supported due to missing kernel prerequisites.

For development, run the operator outside of the k8s cluster by running:

$ make local-run

Build operator container image:

$ make image/cluster-operator OPERATOR_IMAGE=storageos/cluster-operator:test

This builds all the components and copies the binaries into the same container.

After creating a resource, query the resource:

$ kubectl get storageoscluster
NAME                READY     STATUS    AGE
example-storageos   3/3       Running   4m

Inspect a StorageOSCluster Resource

Get all the details about the cluster:

$ kubectl describe storageoscluster/example-storageos
Name:         example-storageos
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storageos.com/v1alpha1","kind":"StorageOSCluster","metadata":{"annotations":{},"name":"example-storageos","namespace":"default"},"spec":{"...
API Version:  storageos.com/v1alpha1
Kind:         StorageOSCluster
Metadata:
  Creation Timestamp:  2018-07-21T12:57:11Z
  Generation:          1
  Resource Version:    10939030
  Self Link:           /apis/storageos.com/v1alpha1/namespaces/default/storageosclusters/example-storageos
  UID:                 955b24a4-8ce5-11e8-956a-1866da35eee2
Spec:
  Join:  test07
Status:
  Node Health Status:
  ...
  ...
  Nodes:
    test09
    test08
    test07
  Phase:  Running
  Ready:  3/3
Events:   <none>

StorageOSCluster Resource Configuration

Once the StorageOS operator is running, a StorageOS cluster can be deployed by creating a Cluster Configuration. The parameters specified in the configuration will define how StorageOS is deployed, the rest of the installation details are handled by the operator.

The following tables lists the configurable spec parameters of the StorageOSCluster custom resource and their default values.

Upgrading a StorageOS Cluster

An existing StorageOS cluster can be upgraded to a new version of StorageOS by creating an Upgrade Configuration. The cluster-operator takes care of downloading the new container image and updating all the nodes with new version of StorageOS. An example of of StorageOSUpgrade resource is storageos_v1alpha1_storageosupgrade_cr.yaml.

Only offline upgrade is supported for now by cluster-operator. During the upgrade, the applications that use StorageOS volumes are scaled down and the whole StorageOS cluster is restarted with a new version. Once the StorageOS cluster becomes usable, the applications are scaled up to their previous configuration.

Once an upgrade resource is created, events related to the upgrade can be viewed in the upgrade object description. All the status and errors, if any, encountered during the upgrade are posted as events.

$ kubectl describe storageosupgrades example-storageosupgrade
Name:         example-storageosupgrade
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storageos.com/v1alpha1","kind":"StorageOSUpgrade","metadata":{"annotations":{},"name":"example-storageosupgrade","namespace":"default"},...
API Version:  storageos.com/v1alpha1
Kind:         StorageOSUpgrade
...
Spec:
  New Image:  storageos/node:1.0.0
Events:
  Type    Reason           Age   From                Message
  ----    ------           ----  ----                -------
  Normal  UpgradeComplete  21s   storageos-upgrader  StorageOS upgraded to storageos/node:1.0.0

StorageOSUpgrade Resource Configuration

The following table lists the configurable spec parameters of the StorageOSUpgrade custom resource and their default values.

TLS Support

To enable TLS, ensure that an ingress controller is installed in the cluster. Set ingress.enable and ingress.tls to true. Store the TLS cert and key as part of the storageos secret as:

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
...
...
data:
  # echo -n '<secret>' | base64
  ...
  ...
  # Add base64 encoded TLS cert and key.
  tls.crt:
  tls.key:

CSI

StorageOS also supports the Container Storage Interface (CSI) to communicate with Kubernetes. Only versions 1.10+ are supported. CSI ensures forward compatibility with future releases of Kubernetes, as vendor-specific drivers will soon be deprecated from Kubernetes. However, some functionalities are not supported yet.

To enable CSI, set csi.enable to true in the StorageOSCluster resource config.

apiVersion: "storageos.com/v1alpha1"
kind: "StorageOSCluster"
metadata:
  name: "example-storageos"
  namespace: "default"
spec:
  secretRefName: "storageos-api"
  secretRefNamespace: "default"
  csi:
    enable: true

CSI Credentials

To enable CSI Credentials, ensure that CSI is enabled by setting csi.enable to true. Based on the type of credentials to enable, set the csi fields to true:

apiVersion: "storageos.com/v1alpha1"
kind: "StorageOSCluster"
metadata:
  name: "example-storageos"
  namespace: "default"
spec:
  ...
  ...
  csi:
    enable: true
    enableProvisionCreds: true
    enableControllerPublishCreds: true
    enableNodePublishCreds: true
  ...

Specify the CSI credentials as part of the storageos secret object as:

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
...
...
data:
  # echo -n '<secret>' | base64
  ...
  ...
  csiProvisionUsername:
  csiProvisionPassword:
  csiControllerPublishUsername:
  csiControllerPublishPassword:
  csiNodePublishUsername:
  csiNodePublishPassword: