Red Hat OpenShift - Group home

Kubernetes NMState Operator available with Red Hat OpenShift 4.10

By Gerald Hosch posted Mon November 28, 2022 07:20 AM

Written by Muhammad Adeel

What is the Kubernetes NMState Operator?

The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across the Red Hat OpenShift nodes with NMState. The Kubernetes NMState Operator provides users with functionality to configure various network interface types, DNS, and routing on cluster nodes. Additionally, the daemons on the cluster nodes periodically report on the state of each node's network interfaces to the API server.

Why do we need a Kubernetes NMState Operator?

Red Hat OpenShift is usually installed with a primary network backed by a single physical network interface card (NIC). There can be additional networks attached to Red Hat OpenShift, which can be used for a variety of reasons, including security, reliability and scalability.
With Red Hat OpenShift 4.9 it is possible to add secondary networks to Red Hat OpenShift that are hosted on additional NICs. As a Day-1 operation, it can be easily done by using the kernel command line argument. As a Day-2 operation, it is possible to assign an IP address with the machine config, but  the user has to wait until the node is restarted to get it working.

Kubernetes NMState Operator available with Red Hat OpenShift 4.10
With Red Hat OpenShift 4.10 it is possible to quickly configure the network interfaces as a Day-2 operation using the Kubernetes NMState Operator.
The Kubernetes NMState Operator is available on Red Hat OpenShift 4.10 for IBM® zSystems® and IBM LinuxONE via the standard Red Hat operator catalog on OperatorHub and can be installed like any other operator. The operator should be installed in openshift-nmstate namespace.
After installing the operator, an additional step is required to create NMState Custom Resource (CR) that is needed before the operator can be used. Once the NMState CR is created, the status of the daemon and webhook pods in openshift-nmstate namespace should look as shown in Figure 1.

Figure 1: All Kubernetes NMState pods in running state

Example: Using Kubernetes NMState Operator to configure networking on Red Hat OpenShift nodes
Kubernetes NMState Operator can be used to configure networking on Red Hat OpenShift nodes including assigning IP addresses to a NIC like OSA or RoCE on zSystems and LinuxONE. In this example, we assume that additional OSA devices are attached to a z/VM-based compute node.

How to attach additional OSA devices?

Additional OSA devices can be attached to the worker node as follows:
1. Attach OSA device to the worker-0 node
vmcp att XXXX-XXXX to zVM-UserID
2. Login to the node
    oc debug node/worker-0-node
3. Change root to use host binaries:
    chroot /host
4. Enable OSA device XXXX
     chzdev -e XXXX-XXXX

Similarly, RoCE card functional IDs (FIDs) can be attached to the worker node as follows:
Attach RoCE card FID XXX to worker-0 node
vmcp att pcif XXX to zVM-UserID
The FID will be attached and enabled in the node. There is no extra step of manual enablement(step 4 above) required in case of RoCE card.

An IP address can be assigned to the attached OSA device (encXXXX) with NMState Operator node network configuration policy as follows:

1. Create a worker-0.yaml policy file on bastion:

kind: NodeNetworkConfigurationPolicy
  name: worker0-osa1000
  nodeSelector: worker-0-node
    - name: enc1000
      description: Configuring OSA ID 1000 on worker-0
      type: ethernet
      state: up
        - ip:
          prefix-length: 24
        enabled: true

# Note: If there is any syntax error in the above policy file,
# it will still be applied on the node, but the desired result
# will not be achieved. In the above case the IP address will
# not be assigned to the node and the status of the policy will
# become Degraded.

2. Apply the policy file from bastion:

oc apply -f worker-0.yaml

3. Check the status of the applied policy:

oc get nncp

4. If the status become failed, check the status of the network policy enacted upon worker-0 node:

oc get nnce -o yaml

5. Check the Red Hat documentation mentioned in the references for a possible troubleshoot of the failed policy.

The Kubernetes NMState Operator will ensure that the desiredState mentioned in the policy file will be kept upon the node all the time. If a mismatch occurs in the desired state, the operator will try to automatically render the desired state, same as when the node is rebooted. If this fails, an error is reported.

To apply the IP addresses on other Compute nodes, a separate policy file must be created with the desired IP address. However, if an identical network configuration is required on all the nodes, like setting up a new DNS server or setting up a new route, a single policy file can be used to achieve the desired network configuration for all the nodes. Only in this case the nodeSelector should be:

  "": ""

 More information

Red Hat OpenShift Container Platform 4.11 is available, see the release notes for Red Hat OpenShift 4.11 on IBM Z and IBM LinuxONE.