Infrastructure as a Service

 View Only

MicroShift – Part 26: Raspberry Pi 4 with AlmaLinux 9

By Alexei Karve posted Mon December 12, 2022 12:05 PM

  

MicroShift with KubeVirt and Kata Containers on Raspberry Pi 4 with AlmaLinux 9 (Lime Lynx)

Introduction

MicroShift is a Red Hat-led open-source community project that is exploring how OpenShift OKD Kubernetes distribution can be optimized for small form factor devices and edge computing. Red Hat Device Edge delivers an enterprise-ready and supported distribution of MicroShift. Red Hat Device Edge is planned as a developer preview early next year and expected to be generally available with full support later in 2023.

Over the last 25 parts, we have worked with MicroShift on multiple distros of Linux on the Raspberry Pi 4 and Jetson Nano. In Part 17, we worked with MicroShift on AlmaLinux 8.5. In this Part 26, we will work with MicroShift on AlmaLinux 9. We will run an object detection sample and send messages to Node Red installed on MicroShift. Further, we will setup KubeVirt and the OKD Web Console and run Virtual Machine Instances in MicroShift. We will also create CentOS 9 Stream and Ubuntu Virtual Machines and use the Containerized Data Importer (CDI). We will run a MongoDB sample and finally finish by building and running Kata containers on MicroShift.

AlmaLinux is a free and open-source Linux distribution, created originally by CloudLinux to provide a community-supported, production-grade enterprise operating system that is binary-compatible with Red Hat Enterprise Linux. Both AlmaLinux and Rocky Linux emerged in response to Red Hat’s December 8, 2020 announcement stating that it will discontinue CentOS based on RedHat releases, instead shifting focus to CentOS Stream, which tracks just ahead of a current RHEL release.

Setting up the Raspberry Pi 4 with AlmaLinux 9

Run the following steps to download the Alma Linux image and setup the Raspberry Pi 4.

  1. Download the latest AlmaLinux 9 64-bit Arm (aarch64) for use with RPi 4 from http://repo.almalinux.org/rpi/9/images/
  2. Write to Microsdxc card using balenaEtcher or the Raspberry Pi Imager
  3. Have a Keyboard and Monitor connected to the Raspberry Pi 4
  4. Insert Microsdxc into Raspberry Pi4 and poweron
  5. Login using root/almalinux using the keyboard connected to the Raspberry Pi 4
  6. Edit the /etc/ssh/sshd_config. Add the following line and restart the ssh daemon

    PermitRootLogin yes

    systemctl restart sshd
  7. Find the ethernet dhcp ipaddress of your Raspberry Pi 4 by running the nmap on your Macbook with your subnet
    $ sudo nmap -sn 192.168.1.0/24
    
    Nmap scan report for 192.168.1.209
    Host is up (0.010s latency).
    MAC Address: E4:5F:01:2E:D8:95 (Raspberry Pi Trading)
    
  8. Now you can login from your Laptop using ssh with root/almalinux to ipaddress above. You may add your public ssh key to login without password.
    $ ssh root@$ipaddress
    [root@rpi ~]# mkdir ~/.ssh
    [root@rpi ~]# vi ~/.ssh/authorized_keys
    [root@rpi ~]# chmod 700 ~/.ssh
    [root@rpi ~]# chmod 600 ~/.ssh/authorized_keys
    [root@rpi ~]# #chcon -R -v system_u:object_r:usr_t:s0 ~/.ssh/
    

    Check that your key is RSA 2048 or larger with the following command. The RSA 1024 will not work.

    ssh-keygen -l -v -f ~/.ssh/id_rsa.pub
    

    If it is 1024, you will get the error

    [root@rpi ~]# cat /var/log/secure
    Nov 26 16:26:42 rpi sshd[3927]: refusing RSA key: Invalid key length [preauth]
    
  9. Extend the partition to maximize disk usage
    rootfs-expand
  10. Update and set the hostname with a domain and add the ipv4 address to /etc/hosts
    hostnamectl set-hostname microshift.example.com
    echo "$ipaddress microshift microshift.example.com" >> /etc/hosts
    dnf update -y
  11. Optionally, enable wifi
    nmcli device wifi list # Note your ssid
    nmcli device wifi connect $ssid --ask
  12. Check the release and file system type.
    cat /etc/os-release
    lsblk -f
    

    Output:

    [root@microshift ~]# cat /etc/os-release
    NAME="AlmaLinux"
    VERSION="9.1 (Lime Lynx)"
    ID="almalinux"
    ID_LIKE="rhel centos fedora"
    VERSION_ID="9.1"
    PLATFORM_ID="platform:el9"
    PRETTY_NAME="AlmaLinux 9.1 (Lime Lynx)"
    ANSI_COLOR="0;34"
    LOGO="fedora-logo-icon"
    CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
    HOME_URL="https://almalinux.org/"
    DOCUMENTATION_URL="https://wiki.almalinux.org/"
    BUG_REPORT_URL="https://bugs.almalinux.org/"
    
    ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
    ALMALINUX_MANTISBT_PROJECT_VERSION="9.1"
    REDHAT_SUPPORT_PRODUCT="AlmaLinux"
    REDHAT_SUPPORT_PRODUCT_VERSION="9.1"
    
    [root@microshift ~]# lsblk -f
    NAME        FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
    mmcblk0
    ├─mmcblk0p1 vfat   FAT16       7072-6317                             192.7M    33% /boot
    ├─mmcblk0p2 swap   1     _swap 09f0c50e-3dd9-43d7-80c8-18d379a9fccd                [SWAP]
    └─mmcblk0p3 ext4   1.0   _/    5d5b6a44-2817-44e8-910c-5e8aea35d03f     29G    50% /var/lib/containers/storage/overlay
                                                                                       /
    
    
  13. Update the kernel parameters - Concatenate the following onto the end of the existing line (do not add a new line) in /boot/cmdline.txt
     cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

    A control group (cgroup) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, and so on) of a collection of processes. Cgroups are a key component of containers because there are often multiple processes running in a container that you need to control together. In MicroShift, cgroups are used to implement resource requests and limits and corresponding QoS classes at the pod level.

    reboot
    

    Verify

    ssh root@$ipaddress
    cat /proc/cmdline
    mount | grep cgroup # Check cgroup2
    cat /proc/cgroups | column -t # Check that memory and cpuset are present

    Note that this does not have the hugetlb:

    [root@microshift ~]# cat /proc/cmdline
    coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=1920 bcm2708_fb.fbheight=1200 bcm2708_fb.fbswap=1 smsc95xx.macaddr=E4:5F:01:2E:D8:95 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyAMA0,115200 console=tty1 root=PARTUUID=5ea2038d-03 rootfstype=ext4 elevator=deadline rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
    [root@microshift ~]# mount | grep cgroup # Check cgroup2
    cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
    [root@microshift ~]# cat /proc/cgroups | column -t # Check that memory and cpuset are present - Note that this does not have hugepages this will cause errors in microshift logs
    #subsys_name  hierarchy  num_cgroups  enabled
    cpuset        0          58           1
    cpu           0          58           1
    cpuacct       0          58           1
    blkio         0          58           1
    memory        0          58           1
    devices       0          58           1
    freezer       0          58           1
    net_cls       0          58           1
    perf_event    0          58           1
    net_prio      0          58           1
    pids          0          58           1
    

Install sense_hat and RTIMULib on AlmaLinux

The Sense HAT is an add-on board for the Raspberry Pi. The Sense HAT has an 8 × 8 RGB LED matrix, a five – button joystick and includes the following sensors: Inertial Measurement Unit (Accelerometer, Gyroscope, Magnetometer), Temperature, Barometric pressure, Humidity. If you have the Sense HAT attached, install the libraries.

Install sensehat

dnf -y install zlib zlib-devel libjpeg-devel gcc gcc-c++ i2c-tools python3-devel python3 python3-pip cmake
pip3 install Cython Pillow numpy sense_hat smbus

Check the Sense Hat with i2cdetect

i2cdetect -y 1
[root@microshift ~]# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- 1c -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- UU -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- 5c -- -- 5f
60: -- -- -- -- -- -- -- -- -- -- 6a -- -- -- -- --
70: -- -- -- -- -- -- -- --

Install RTIMULib

dnf -y install git
cd ~
git clone https://github.com/RPi-Distro/RTIMULib.git
cd RTIMULib/
cd Linux/python
python3 setup.py build
python3 setup.py install
cd ../..
cd RTIMULib
mkdir build
cd build
cmake ..
make -j4
make install
ldconfig

# Optional test the sensors
cd /root/RTIMULib/Linux/RTIMULibDrive11
make -j4
make install
RTIMULibDrive11 # Ctrl-C to break

cd /root/RTIMULib/Linux/RTIMULibDrive10
make -j4
make install
RTIMULibDrive10 # Ctrl-C to break

# Optional
dnf -y install qt5-qtbase-devel
cd /root/RTIMULib/Linux/RTIMULibDemoGL
qmake-qt5
make -j4
make install

Test the SenseHat samples for the Sense Hat's LED matrix and sensors. With the latest version of sense_hat you will get “WARNING:root:Failed to initialise TCS34725 colour sensor. (sensor not present)” that you can ignore.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/sensehat-fedora-iot


# Enable random LEDs
python3 sparkles.py # Ctrl-C to interrupt

# Show multiple screens to test LEDs
python3 rainbow.py # Ctrl-C to interrupt

# First time you run the temperature.py, you may see “Temperature: 0 C”. Just run it again.
python3 temperature.py 

# Show the Temperature, Pressure and Humidity
python3 testsensehat.py # Ctrl-C to interrupt

# Show two digits for multiple numbers
sed -i "s/32,32,32/255,255,255/" digits.py
python3 digits.py

# When a magnet gets close to SenseHAT, the LEDs will all turn red for 1/5 of a second
python3 magnetometer.py

# Find Magnetic North
python3 compass.py

Install MicroShift on the Raspberry Pi 4 AlmaLinux host

Setup crio and MicroShift Nightly CentOS Stream 9 aarch64

rpm -qi selinux-policy # selinux-policy-34.1.43
dnf -y install 'dnf-command(copr)'
curl https://copr.fedorainfracloud.org/coprs/g/redhat-et/microshift-nightly/repo/centos-stream-9/group_redhat-et-microshift-nightly-centos-stream-9.repo -o /etc/yum.repos.d/microshift-nightly-centos-stream-9.repo
cat /etc/yum.repos.d/microshift-nightly-centos-stream-9.repo

VERSION=1.24 # You can also use 1.21 or 1.25 with CentOS_9_Stream
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Fedora_36/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_8_Stream/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo
cat /etc/yum.repos.d/devel\:kubic\:libcontainers\:stable\:cri-o\:${VERSION}.repo

dnf -y install firewalld cri-o cri-tools microshift containernetworking-plugins # Be patient, this takes a few minutes

Install KVM on the host and validate the Host Virtualization Setup. The virt-host-validate command validates that the host is configured in a suitable way to run libvirt hypervisor driver qemu.

dnf -y install libvirt-client libvirt-nss qemu-kvm virt-manager virt-install virt-viewer
systemctl enable --now libvirtd
virt-host-validate qemu

Output:

[root@microshift ~]# systemctl enable --now libvirtd
Created symlink /etc/systemd/system/multi-user.target.wants/libvirtd.service → /usr/lib/systemd/system/libvirtd.service.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd.socket → /usr/lib/systemd/system/libvirtd.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd-ro.socket → /usr/lib/systemd/system/libvirtd-ro.socket.
[root@microshift ~]# virt-host-validate qemu
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : WARN (Unknown if this platform has IOMMU support)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)

Check that cni plugins are present

ls /opt/cni/bin/ # empty
ls /usr/libexec/cni # cni plugins

We will have systemd start and manage MicroShift. Refer to the microshift service for the three approaches.

systemctl enable --now crio microshift

# Copy flannel
#  cp /opt/cni/bin/flannel /usr/libexec/cni/.

You may read about selecting zones for your interfaces.

systemctl enable firewalld --now
firewall-cmd --zone=trusted --add-source=10.42.0.0/16 --permanent
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --zone=public --add-port=5353/udp --permanent
firewall-cmd --reload

Additional ports may need to be opened. For external access to run kubectl or oc commands against MicroShift, add the 6443 port:

firewall-cmd --zone=public --permanent --add-port=6443/tcp

For access to services through NodePort, add the port range 30000-32767:

firewall-cmd --zone=public --permanent --add-port=30000-32767/tcp

firewall-cmd --reload
firewall-cmd --list-all --zone=public
firewall-cmd --get-default-zone
#firewall-cmd --set-default-zone=public
#firewall-cmd --get-active-zones
firewall-cmd --list-all

Check the microshift and crio logs

journalctl -u microshift -f
journalctl -u crio -f

The microshift service references the microshift binary in the /usr/bin directory

[root@microshift ~]# cat /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift
Wants=network-online.target crio.service
After=network-online.target crio.service

[Service]
WorkingDirectory=/usr/bin/
ExecStart=microshift run
Restart=always
User=root

[Install]
WantedBy=multi-user.target

Install the kubectl and the openshift oc client

ARCH=arm64
cd /tmp
dnf -y install tar
export OCP_VERSION=4.9.11 && \
    curl -o oc.tar.gz https://mirror2.openshift.com/pub/openshift-v4/$ARCH/clients/ocp/$OCP_VERSION/openshift-client-linux-$OCP_VERSION.tar.gz && \
    tar -xzvf oc.tar.gz && \
    rm -f oc.tar.gz && \
    install -t /usr/local/bin {kubectl,oc} && \
    rm -f {README.md,kubectl,oc}

It will take around 3 minutes for all pods to start. Check the status of node and pods using kubectl or oc client.

export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
#watch "kubectl get nodes;kubectl get pods -A;crictl pods;crictl images"
watch "oc get nodes;oc get pods -A;crictl pods;crictl images"

You may also want to setup the Kubectl and oc autocomplete. The shell command-line completion allows you to quickly build your command without having to type every character.

dnf -y install bash-completion

source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

source <(oc completion bash)
echo "source <(oc completion bash)" >> ~/.bashrc

and use use a shorthand alias for kubectl that also works with completion

alias k=kubectl
complete -o default -F __start_kubectl k

Install podman - We will use podman for containerized deployment of MicroShift and building images for the samples.

dnf -y install podman

Samples to run on MicroShift

We will run samples that will show the use of dynamic persistent volume, SenseHat and the USB camera.

1. InfluxDB/Telegraf/Grafana

The source code is available for this influxdb sample in github.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/influxdb

If you want to run all the steps in a single command, get the nodename.

oc get nodes

Output:

[root@microshift influxdb]# oc get nodes
NAME                     STATUS   ROLES    AGE   VERSION
microshift.example.com   Ready    <none>   55m   v1.21.0

Replace the annotation kubevirt.io/provisionOnNode with the above nodename and execute the runall-balena-dynamic.sh. Note that the node name is different when running MicroShift with the all-in-one containerized approach. So, you will use the microshift.example.com instead of the rpi.example.com.

sed -i "s|coreos|microshift.example.com|" influxdb-data-dynamic.yaml
sed -i "s|coreos|microshift.example.com|" grafana/grafana-data-dynamic.yaml

./runall-balena-dynamic.sh

We create and push the “measure:latest” image using the Dockerfile. The script will create a new project influxdb for this sample, install InfluxDB, install the pod for SenseHat measurements, install Telegraf and check the measurements for the telegraf database in InfluxDB. Finally, it will install Grafana.

This script will allocate dynamic persistent volumes using influxdb-data-dynamic.yaml and grafana-data-dynamic.yaml. The annotation provisionOnNode and the storageClassName are required for dynamic PV.

  annotations:
    kubevirt.io/provisionOnNode: microshift.example.com
spec:
  storageClassName: kubevirt-hostpath-provisioner

Persistent Volumes and Claims Output:

[root@microshift influxdb]# oc get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS                    REASON   AGE
persistentvolume/pvc-8fc99c1b-1c7d-4af1-bcd6-745e102ab91f   57Gi       RWO            Delete           Bound    influxdb/influxdb-data   kubevirt-hostpath-provisioner            3m42s
persistentvolume/pvc-e8fe1dee-ea97-448c-83fc-1ea006180d31   57Gi       RWO            Delete           Bound    influxdb/grafana-data    kubevirt-hostpath-provisioner            63s

NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/grafana-data    Bound    pvc-e8fe1dee-ea97-448c-83fc-1ea006180d31   57Gi       RWO            kubevirt-hostpath-provisioner   63s
persistentvolumeclaim/influxdb-data   Bound    pvc-8fc99c1b-1c7d-4af1-bcd6-745e102ab91f   57Gi       RWO            kubevirt-hostpath-provisioner   3m42s

Add the "<RaspberryPiIPAddress> grafana-service-influxdb.cluster.local" to /etc/hosts on your laptop and login to http://grafana-service-influxdb.cluster.local/login using admin/admin. You can change the password on first login or click on skip. Go to the Dashboards list (left menu > Dashboards > Manage). Open the Analysis Server dashboard to display monitoring information for MicroShift. Open the Balena Sense dashboard to show the temperature, pressure, and humidity from SenseHat.

Finally, after you are done working with this sample, you can run the deleteall-balena-dynamic.sh

./deleteall-balena-dynamic.sh

Deleting the persistent volume claims automatically deletes the persistent volumes.

2. Node Red live data dashboard with SenseHat sensor charts

We will install Node Red on the ARM device as a deployment within MicroShift, add the dashboard and view the gauges for temperature/pressure/humidity data from SenseHat on the dashboard.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/nodered

Build and push the arm64v8 image. I create and use the “karve/nodered:arm64”.

cd docker-custom/
# Replace docker with podman in docker-debian.sh, replace the tag
./docker-debian.sh
podman push karve/nodered:arm64
cd ..

Deploy Node Red with persistent volume for /data within the node red container

mkdir /var/hpvolumes/nodered
restorecon -R -v "/var/hpvolumes/*"
rm -rf /var/hpvolumes/nodered/*;cp -r nodered-volume/* /var/hpvolumes/nodered/.
oc new-project nodered
oc apply -f noderedpv.yaml -f noderedpvc.yaml -f nodered3.yaml -f noderedroute.yaml
oc get routes
oc -n nodered wait deployment nodered-deployment --for condition=Available --timeout=300s
oc logs deployment/nodered-deployment -f

Add the ipaddress of the Raspberry Pi 4 device for nodered-svc-nodered.cluster.local to /etc/hosts on your Laptop and browse to http://nodered-svc-nodered.cluster.local/

The following modules required for the dashboard have been preinstalled node-red-dashboard, node-red-node-smooth, node-red-node-pi-sense-hat. These can be seen under “Manage Palette - Install”. The Flow 1 or Flow 2 have already been imported from the nodered sample. This import to the Node Red can be done manually under “Import Nodes” and then click “Deploy”.

Double click the Sense HAT input node and make sure that all the events are checked. Select the Dashboard. Click on the down arrow on the top right and then the outward arrow below it in the tabs to view the sensor charts. You will see the Home by Default. You can see the state of the Joystick Up, Down, Left, Right or Pressed. Click on the Hamburger Menu (3 lines) and select PiSenseHAT. If you selected the Flow 1, you could click on the Input for the Timestamp under “Dot Matrix” to see the “Alarm” message scroll on the SenseHat LED. You can see the screenshots for these dashboards in previous blogs.

We can continue running the next sample that will reuse this Node Red deployment. If the Node Red Deployment is no longer required, we can delete it as follows:

cd ~/microshift/raspberry-pi/nodered
oc delete -f noderedpv.yaml -f noderedpvc.yaml -f nodered3.yaml -f noderedroute.yaml -n nodered
oc project default
oc delete project nodered

3. TensorFlow Lite Python object detection example in MicroShift with SenseHat and Node Red

This example requires the same Node Red setup as in the previous Sample 2.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/object-detection

We will build the image for object detection send pictures and web socket chat messages to Node Red when a person is detected using a pod in microshift.

podman build -t docker.io/karve/object-detection-raspberrypi4 .
podman push docker.io/karve/object-detection-raspberrypi4:latest

Update the env WebSocketURL and ImageUploadURL in object-detection.yaml as shown below. Also update the hostAliases in object-detection.yaml to point to your raspberry pi 4 ip address (192.168.1.209 shown below).

        env:
          - name: WebSocketURL
            value: "ws://nodered-svc-nodered.cluster.local/ws/chat"
          - name: ImageUploadURL
            value: http://nodered-svc-nodered.cluster.local/upload

      hostAliases:
      - hostnames:
        - nodered-svc-nodered.cluster.local
        ip: 192.168.1.209

Create the deployment

oc project default
oc apply -f object-detection.yaml
oc -n default wait deployment object-detection-deployment --for condition=Available --timeout=300s

We will see pictures being sent to Node Red when a person is detected at http://nodered-svc-nodered.cluster.local/#flow/3e30dc50ae28f61f and chat messages at http://nodered-svc-nodered.cluster.local/chat. When we are done testing, we can delete the deployment

oc delete -f object-detection.yaml

4. Running a Virtual Machine Instance on MicroShift

Find the latest version of the KubeVirt Operator and install.

LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest-arm64)
echo $LATEST
#LATEST=20221123 # I used this version
oc apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml
oc apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml
oc adm policy add-scc-to-user privileged -n kubevirt -z kubevirt-operator

# The .status.phase will show Deploying multiple times and finally Deployed
oc get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}" -w # Ctrl-C to break
oc -n kubevirt wait kv kubevirt --for condition=Available --timeout=300s
oc get pods -n kubevirt

We can build the OKD Web Console (Codename: “bridge”) from the source as mentioned in Part 9. We will run the “bridge” as a container image that we run within MicroShift.

cd /root/microshift/raspberry-pi/console
oc create serviceaccount console -n kube-system
oc create clusterrolebinding console --clusterrole=cluster-admin --serviceaccount=kube-system:console -n kube-system
sleep 5
oc get serviceaccount console --namespace=kube-system -o jsonpath='{.secrets[0].name}' | grep console-token
oc get serviceaccount console --namespace=kube-system -o jsonpath='{.secrets[1].name}' | grep console-token

Replace BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT value https://192.168.1.209:6443 with your raspberry pi 4's ip address, and secretRef token with the console-token-* from above two secret names for BRIDGE_K8S_AUTH_BEARER_TOKEN in okd-web-console-install.yaml. Then apply/create the okd-web-console-install.yaml.

vi okd-web-console-install.yaml
oc apply -f okd-web-console-install.yaml
oc expose svc console-np-service -n kube-system
oc get routes -n kube-system
oc -n kube-system wait deployment console-deployment --for condition=Available --timeout=300s
oc logs deployment/console-deployment -f -n kube-system

Add the Raspberry Pi IP address to /etc/hosts on your Macbook Pro to resolve console-np-service-kube-system.cluster.local. Now you can access the OKD Web Console from your Laptop http://console-np-service-kube-system.cluster.local/. If you see a blank page, you probably have the value of BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT set incorrectly.

We can optionally preload the fedora image into crio (if using the all-in-one containerized approach, this needs to be run within the microshift pod running in podman)

crictl pull quay.io/kubevirt/fedora-cloud-container-disk-demo:20210811_9fec1f849-arm64

Now let’s create a Fedora Virtual Machine Instance using the vmi-fedora.yaml.

cd /root/microshift/raspberry-pi/vmi
oc apply -f vmi-fedora.yaml
watch oc get vmi,pods

The output for the virtualmachineinstance PHASE goes from “Scheduling” to “Scheduled” to “Running” after the virt-launcher-vmi-fedora pod STATUS goes from “Init” to “Running”. Note down the ip address of the vmi-fedora Virtual Machine Instance. Directly connect to the VMI from the Raspberry Pi 4 with fedora as the user and password. Note that it will take another minute or two after the VMI goes to Running state to ssh to the instance. You can look at the console (GUI) or use virtctl (third way below) to see if the VMI is ready.

Output:

[root@microshift vmi]# oc get vmi
NAME         AGE     PHASE     IP           NODENAME                 READY
vmi-fedora   2m38s   Running   10.42.0.19   microshift.example.com   True
[root@microshift vmi]# ssh fedora@10.42.0.19 ping -c 2 google.com
ssh: connect to host 10.42.0.19 port 22: No route to host
[root@microshift vmi]# ssh fedora@10.42.0.19 ping -c 2 google.com
The authenticity of host '10.42.0.19 (10.42.0.19)' can't be established.
ED25519 key fingerprint is SHA256:YGTHSuTB+I2v+vxd/7OfF5INarCZNzXmrzHvMlhrFdI.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.42.0.19' (ED25519) to the list of known hosts.
fedora@10.42.0.19's password:
PING google.com (142.250.65.174) 56(84) bytes of data.
64 bytes from lga25s71-in-f14.1e100.net (142.250.65.174): icmp_seq=1 ttl=59 time=4.70 ms
64 bytes from lga25s71-in-f14.1e100.net (142.250.65.174): icmp_seq=2 ttl=59 time=4.69 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 4.690/4.693/4.696/0.003 ms

Alternatively, a second way is to create a Pod to run the ssh client and connect to the Fedora VM from this pod. Let’s create that openssh-client pod:

oc run alpine --privileged --rm -ti --image=alpine -- /bin/sh
apk update && apk add --no-cache openssh-client

or

oc run sshclient --privileged --rm -ti --image=karve/alpine-sshclient:arm64 -- /bin/sh
#oc attach sshclient -c sshclient -i -t

Then, ssh to the Fedora VMI from this openssh-client container.

Output:

[root@microshift vmi]# oc run alpine --privileged --rm -ti --image=alpine -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # apk update && apk add --no-cache openssh-client
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
v3.17.0_rc4-69-gfea827d99a [https://dl-cdn.alpinelinux.org/alpine/v3.17/main]
v3.17.0-6-gba64c76844 [https://dl-cdn.alpinelinux.org/alpine/v3.17/community]
OK: 17672 distinct packages available
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
(1/6) Installing openssh-keygen (9.1_p1-r1)
(2/6) Installing ncurses-terminfo-base (6.3_p20221119-r0)
(3/6) Installing ncurses-libs (6.3_p20221119-r0)
(4/6) Installing libedit (20221030.3.1-r0)
(5/6) Installing openssh-client-common (9.1_p1-r1)
(6/6) Installing openssh-client-default (9.1_p1-r1)
Executing busybox-1.35.0-r29.trigger
OK: 13 MiB in 21 packages
/ # ssh fedora@10.42.0.19 "bash -c \"ping -c 2 google.com\""
The authenticity of host '10.42.0.19 (10.42.0.19)' can't be established.
ED25519 key fingerprint is SHA256:YGTHSuTB+I2v+vxd/7OfF5INarCZNzXmrzHvMlhrFdI.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.42.0.19' (ED25519) to the list of known hosts.
fedora@10.42.0.19's password:
PING google.com (142.250.65.238) 56(84) bytes of data.
64 bytes from lga25s73-in-f14.1e100.net (142.250.65.238): icmp_seq=1 ttl=117 time=3.89 ms
64 bytes from lga25s73-in-f14.1e100.net (142.250.65.238): icmp_seq=2 ttl=117 time=3.88 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 3.882/3.885/3.889/0.003 ms
/ # exit
Session ended, resume using 'oc attach alpine -c alpine -i -t' command when the pod is running
pod "alpine" deleted

A third way to connect to the VM is to use the virtctl console. You can compile your own virtctl as was described in Part 9. To simplify, we copy virtctl arm64 binary from prebuilt container image to /usr/local/bin on the Raspberry Pi 4 and connect to the VMI using “virtctl console” command.

id=$(podman create docker.io/karve/kubevirt:arm64)
podman cp $id:_out/cmd/virtctl/virtctl /usr/local/bin
podman rm -v $id
virtctl console vmi-fedora

Output:

[root@microshift vmi]# id=$(podman create docker.io/karve/kubevirt:arm64)
Trying to pull docker.io/karve/kubevirt:arm64...
Getting image source signatures
Copying blob 7065f6098427 done
Copying config 1c7a5aa443 done
Writing manifest to image destination
Storing signatures
[root@microshift vmi]# podman cp $id:_out/cmd/virtctl/virtctl /usr/local/bin
[root@microshift vmi]# podman rm -v $id
fcd356f9c9e99fbd6dc49d39707a94c61d2ca4e9ca75ac7692fcaa7dabac0987
[root@microshift vmi]# virtctl console vmi-fedora
Successfully connected to vmi-fedora console. The escape sequence is ^]

vmi-fedora login: fedora
Password:
[fedora@vmi-fedora ~]$ ping -c 2 ibm.com
PING ibm.com (184.87.72.96) 56(84) bytes of data.
64 bytes from a184-87-72-96.deploy.static.akamaitechnologies.com (184.87.72.96): icmp_seq=1 ttl=57 time=6.04 ms
64 bytes from a184-87-72-96.deploy.static.akamaitechnologies.com (184.87.72.96): icmp_seq=2 ttl=57 time=4.63 ms

--- ibm.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.627/5.335/6.044/0.708 ms
[fedora@vmi-fedora ~]$ # ^] to detach
[root@microshift vmi]#

When done, we can delete the VMI

oc delete -f vmi-fedora.yaml

We can run other VM and VMI samples for alpine, cirros and fedora images as in Part 9. You may continue to the next sample 4 where we create a new CentOS VM image. When done, you may delete kubevirt operator

oc delete -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml
oc delete -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml

5. Running a Virtual Machine on MicroShift

We will create a Virtual Machine (VM) using the vm-centos9.yaml. When a VMI is owned by a VM or by another object, it is managed through its owner in the web console or by using the oc command-line interface (CLI). A VirtualMachine provides additional management capabilities to a VirtualMachineInstance inside the cluster. A VirtualMachine will make sure that a VirtualMachineInstance object with an identical name will be present in the cluster, if spec.running is set to true. Further it will make sure that a VirtualMachineInstance will be removed from the cluster if spec.running is set to false. Once a VirtualMachineInstance is created, its state will be tracked via status.created and status.ready fields of the VirtualMachine.

We need to first create an image for use in vm-centos9.yaml. I have already created the docker.io/karve/centos-stream-genericcloud-9-20221206:arm64 as follows:

a. Download the required/latest image from https://cloud.centos.org/altarch/9-stream/aarch64/images/. I used the https://cloud.centos.org/altarch/9-stream/aarch64/images/CentOS-Stream-GenericCloud-9-20221206.0.aarch64.qcow2 image.

b. Create the following Dockerfile. You can do this on your Laptop/Macbook.

FROM docker.io/kubevirt/container-disk-v1alpha
ADD CentOS-Stream-GenericCloud-9-20221206.0.aarch64.qcow2 /disk/

Build and push the image to repository

podman build -t docker.io/karve/centos-stream-genericcloud-9-20221206:arm64 .
podman push docker.io/karve/centos-stream-genericcloud-9-20221206:arm64

Now let’s create and start the CentOS 9 Stream VM on the Raspberry Pi 4. Optionally, pull the image using crictl.

crictl pull docker.io/karve/centos-stream-genericcloud-9-20221206:arm64
cd /root/microshift/raspberry-pi/vmi

Update the vm-centos9.yaml with your public key in ssh_authorized_keys so you can ssh

vi vm-centos9.yaml
oc apply -f vm-centos9.yaml
virtctl start vm-centos9
oc get vm,vmi
virtctl console vm-centos9

Output:

[root@microshift vmi]# cd /root/microshift/raspberry-pi/vmi
[root@microshift vmi]# oc apply -f vm-centos9.yaml
virtualmachine.kubevirt.io/vm-centos9 created
[root@microshift vmi]# oc get vm
NAME         AGE   STATUS    READY
vm-centos9   18s   Stopped   False
[root@microshift vmi]# virtctl start vm-centos9
VM vm-centos9 was scheduled to start
[root@microshift vmi]# oc get vm,vmi
NAME                                    AGE     STATUS     READY
virtualmachine.kubevirt.io/vm-centos9   2m34s   Starting   False

NAME                                            AGE   PHASE     IP    NODENAME   READY
virtualmachineinstance.kubevirt.io/vm-centos9   1s    Pending
[root@microshift vmi]# oc get pods
NAME                             READY   STATUS    RESTARTS   AGE
virt-launcher-vm-centos9-lv4vb   2/2     Running   0          56s
[root@microshift vmi]# oc get vm,vmi
NAME                                    AGE     STATUS    READY
virtualmachine.kubevirt.io/vm-centos9   3m47s   Running   True

NAME                                            AGE   PHASE     IP           NODENAME                 READY
virtualmachineinstance.kubevirt.io/vm-centos9   74s   Running   10.42.0.14   microshift.example.com   True
[root@microshift vmi]# virtctl console vm-centos9
Successfully connected to vm-centos9 console. The escape sequence is ^]
… [You will see the output as the system boots up]
CentOS Stream 9
Kernel 5.14.0-206.el9.aarch64 on an aarch64

Activate the web console with: systemctl enable --now cockpit.socket

vm-centos9 login: cloud-user
Password:
[cloud-user@vm-centos9 ~]$ ping -c 2 google.com
PING google.com (142.251.32.110) 56(84) bytes of data.
64 bytes from lga25s77-in-f14.1e100.net (142.251.32.110): icmp_seq=1 ttl=117 time=9.91 ms
64 bytes from lga25s77-in-f14.1e100.net (142.251.32.110): icmp_seq=2 ttl=117 time=5.24 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 5.244/7.576/9.909/2.332 ms
[cloud-user@vm-centos9 ~]$ # ^] to exit
[root@microshift vmi]#

We can also ssh to the IP address of the VM as the root (using the ssh private key) or as cloud-user using the password centos. Using the Web Console, we can see the VMs and VMIs in a single list view. VMIs are not surfaced in the list if they are owned by a VM. We can also execute the Actions from the UI.

When done, we can stop the Virtual Machine Instance and delete the Virtual Machine.

virtctl stop vm-centos9
oc delete -f vm-centos9.yaml

With MicroShift Virtualization, we let VMs run as a native citizen next to container workloads. In addition to exporting the ssh as a service, we can expose other services in the project the VM is running. Deployed in the same project both use the same network and can access each other’s services.

6. Containerized Data Importer (CDI)

CDI is a utility designed to import Virtual Machine images for use with Kubevirt. At a high level, a PersistentVolumeClaim (PVC) is created. A custom controller watches for importer specific claims, and when discovered, starts an import process to create a raw image with the desired content into the associated PVC.

Update the volume.kubernetes.io/selected-node to your node name (microshift.example.com) in centos9-pv.yaml.

Check the latest arm64 version at https://quay.io/repository/kubevirt/cdi-operator?tab=tags&tag=latest

VERSION=v1.55.2
ARM_VERSION=20221210_a6ebd75e-arm64 # Use the arm64 tag from https://quay.io/repository/kubevirt/cdi-operator?tab=tags&tag=latest


# The version does not work with arm64 images
# oc apply -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml

# So we use the ARM_VERSION
curl -sL https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml | sed "s/$VERSION/$ARM_VERSION/g" | oc apply -f -
# Wait for cdi-operator to start

# Next create the cdi-cr that will create the apiserver, deployment and uploadproxy
oc apply -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

oc get apiservices
oc api-resources --api-group=cdi.kubevirt.io

cd ~/microshift/raspberry-pi/vmi
oc apply -f centos9-dv.yaml # Create a persistent volume by downloading the CentOS image
oc apply -f vm-centos9-datavolume.yaml # Create a CentOS VM by cloning the persistent volume with the above CentOS image

Output:

[root@microshift vmi]# curl -sL https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml | sed "s/$VERSION/$ARM_VERSION/g" | oc apply -f -
namespace/cdi created
customresourcedefinition.apiextensions.k8s.io/cdis.cdi.kubevirt.io created
clusterrole.rbac.authorization.k8s.io/cdi-operator-cluster created
clusterrolebinding.rbac.authorization.k8s.io/cdi-operator created
serviceaccount/cdi-operator created
role.rbac.authorization.k8s.io/cdi-operator created
rolebinding.rbac.authorization.k8s.io/cdi-operator created
deployment.apps/cdi-operator created
configmap/cdi-operator-leader-election-helper created 
[root@microshift vmi]# oc get deployments,pods -n cdi
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cdi-operator   1/1     1            1           19s

NAME                                READY   STATUS    RESTARTS   AGE
pod/cdi-operator-794d4f98fc-hswtk   1/1     Running   0          18s


[root@microshift vmi]# oc apply -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
cdi.cdi.kubevirt.io/cdi created

[root@microshift vmi]# oc get deployments,pods -n cdi
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cdi-apiserver     1/1     1            1           88s
deployment.apps/cdi-deployment    1/1     1            1           85s
deployment.apps/cdi-operator      1/1     1            1           2m46s
deployment.apps/cdi-uploadproxy   1/1     1            1           82s

NAME                                   READY   STATUS    RESTARTS   AGE
pod/cdi-apiserver-6679949df6-7lzc8     1/1     Running   0          87s
pod/cdi-deployment-798bd75486-6chlm    1/1     Running   0          85s
pod/cdi-operator-794d4f98fc-hswtk      1/1     Running   0          2m45s
pod/cdi-uploadproxy-7fb8544d88-574xj   1/1     Running   0          81s

Notice the cdi-apiserver logs that show the "no valid subject specified". We need to change the requestheader-allowed-names ~/microshift/pkg/controllers/kube-apiserver.go. We will fix this issue later by setting the requestheader-allowed-names="". This blank option indicates to an extension apiserver that any CN is acceptable.

[root@microshift vmi]# oc logs deployment/cdi-apiserver -f
2022/12/10 11:31:16 http: TLS handshake error from 10.42.0.1:51380: no valid subject specified
2022/12/10 11:31:16 http: TLS handshake error from 10.42.0.1:51404: no valid subject specified
…

The microshift logs show that it is trying to get a response directly from the pod ip address

Dec 10 13:41:49 microshift.example.com microshift[601]: E1210 13:41:49.572183     601 available_controller.go:508] v1beta1.upload.cdi.kubevirt.io failed with: failing or missing response from https://10.42.0.7:8443/apis/upload.cdi.kubevirt.io/v1beta1: Get "https://10.42.0.7:8443/apis/upload.cdi.kubevirt.io/v1beta1": remote error: tls: bad certificate
Dec 10 13:41:49 microshift.example.com microshift[601]: E1210 13:41:49.958797     601 available_controller.go:508] v1alpha1.upload.cdi.kubevirt.io failed with: failing or missing response from https://10.42.0.7:8443/apis/upload.cdi.kubevirt.io/v1alpha1: Get "https://10.42.0.7:8443/apis/upload.cdi.kubevirt.io/v1alpha1": remote error: tls: bad certificate
[root@microshift ~]# kubectl api-resources --api-group=cdi.kubevirt.io
NAME              SHORTNAMES   APIVERSION                NAMESPACED   KIND
cdiconfigs                     cdi.kubevirt.io/v1beta1   false        CDIConfig
cdis              cdi,cdis     cdi.kubevirt.io/v1beta1   false        CDI
dataimportcrons   dic,dics     cdi.kubevirt.io/v1beta1   true         DataImportCron
datasources       das          cdi.kubevirt.io/v1beta1   true         DataSource
datavolumes       dv,dvs       cdi.kubevirt.io/v1beta1   true         DataVolume
objecttransfers   ot,ots       cdi.kubevirt.io/v1beta1   false        ObjectTransfer
storageprofiles                cdi.kubevirt.io/v1beta1   false        StorageProfile
error: unable to retrieve the complete list of server APIs: upload.cdi.kubevirt.io/v1alpha1: the server is currently unable to handle the request, upload.cdi.kubevirt.io/v1beta1: the server is currently unable to handle the request

Update the kubevirt.io/provisionOnNode in centos-dv.yaml with your microshift node name.

[root@microshift vmi]# oc apply -f centos9-dv.yaml
datavolume.cdi.kubevirt.io/centos9-dv created

# This will create the pod/importer-centos9-dv to import the image
[root@microshift vmi]# oc get dv,pvc,pods,vmi
NAME                                    PHASE              PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/centos9-dv   ImportInProgress   N/A                   12s

NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv   Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   12s

NAME                      READY   STATUS    RESTARTS   AGE
pod/importer-centos9-dv   1/1     Running   0          12s 

[root@microshift vmi]# oc get dv -w
NAME         PHASE              PROGRESS   RESTARTS   AGE
centos9-dv   ImportInProgress   8.19%                 69s
centos9-dv   ImportInProgress   9.19%                 70s
…
centos9-dv   ImportInProgress   99.71%                14m
centos9-dv   Succeeded          100.0%                14m
centos9-dv   Succeeded          100.0%                14m

[root@microshift vmi]# oc get dv,pvc,pods,vmi
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv   Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   15m

Update the persistent volume claim annotation kubevirt.io/provisionOnNode to your microshift node name, update with your public key and apply the vm-centos9-datavolume.yaml.

[root@microshift vmi]# oc apply -f vm-centos9-datavolume.yaml
virtualmachine.kubevirt.io/centos9instance1 created

[root@microshift vmi]# watch oc get dv,pvc,pods,vmi
NAME                                          PHASE             PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/centos9instance1   CloneInProgress   N/A                   37s

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
AGE
persistentvolumeclaim/centos9-dv         Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi	RWO            kubevirt-hostpath-provisioner
16m
persistentvolumeclaim/centos9instance1   Bound    pvc-efd3b974-32f9-4357-bbb0-df6103445435   57Gi	RWO            kubevirt-hostpath-provisioner
37s

NAME                                                  READY   STATUS    RESTARTS   AGE
pod/cdi-upload-centos9instance1                       1/1     Running   0          32s
pod/efd3b974-32f9-4357-bbb0-df6103445435-source-pod   1/1     Running   0          21s

[root@microshift vmi]# oc get dv -w
NAME               PHASE             PROGRESS   RESTARTS   AGE
centos9instance1   CloneInProgress   0.98%                 70s
centos9instance1   CloneInProgress   1.02%                 70s
centos9instance1   CloneInProgress   1.12%                 70s
…
centos9instance1   CloneInProgress   100.00%               5m26s
centos9instance1   Succeeded         100.0%                5m28s
centos9instance1   Succeeded         100.0%                5m29s

[root@microshift vmi]# oc get dv,pvc,pods,vmi
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv         Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   22m
persistentvolumeclaim/centos9instance1   Bound    pvc-efd3b974-32f9-4357-bbb0-df6103445435   57Gi       RWO            kubevirt-hostpath-provisioner   6m27s

NAME                                       READY   STATUS    RESTARTS   AGE
pod/virt-launcher-centos9instance1-5b4xk   1/1     Running   0          59s

NAME                                                  AGE   PHASE     IP           NODENAME                 READY
virtualmachineinstance.kubevirt.io/centos9instance1   59s   Running   10.42.0.34   microshift.example.com   True

Connect to the VM console using virtctl

[root@microshift vmi]# virtctl console centos9instance1
Successfully connected to centos9instance1 console. The escape sequence is ^]
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.14.0-206.el9.aarch64 (mockbuild@aarch64-01.stream.rdu2.redhat.com) (gcc (GCC) 11.3.1 20221121 (Red Hat 11.3.1-4), GNU ld version 2.35.2-24.el9) #1 SMP PREEMPT_DYNAMIC Mon Dec 5 09:08:53 UTC 2022
[    0.000000] The list of certified hardware and cloud instances for Red Hat Enterprise Linux 9 can be viewed at the Red Hat Ecosystem Catalog, https://catalog.redhat.com.
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi: SMBIOS 3.0=0x7f5b0000 MEMATTR=0x7e755018 ACPI 2.0=0x7c030018 MEMRESERVE=0x7c223e18
…
CentOS Stream 9
Kernel 5.14.0-206.el9.aarch64 on an aarch64

Activate the web console with: systemctl enable --now cockpit.socket

centos9instance1 login: cloud-user
Password:
[cloud-user@centos9instance1 ~]$ ping google.com
PING google.com (142.250.65.174) 56(84) bytes of data.
64 bytes from lga25s71-in-f14.1e100.net (142.250.65.174): icmp_seq=1 ttl=59 time=23.9 ms
64 bytes from lga25s71-in-f14.1e100.net (142.250.65.174): icmp_seq=2 ttl=59 time=4.80 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 4.795/14.366/23.937/9.571 ms
[cloud-user@centos9instance1 ~]$ # ^]  to exit
[root@microshift vmi]#

Directly ssh to the VMI as cloud-user using your private key

[root@microshift vmi]# ssh -i ~/amazontestkey.pem cloud-user@10.42.0.34
The authenticity of host '10.42.0.34 (10.42.0.34)' can't be established.
ED25519 key fingerprint is SHA256:C6Yr+wIm7gbinR7pr9pT2kWIostw8c9fwCa9QBWj7I0.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.42.0.34' (ED25519) to the list of known hosts.
Last login: Fri Dec  9 18:48:06 2022
[cloud-user@centos9instance1 ~]$ cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[cloud-user@centos9instance1 ~]$ uname -a
Linux centos9instance1 5.14.0-206.el9.aarch64 #1 SMP PREEMPT_DYNAMIC Mon Dec 5 09:08:53 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
[cloud-user@centos9instance1 ~]$ exit
logout
Connection to 10.42.0.34 closed.

Directly ssh to the VMI as root using your private key

[root@microshift vmi]# ssh -i ~/amazontestkey.pem root@10.42.0.34
Activate the web console with: systemctl enable --now cockpit.socket

[root@centos9instance1 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           351M     0  351M   0% /dev/shm
tmpfs           141M  4.1M  137M   3% /run
/dev/vda2       9.5G  1.2G  8.3G  13% /
/dev/vda1       599M  7.0M  592M   2% /boot/efi
tmpfs            71M     0   71M   0% /run/user/1000
tmpfs            71M     0   71M   0% /run/user/0
[root@centos9instance1 ~]# exit
logout
Connection to 10.42.0.34 closed.

[root@microshift vmi]# oc get vm,vmi,pvc
NAME                                          AGE   STATUS    READY
virtualmachine.kubevirt.io/centos9instance1   19m   Running   True

NAME                                                  AGE   PHASE     IP           NODENAME                 READY
virtualmachineinstance.kubevirt.io/centos9instance1   14m   Running   10.42.0.34   microshift.example.com   True

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv         Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   35m
persistentvolumeclaim/centos9instance1   Bound    pvc-efd3b974-32f9-4357-bbb0-df6103445435   57Gi       RWO            kubevirt-hostpath-provisioner   19m

We can delete the Virtual Machine, the VMI and the persistent volume

[root@microshift vmi]# oc delete -f vm-centos9-datavolume.yaml
virtualmachine.kubevirt.io "centos9instance1" deleted

Or, we can delete separately without cascading

[root@microshift vmi]# oc delete vm centos9instance1 --cascade=false
warning: --cascade=false is deprecated (boolean value) and can be replaced with --cascade=orphan.
virtualmachine.kubevirt.io "centos9instance1" deleted
[root@microshift vmi]# oc get vm,vmi,pvc
NAME                                                  AGE   PHASE     IP           NODENAME                 READY
virtualmachineinstance.kubevirt.io/centos9instance1   14m   Running   10.42.0.34   microshift.example.com   True

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv         Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   36m
persistentvolumeclaim/centos9instance1   Bound    pvc-efd3b974-32f9-4357-bbb0-df6103445435   57Gi       RWO            kubevirt-hostpath-provisioner   20m
[root@microshift vmi]# oc delete vmi centos9instance1 --cascade=orphan
virtualmachineinstance.kubevirt.io "centos9instance1" deleted
[root@microshift vmi]# oc get vm,vmi,pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv         Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   36m
persistentvolumeclaim/centos9instance1   Bound    pvc-efd3b974-32f9-4357-bbb0-df6103445435   57Gi       RWO            kubevirt-hostpath-provisioner   20m
[root@microshift vmi]# oc delete pvc centos9instance1
persistentvolumeclaim "centos9instance1" deleted
[root@microshift vmi]# oc get vm,vmi,pvc
NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
persistentvolumeclaim/centos9-dv   Bound    pvc-084d9711-410c-4ef8-8614-66b73f96ecb1   57Gi       RWO            kubevirt-hostpath-provisioner   36m

7. Virtctl image-upload with CDI

DataVolume objects are custom resources that are provided by the Containerized Data Importer (CDI) project. Data volumes orchestrate import, clone, and upload operations that are associated with an underlying persistent volume claim (PVC). Data volumes are integrated with OKD Virtualization, and they prevent a virtual machine from being started before the PVC has been prepared. The Containerized Data Importer (CDI) requires scratch space (temporary storage) to complete some operations, such as importing and uploading virtual machine images. During this process, CDI provisions a scratch space PVC equal to the size of the PVC backing the destination data volume (DV). The scratch space PVC is deleted after the operation completes or aborts. The CDI supported operations matrix shows the supported CDI operations for content types against endpoints, and which of these operations requires scratch space (qcow2 or raw with gz or xz compression are supported, do not use the tar.gz or tar.xz). 

Add the ipaddress of your Raspberry Pi 4 for the cdi-uploadproxy-cdi.cluster.local to /etc/hosts (on the Raspberry Pi 4) and update the kubevirt.io/provisionOnNode in example-upload-dv.yaml with your microshift node name. Note that it creates a pod for uploading the image and two volumes, one of them is a scratch volume. To allow insecure server connections when using HTTPS, use the --insecure parameter. Be aware that when you use the --insecure flag, the authenticity of the upload endpoint is not verified.

echo $ipaddress cdi-uploadproxy-cdi.cluster.local >> /etc/hosts
oc apply -f example-upload-dv.yaml
virtctl image-upload dv example-upload-dv --namespace default --size 10Gi --image-path `pwd`/CentOS-Stream-GenericCloud-9-20221206.0.aarch64.qcow2 --wait-secs 1200 --no-create --uploadproxy-url=https://cdi-uploadproxy-cdi.cluster.local --insecure

Output:

[root@microshift vmi]# oc apply -f example-upload-dv.yaml
datavolume.cdi.kubevirt.io/example-upload-dv created
[root@microshift vmi]# watch oc get dv,pvc,pods -n default
NAME                                           PHASE             PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/example-upload-dv   UploadScheduled   N/A                   41s

NAME                                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
         AGE
persistentvolumeclaim/example-upload-dv           Bound    pvc-a36a235b-3683-44d6-ba1e-9754155f632f   57Gi	 RWO            kubevirt-hostpath-provi
sioner   40s
persistentvolumeclaim/example-upload-dv-scratch   Bound    pvc-4d836e63-f705-4ca0-9f89-c44566f3a2f6   57Gi	 RWO            kubevirt-hostpath-provi
sioner   38s

NAME                               READY   STATUS    RESTARTS   AGE
pod/cdi-upload-example-upload-dv   0/1     Running   0          38s

The default microshift binary will give the following error with image-upload

[root@microshift vmi]# virtctl image-upload dv example-upload-dv --namespace default --size 10Gi --image-path `pwd`/CentOS-Stream-GenericCloud-9-20221206.0.aarch64.qcow2 --wait-secs 1200 --no-create --uploadproxy-url=https://cdi-uploadproxy-cdi.cluster.local --insecure
Using existing PVC default/example-upload-dv
Uploading data to https://cdi-uploadproxy-cdi.cluster.local
the server is currently unable to handle the request (post uploadtokenrequests.upload.cdi.kubevirt.io)

We need to fix this. Modify the ~/microshift/pkg/controllers/kube-apiserver.go and update the line requestheader-allowed-names to empty and recompile microshift binary.

                "--requestheader-allowed-names=",

Then run the cleanup, replace the /usr/bin/microshift binary and restart microshift.

[root@microshift hack]# ./cleanup.sh
DATA LOSS WARNING: Do you wish to stop and cleanup ALL MicroShift data AND cri-o container workloads?
1) Yes
2) No
#? 1
Stopping microshift
Removing crio pods 
…
[root@microshift hack]# cd ..
[root@microshift microshift]# vi pkg/controllers/kube-apiserver.go
[root@microshift microshift]# make
[root@microshift microshift]# cp microshift /usr/bin/microshift
cp: overwrite '/usr/bin/microshift'? y
[root@microshift microshift]# restorecon -R -v /usr/bin/microshift
[root@microshift microshift]# systemctl start microshift

If you get 503 Service Unavailable in the microshift journalctl logs, just run “systemctl restart microshift”. Then, install the CDI as shown previously and create the example-upload-dv.yaml.

[root@microshift microshift]# cd raspberry-pi/vmi 
[root@microshift vmi]# oc apply -f example-upload-dv.yaml
datavolume.cdi.kubevirt.io/example-upload-dv created
[root@microshift vmi]# virtctl image-upload dv example-upload-dv --namespace default --size 10Gi --image-path `pwd`/CentOS-Stream-GenericCloud-9-20221206.0.aarch64.qcow2 --wait-secs 1200 --no-create --uploadproxy-url=https://cdi-uploadproxy-cdi.cluster.local --insecure
Using existing PVC default/example-upload-dv
Uploading data to https://cdi-uploadproxy-cdi.cluster.local

 748.12 MiB / 748.12 MiB [==============================================================================================================] 100.00% 1m38s

Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading /root/microshift/raspberry-pi/vmi/CentOS-Stream-GenericCloud-9-20221206.0.aarch64.qcow2 completed successfully

The upload pod logs show the image being processed as it is uploaded.

[root@microshift vmi]# oc logs cdi-upload-example-upload-dv -f
I1211 10:03:18.977355       1 uploadserver.go:74] Running server on 0.0.0.0:8443
I1211 10:19:49.234784       1 uploadserver.go:324] Content type header is ""
I1211 10:19:49.237511       1 data-processor.go:379] Calculating available size
I1211 10:19:49.238101       1 data-processor.go:391] Checking out file system volume size.
I1211 10:19:49.238309       1 data-processor.go:399] Request image size not empty.
I1211 10:19:49.238394       1 data-processor.go:404] Target size 10Gi.
I1211 10:19:49.238782       1 data-processor.go:282] New phase: TransferScratch
I1211 10:19:49.242050       1 util.go:191] Writing data...
I1211 10:20:59.306801       1 data-processor.go:282] New phase: ValidatePause
I1211 10:20:59.311513       1 data-processor.go:288] Validating image
I1211 10:20:59.441552       1 data-processor.go:282] New phase: Pause
I1211 10:20:59.441700       1 uploadserver.go:367] Returning success to caller, continue processing in background
I1211 10:20:59.441832       1 data-processor.go:193] Resuming processing at phase Convert
I1211 10:20:59.441913       1 data-processor.go:288] Validating image
I1211 10:23:53.110341       1 data-processor.go:282] New phase: Resize
W1211 10:23:53.176981       1 data-processor.go:361] Available space less than requested size, resizing image to available space 10146021376.
I1211 10:23:53.177109       1 data-processor.go:372] Expanding image size to: 10146021376
I1211 10:23:53.372418       1 data-processor.go:288] Validating image
I1211 10:23:53.409971       1 data-processor.go:282] New phase: Complete
I1211 10:23:53.410194       1 uploadserver.go:364] Wrote data to /data/disk.img
I1211 10:23:53.410273       1 uploadserver.go:203] Shutting down http server after successful upload
I1211 10:23:53.420251       1 uploadserver.go:103] UploadServer successfully exited

The scratch volume and the upload pod will be deleted after the upload is complete. The persistent volume with the CentOS 9 Stream image can now be used as a source to clone multiple VMs. Now you can install the KubeVirt Operator as shown in Sample 5 (if not already installed) and create a VM by cloning this image.

[root@microshift vmi]# oc apply -f vm-centos9-uploadvolume.yaml
virtualmachine.kubevirt.io/centos9instance2 created
[root@microshift vmi]# watch oc get dv,pvc,pods,vm,vmi -n default
NAME                                          PHASE             PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/centos9instance2   CloneInProgress   N/A                   15s

NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
 AGE
persistentvolumeclaim/centos9instance2    Bound    pvc-3cde8c3f-3d15-42bc-9961-f5c969deb1a4   57Gi	 RWO            kubevirt-hostpath-provisioner
 14s
persistentvolumeclaim/example-upload-dv   Bound    pvc-347eb116-b02f-4790-adb5-adb239e8dcc5   57Gi	 RWO            kubevirt-hostpath-provisioner
 23m

NAME                                                  READY   STATUS              RESTARTS   AGE
pod/3cde8c3f-3d15-42bc-9961-f5c969deb1a4-source-pod   0/1     ContainerCreating   0          4s
pod/cdi-upload-centos9instance2                       1/1     Running      	  0          10s

NAME                                          AGE   STATUS         READY
virtualmachine.kubevirt.io/centos9instance2   15s   Provisioning   False

Update the persistent volume claim annotation kubevirt.io/provisionOnNode to your microshift node name in the vm-centos9-uploadvolume.yaml and apply this yaml. The DataVolume for centos9instance2 will be cloned. You will see the CloneInProgress for the data volume with a percentage of progress. The VM will be provisioned and the VMI for centos9instance2 will be started. You can connect to it as in previous example. When done, run the “oc delete -f vm-centos9-uploadvolume.yaml” to delete the VM and the cloned persistent volume.


You can also use the arm64 Ubuntu Cloud Images. We will run the Ubuntu VM using the https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img. When the VM is started, you can login using virtctl console with userid ubuntu and password ubuntu provided in the vm-ubuntujammy-uploadvolume.yaml. You can also ssh to the VMI ip address (10.42.0.20 shown below).

oc delete example-upload-dv
oc apply -f example-upload-dv.yaml
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-arm64.img
virtctl image-upload dv example-upload-dv --namespace default --size 10Gi --image-path jammy-server-cloudimg-arm64.img --wait-secs 1200 --no-create --uploadproxy-url=https://cdi-uploadproxy-cdi.cluster.local --insecure
oc apply -f vm-ubuntujammy-uploadvolume.yaml
ssh -i ~/amazontestkey.pem root@10.42.0.20
ssh -i ~/amazontestkey.pem ubuntu@10.42.0.20

You can do the same with the https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-arm64.img using the vm-ubuntulunar-uploadvolume.yaml.

You can even test by compressing and uploading the lunar-server-cloudimg-arm64.img.xz image.

xz -v lunar-server-cloudimg-arm64.img
Ubuntu Lunar VM in OKD Console


Finally, you can delete the CDI resource and Operator as follows:

VERSION=v1.55.2
oc delete -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
oc delete -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml

Output:

[root@microshift vmi]# oc delete -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
cdi.cdi.kubevirt.io "cdi" deleted
root@microshift vmi]# oc get pods -n cdi
NAME                               READY   STATUS        RESTARTS   AGE
cdi-apiserver-7d579fc44c-jgwqs     0/1     Terminating   0          152m
cdi-operator-794d4f98fc-dxv5b      1/1     Running       0          110m
cdi-uploadproxy-6546b64f8f-zp9nv   1/1     Terminating   0          151m
[root@microshift vmi]# oc delete -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
namespace "cdi" deleted
customresourcedefinition.apiextensions.k8s.io "cdis.cdi.kubevirt.io" deleted
clusterrole.rbac.authorization.k8s.io "cdi-operator-cluster" deleted
clusterrolebinding.rbac.authorization.k8s.io "cdi-operator" deleted
serviceaccount "cdi-operator" deleted
role.rbac.authorization.k8s.io "cdi-operator" deleted
rolebinding.rbac.authorization.k8s.io "cdi-operator" deleted
deployment.apps "cdi-operator" deleted
configmap "cdi-operator-leader-election-helper" deleted

7. MongoDB

We will deploy and use the mongodb database using the image: docker.io/arm64v8/mongo:4.4.18. Do not use the latest tag for the image. It will result in "WARNING: MongoDB 5.0+ requires ARMv8.2-A or higher, and your current system does not appear to implement any of the common features for that!" and fail to start. Raspberry Pi 4 uses an ARM Cortex-A72 which is ARM v8-A.

A new PersistentVolumeClaim mongodb will use the storageClassName: kubevirt-hostpath-provisioner for the Persistent Volume. The mongodb-root-username uses the root user with a the mongodb-root-password set to a default of mongodb-password. Update the volume.kubernetes.io/selected-node in mongodb-pv.yaml to your microshift node name.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/mongodb
vi mongodb-pv.yaml # Update the microshift node name
oc project default
oc apply -f .

We create the school db and insert 1000 records into the student collection.

Output:

[root@microshift mongodb]# oc exec -it statefulset/mongodb -- bash
groups: cannot find name for group ID 1001
1001@mongodb-0:/$ mongo admin --host mongodb.default.svc.cluster.local:27017 --authenticationDatabase admin -u root -p mongodb-password
MongoDB shell version v4.4.18
connecting to: mongodb://mongodb.default.svc.cluster.local:27017/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("604ba70d-da1b-4afa-99d5-00d12f75e3e8") }
MongoDB server version: 4.4.18
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
	https://community.mongodb.com
---
The server generated these startup warnings when booting:
        2022-12-08T17:42:59.611+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
school  0.000GB
> use admin
switched to db admin
> show users
{
	"_id" : "admin.root",
	"userId" : UUID("658e0dae-b995-4912-aacc-131710f62846"),
	"user" : "root",
	"db" : "admin",
	"roles" : [
		{
			"role" : "root",
			"db" : "admin"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}
> use school
switched to db school
> for(var i = 1;i<=1000;i++){ db.students.insert({student_id:i,class:Math.ceil(Math.random()*20),scores:[{type:"exam" , score:Math.ceil(Math.random()*100)},{type:"quiz" , score:Math.ceil(Math.random()*100)},{type:"homework" , score:Math.ceil(Math.random()*100)},{type:"homework" , score:Math.ceil(Math.random()*100)}]}); if(i%100 == 0) {print(i);} }
100
200
300
400
500
600
700
800
900
1000
> db.students.findOne()
{
	"_id" : ObjectId("639223deebe39f904788264f"),
	"student_id" : 1,
	"class" : 19,
	"scores" : [
		{
			"type" : "exam",
			"score" : 6
		},
		{
			"type" : "quiz",
			"score" : 67
		},
		{
			"type" : "homework",
			"score" : 52
		},
		{
			"type" : "homework",
			"score" : 11
		}
	]
}
> exit
bye
1001@mongodb-0:/$ exit
exit

We can run another client pod to connect to the mongodb service and check the number of students in the collection in the school db we created earlier.

[root@microshift mongodb]# oc run --namespace default mongodb-client --rm --tty -i --restart='Never' --image docker.io/arm64v8/mongo:4.4.18 -- bash
If you don't see a command prompt, try pressing enter.
root@mongodb-client:/# mongo admin --host mongodb.default.svc.cluster.local:27017 --authenticationDatabase admin -u root -p mongodb-password
MongoDB shell version v4.4.18
connecting to: mongodb://mongodb.default.svc.cluster.local:27017/admin?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("d5ed2632-3bec-4527-a8b3-cd7651c14fbb") }
MongoDB server version: 4.4.18
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	https://docs.mongodb.com/
Questions? Try the MongoDB Developer Community Forums
	https://community.mongodb.com
---
The server generated these startup warnings when booting:
        2022-12-08T18:10:02.794+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> use school
switched to db school
> db.students.count()
1000
> exit
bye
root@mongodb-client:/# exit
exit
pod "mongodb-client" deleted

When we are done, we can delete the mongodb

cd ~/microshift/raspberry-pi/mongodb
oc delete -f .
You can alternatively install mongodb using the bitnami helm chart, however the auth will not be set because the chart uses different env variables for username and password within the pod.
helm install mongodb bitnami/mongodb --set image.registry=docker.io --set image.repository=arm64v8/mongo --set image.tag=4.4.18 --set persistence.mountPath=/data/db --set livenessProbe.enabled=false --set readinessProbe.enabled=false --set auth.enabled=false

Cleanup MicroShift

We can use the cleanup.sh script available on github to cleanup the pods and images. If you already cloned the microshift repo from github, you have the script in the ~/microshift/hack directory.

cd ~/microshift/hack
./cleanup.sh

Containerized MicroShift on AlmaLinux (64 bit)

We can run MicroShift within containers in two ways:

  1. MicroShift Containerized – The MicroShift binary runs in a Podman container, CRI-O Systemd service runs directly on the host and data is stored in a podman volume
  2. MicroShift Containerized All-In-One – The MicroShift binary and CRI-O service run within a container and data is stored in a podman volume, microshift-data. This should be used for “Testing and Development” only

Microshift Containerized

If you did not already install podman, you can do it now.

dnf install -y podman

We will use a new microshift.service that runs microshift in a pod using the prebuilt image and uses a podman volume. Rest of the pods run using crio on the host. We use the image in the microshift.service below with docker.io/karve/microshift:latest that was built using instructions in Part 4. This container image contains the microshift binary that fixes the hugetlb and CDI requestheader-allowed-names errors.

cat << EOF > /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift Containerized
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target crio.service
After=network-online.target crio.service
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutSec=3000
ExecStartPre=/usr/bin/mkdir -p /var/lib/kubelet ; /usr/bin/mkdir -p /var/hpvolumes
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/bin/podman run \
  --cidfile=%t/%n.ctr-id \
  --cgroups=no-conmon \
  --rm \
  --replace \
  --sdnotify=container \
  --label io.containers.autoupdate=registry \
  --network=host \
  --privileged \
  -d \
  --name microshift \
  -v /var/hpvolumes:/var/hpvolumes:z,rw,rshared \
  -v /var/run/crio/crio.sock:/var/run/crio/crio.sock:rw,rshared \
  -v microshift-data:/var/lib/microshift:rw,rshared \
  -v /var/lib/kubelet:/var/lib/kubelet:z,rw,rshared \
  -v /var/log:/var/log \
  -v /etc:/etc quay.io/microshift/microshift:latest
ExecStop=/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target default.target
EOF


systemctl daemon-reload
systemctl enable --now crio microshift
podman ps -a
podman volume inspect microshift-data # Get the Mountpoint where kubeconfig is located
export KUBECONFIG=/var/lib/containers/storage/volumes/microshift-data/_data/resources/kubeadmin/kubeconfig
watch "oc get nodes;oc get pods -A;crictl pods;crictl images"

The microshift process may not work, no nodes and no pods, it remains in the following state:

No resources found
No resources found
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT             RUNTIME
IMAGE                           TAG                 IMAGE ID            SIZE
quay.io/microshift/microshift   latest              bdccb7de6c282	406MB
CONTAINER ID  IMAGE                                 COMMAND     CREATED         STATUS             PORTS       NAMES
31fba8a38d3b  quay.io/microshift/microshift:latest  run         33 minutes ago  Up 33 minutes ago              microshift

In above case, the podman logs for microshift show “ResponseCode: 503, Body: service unavailable”.

podman logs -f microshift 2>&1 | grep 503
E1127 16:56:29.039225       1 controller.go:116] loading OpenAPI spec for "v1.user.openshift.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

Restart the microshift.service and wait for pods to start. I will update this blog if I find the cause and a better way to fix this!

[root@microshift hack]# podman exec -it microshift kill 1
[root@microshift hack]# systemctl start microshift

or

[root@microshift hack]# systemctl restart microshift

You will see errors about the hugepages with default image because the /proc/cgroups does not have hugepages. You may ignore them. Alternatively, this can be fixed by rebuilding the microshift binary as shown in later section (docker.io/karve/microshift:latest) and updating the microshift container.

W1127 17:45:34.302111       1 container.go:586] Failed to update stats for container "/system.slice": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
failed to fetch hugetlb info
github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/google/cadvisor/container/raw/handler.go:232
github.com/google/cadvisor/manager.(*containerData).updateStats
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/google/cadvisor/manager/container.go:637
github.com/google/cadvisor/manager.(*containerData).housekeepingTick
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/google/cadvisor/manager/container.go:583
github.com/google/cadvisor/manager.(*containerData).housekeeping
	/opt/app-root/src/github.com/redhat-et/microshift/vendor/github.com/google/cadvisor/manager/container.go:531
runtime.goexit
	/usr/lib/golang/src/runtime/asm_arm64.s:1130], continuing to push stats

Now that microshift is started, we can run the samples shown earlier.

After we are done, we can delete the microshift container. The --rm we used in the podman run will delete the container when we stop it.

podman stop microshift && podman volume rm microshift-data

After it is stopped, we can run the cleanup.sh to delete the pods and images from crio.

MicroShift Containerized All-In-One

Let’s stop the crio on the host, we will be creating an all-in-one container in podman that will run crio within the container.

systemctl stop crio
systemctl disable crio
mkdir /var/hpvolumes

We will run the all-in-one microshift in podman using prebuilt images (replace the image in the podman run command below with the latest image). Normally you would run the following to start the all-in-one microshift, but it does not work. You can try to restart microshift with “systemctl restart microshift”, that won’t get the node to the Ready state.

setsebool -P container_manage_cgroup true 
podman volume rm microshift-data;podman volume create microshift-data
podman run -d --rm --name microshift -h microshift-aio.example.com --privileged -v /lib/modules:/lib/modules -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -p 6443:6443 -p 8080:8080 -p 80:80 quay.io/microshift/microshift-aio:4.8.0-0.microshift-2022-04-20-182108-linux-nft-arm64

The microshift logs within the container will show Error in setting cgroup config for procHooks

Nov 27 18:32:46 microshift-aio.example.com microshift[731]: E1127 18:32:46.095391     731 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-flannel-ds-65q9m_kube-system(9f02c0dc-540b-42c1-89ff-ad545b30131d)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-flannel-ds-65q9m_kube-system(9f02c0dc-540b-42c1-89ff-ad545b30131d)\\\": rpc error: code = Unknown desc = container create failed: time=\\\"2022-11-27T18:32:45Z\\\" level=warning msg=\\\"unable to get oom kill count\\\" error=\\\"openat2 /sys/fs/cgroup/system.slice/runc-d265db5497b3bfec60cd9f0b9254aa103973df2edcfade304692c663e1e1a958.scope/memory.events: no such file or directory\\\"\\ntime=\\\"2022-11-27T18:32:45Z\\\" level=error msg=\\\"container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: openat2 /sys/fs/cgroup/system.slice/runc-d265db5497b3bfec60cd9f0b9254aa103973df2edcfade304692c663e1e1a958.scope/cpu.weight: no such file or directory\\\"\\n\"" pod="kube-system/kube-flannel-ds-65q9m" podUID=9f02c0dc-540b-42c1-89ff-ad545b30131d
Nov 27 18:32:47 microshift-aio.example.com microshift[731]: E1127 18:32:47.463863     731 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?" pod="openshift-dns/dns-default-8qrnk" podUID=5f6607e3-f280-4f27-8e29-d059a42a1911
Nov 27 18:32:48 microshift-aio.example.com microshift[731]: E1127 18:32:48.173648     731 kubelet.go:2223] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?"

Let’s fix this. You can stop the above non-working microshift container with

podman stop microshift

Since the “setsebool -P container_manage_cgroup true” does not work, we mount the /sys/fs/cgroup into the container using -v /sys/fs/cgroup:/sys/fs/cgroup:ro. This will volume mount /sys/fs/cgroup into the container as read/only, but the subdir/mount points will be mounted in as read/write.

podman run -d --rm --name microshift -h microshift-aio.example.com --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /lib/modules:/lib/modules -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -p 6443:6443 -p 8080:8080 -p 80:80 quay.io/microshift/microshift-aio:4.8.0-0.microshift-2022-04-20-182108-linux-nft-arm64

You will see the hugepages error in the microshift logs that you can ignore (or recompile the microshift binary as shown later and use it within the container)

Nov 27 18:38:36 microshift-aio.example.com microshift[67]: W1127 18:38:36.566697      67 container.go:586] Failed to update stats for container "/system.slice/crio-4d14e2f74d53bab169368fd9fc1295a8b24ffb0f7e17c0a781cef8eaf6c89d3b.scope": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory

Also, you will initially see the flannel errors “open /run/flannel/subnet.env: no such file or directory”. You can ignore these. These errors will resolve after the flannel pod starts and creates the subnet.env.

Nov 27 19:36:55 microshift-aio.example.com microshift[69]: E1127 19:36:55.469267      69 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kubevirt-hostpath-provisioner-vcdmn_kubevirt-hostpath-provisioner(3b99266e-2387-406e-ad67-d6839807b9d3)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kubevirt-hostpath-provisioner-vcdmn_kubevirt-hostpath-provisioner(3b99266e-2387-406e-ad67-d6839807b9d3)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kubevirt-hostpath-provisioner-vcdmn_kubevirt-hostpath-provisioner_3b99266e-2387-406e-ad67-d6839807b9d3_0(1f1c02ab07d59ad32c530747e653b0c6171063022d0275df534d3bf69a0ac553): error adding pod kubevirt-hostpath-provisioner_kubevirt-hostpath-provisioner-vcdmn to CNI network \\\"cbr0\\\": open /run/flannel/subnet.env: no such file or directory\"" pod="kubevirt-hostpath-provisioner/kubevirt-hostpath-provisioner-vcdmn" podUID=3b99266e-2387-406e-ad67-d6839807b9d3

Now that you know the podman command to start the microshift all-in-one, you may alternatively use the following microshift service.

cat << EOF > /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift all-in-one
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutSec=3000
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --sdnotify=conmon --cgroups=no-conmon --rm --replace -d --name microshift -h microshift-aio.example.com --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -v /lib/modules:/lib/modules --label io.containers.autoupdate=registry -p 6443:6443 -p 80:80 quay.io/microshift/microshift-aio:latest
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target default.target 
EOF

systemctl daemon-reload
systemctl start microshift

Note that the “-h microshift-aio.example.com” hostname used in the container should be different from the hostname of the Raspberry Pi (microshift.example.com). On the host Raspberry Pi 4, we set KUBECONFIG to point to the kubeconfig on the data volume.

export KUBECONFIG=/var/lib/containers/storage/volumes/microshift-data/_data/microshift/resources/kubeadmin/kubeconfig
# crio on host is stopped, so we do not run crictl commands on host
watch "oc get nodes;oc get pods -A;podman exec -it microshift crictl ps -a"

The crio service is stopped on the Raspberry Pi, so crictl command will not work directly on the Pi. The crictl commands will work within the microshift container in podman as shown in the watch command above. You will need to wait for the new pods to be started as seen with crictl (the oc pods command may still show the previous pods from etcd).

Now, we can run the samples shown earlier. To run the Virtual Machine examples in the all-in-one MicroShift, we need to execute the mount with --make-shared as follows in the microshift container to prevent the “Error: path "/var/run/kubevirt" is mounted on "/" but it is not a shared mount” event from virt-handler.

podman exec -it microshift mount --make-shared /

We may also preload the virtual machine images using "crictl pull".

podman exec -it microshift crictl pull quay.io/kubevirt/fedora-cloud-container-disk-demo:20210811_9fec1f849-arm64

For the Virtual Machine Instance Sample 4, we can connect to the vmi-fedora by exposing the ssh port for the Virtual Machine Instance as a NodePort Service after the instance is started. This NodePort is within the all-in-one pod that is running in podman. The ip address of the all-in-one microshift podman container is 10.88.0.6. We expose the target port 22 on the VM as a service on port 22 that is in turn exposed on the microshift container with allocated port 32574 as seen below. We run and exec into a new pod called ssh-proxy, install the openssh-client on the ssh-proxy and ssh to the port 32574 on the all-in-one microshift container. This takes us to the VMI port 22 as shown below:

[root@microshift vmi]# oc get vmi,pods
NAME                                            AGE     PHASE     IP           NODENAME                     READY
virtualmachineinstance.kubevirt.io/vmi-fedora   7m55s   Running   10.42.0.16   microshift-aio.example.com   True

NAME                                 READY   STATUS    RESTARTS   AGE
pod/virt-launcher-vmi-fedora-k9bbt   2/2     Running   0          6m41s
[root@microshift vmi]# virtctl expose vmi vmi-fedora --port=22 --target-port=22 --name=vmi-fedora-ssh --type=NodePort
Service vmi-fedora-ssh successfully exposed for vmi vmi-fedora
[root@microshift vmi]# oc get svc vmi-fedora-ssh
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
vmi-fedora-ssh   NodePort   10.43.181.142   <none>        22:32574/TCP   9s
[root@microshift vmi]# podman inspect --format "{{.NetworkSettings.IPAddress}}" microshift
10.88.0.6
[root@microshift vmi]# oc run -i --tty ssh-proxy --rm --image=karve/alpine-sshclient:arm64 --restart=Never -- /bin/sh -c "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null fedora@10.88.0.6 -p 32574"
If you don't see a command prompt, try pressing enter.

[fedora@vmi-fedora ~]$ sudo dnf install -y qemu-guest-agent >/dev/null
[fedora@vmi-fedora ~]$ sudo systemctl enable --now qemu-guest-agent 
[fedora@vmi-fedora ~]$ ping -c 2 google.com
PING google.com (142.250.72.110) 56(84) bytes of data.
64 bytes from lga34s32-in-f14.1e100.net (142.250.72.110): icmp_seq=1 ttl=115 time=5.88 ms
64 bytes from lga34s32-in-f14.1e100.net (142.250.72.110): icmp_seq=2 ttl=115 time=5.13 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 5.132/5.505/5.878/0.373 ms
[fedora@vmi-fedora ~]$ exit
logout
Connection to 10.88.0.6 closed.
pod "ssh-proxy" deleted

We can install the QEMU guest agent daemon that runs on the virtual machine and passes information to the host about the virtual machine, users, file systems, and secondary networks.

After we are done, we can delete the all-in-one microshift container.

podman rm -f microshift && podman volume rm microshift-data

or if started using systemd, then

systemctl stop microshift

Build and install the Kata Containers

Run the cleanup shown earlier, then create the service for running MicroShift non-containerized.

cat << EOF > /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift
Wants=network-online.target crio.service
After=network-online.target crio.service

[Service]
WorkingDirectory=/usr/bin/
ExecStart=microshift run
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload

Build and install kata-runtime

go get -d -u github.com/kata-containers/kata-containers
cd /root/go/src/github.com/kata-containers/kata-containers/src/runtime/
make
make install

Output:

[root@microshift runtime]# make
     GENERATE data/kata-collect-data.sh
     GENERATE pkg/katautils/config-settings.go
kata-runtime - version 3.1.0-alpha0 (commit 9bde32daa102368b9dbc27a6c03ed2e3e87d65e1)

• architecture:
	Host:
	golang:
	Build: arm64

• golang:
	go version go1.19.3 linux/arm64

• hypervisors:
	Default: qemu
	Known: acrn cloud-hypervisor firecracker qemu
	Available for this architecture: cloud-hypervisor firecracker qemu

• Summary:

	destination install path (DESTDIR) : /
	binary installation path (BINDIR) : /usr/local/bin
	binaries to install :
	 - /usr/local/bin/kata-runtime
	 - /usr/local/bin/containerd-shim-kata-v2
	 - /usr/local/bin/kata-monitor
	 - /usr/local/bin/data/kata-collect-data.sh
	configs to install (CONFIGS) :
	 - config/configuration-clh.toml
 	 - config/configuration-fc.toml
 	 - config/configuration-qemu.toml
	install paths (CONFIG_PATHS) :
	 - /usr/share/defaults/kata-containers/configuration-clh.toml
 	 - /usr/share/defaults/kata-containers/configuration-fc.toml
 	 - /usr/share/defaults/kata-containers/configuration-qemu.toml
	alternate config paths (SYSCONFIG_PATHS) :
	 - /etc/kata-containers/configuration-clh.toml
 	 - /etc/kata-containers/configuration-fc.toml
 	 - /etc/kata-containers/configuration-qemu.toml
	default install path for qemu (CONFIG_PATH) : /usr/share/defaults/kata-containers/configuration.toml
	default alternate config path (SYSCONFIG) : /etc/kata-containers/configuration.toml
	qemu hypervisor path (QEMUPATH) : /usr/bin/qemu-system-aarch64
	cloud-hypervisor hypervisor path (CLHPATH) : /usr/bin/cloud-hypervisor
	firecracker hypervisor path (FCPATH) : /usr/bin/firecracker
	assets path (PKGDATADIR) : /usr/share/kata-containers
	shim path (PKGLIBEXECDIR) : /usr/libexec/kata-containers

     BUILD    /root/go/src/github.com/kata-containers/kata-containers/src/runtime/kata-runtime
     GENERATE config/configuration-qemu.toml
     GENERATE config/configuration-clh.toml
     GENERATE config/configuration-fc.toml
     BUILD    /root/go/src/github.com/kata-containers/kata-containers/src/runtime/containerd-shim-kata-v2
     BUILD    /root/go/src/github.com/kata-containers/kata-containers/src/runtime/kata-monitor
[root@microshift runtime]# make install
kata-runtime - version 3.1.0-alpha0 (commit 9bde32daa102368b9dbc27a6c03ed2e3e87d65e1)

• architecture:
	Host:
	golang:
	Build: arm64

• golang:
	go version go1.19.3 linux/arm64

• hypervisors:
	Default: qemu
	Known: acrn cloud-hypervisor firecracker qemu
	Available for this architecture: cloud-hypervisor firecracker qemu

• Summary:

	destination install path (DESTDIR) : /
	binary installation path (BINDIR) : /usr/local/bin
	binaries to install :
	 - /usr/local/bin/kata-runtime
	 - /usr/local/bin/containerd-shim-kata-v2
	 - /usr/local/bin/kata-monitor
	 - /usr/local/bin/data/kata-collect-data.sh
	configs to install (CONFIGS) :
	 - config/configuration-clh.toml
 	 - config/configuration-fc.toml
 	 - config/configuration-qemu.toml
	install paths (CONFIG_PATHS) :
	 - /usr/share/defaults/kata-containers/configuration-clh.toml
 	 - /usr/share/defaults/kata-containers/configuration-fc.toml
 	 - /usr/share/defaults/kata-containers/configuration-qemu.toml
	alternate config paths (SYSCONFIG_PATHS) :
	 - /etc/kata-containers/configuration-clh.toml
 	 - /etc/kata-containers/configuration-fc.toml
 	 - /etc/kata-containers/configuration-qemu.toml
	default install path for qemu (CONFIG_PATH) : /usr/share/defaults/kata-containers/configuration.toml
	default alternate config path (SYSCONFIG) : /etc/kata-containers/configuration.toml
	qemu hypervisor path (QEMUPATH) : /usr/bin/qemu-system-aarch64
	cloud-hypervisor hypervisor path (CLHPATH) : /usr/bin/cloud-hypervisor
	firecracker hypervisor path (FCPATH) : /usr/bin/firecracker
	assets path (PKGDATADIR) : /usr/share/kata-containers
	shim path (PKGLIBEXECDIR) : /usr/libexec/kata-containers

     INSTALL  install-scripts
     INSTALL  install-completions
     INSTALL  install-configs
     INSTALL  install-configs
     INSTALL  install-bin
     INSTALL  install-containerd-shim-v2
     INSTALL  install-monitor

The build creates the following:

  • runtime binary: /usr/local/bin/kata-runtime and /usr/local/bin/containerd-shim-kata-v2
  • configuration file: /usr/share/defaults/kata-containers/configuration.toml

Output:

[root@microshift runtime]# ls -las /usr/local/bin/kata-runtime /usr/local/bin/containerd-shim-kata-v2 /usr/share/defaults/kata-containers/configuration.toml
36588 -rwxr-xr-x. 1 root root 37459871 Nov 24 10:55 /usr/local/bin/containerd-shim-kata-v2
36164 -rwxr-xr-x. 1 root root 37063056 Nov 24 10:55 /usr/local/bin/kata-runtime
    0 lrwxrwxrwx. 1 root root       23 Nov 24 10:55 /usr/share/defaults/kata-containers/configuration.toml -> configuration-qemu.toml

Check hardware requirements

kata-runtime check --verbose # This will return error because vmlinux.container does not exist yet
which kata-runtime
kata-runtime --version
containerd-shim-kata-v2 --version

Output:

[root@microshift runtime]# kata-runtime check --verbose
ERRO[0000] /usr/share/defaults/kata-containers/configuration-qemu.toml: file /usr/bin/qemu-system-aarch64 does not exist  arch=arm64 name=kata-runtime pid=19912 source=runtime
/usr/share/defaults/kata-containers/configuration-qemu.toml: file /usr/bin/qemu-system-aarch64 does not exist

[root@microshift runtime]# which kata-runtime
/usr/local/bin/kata-runtime
[root@microshift runtime]# kata-runtime --version
kata-runtime  : 3.1.0-alpha0
   commit   : 9bde32daa102368b9dbc27a6c03ed2e3e87d65e1
   OCI specs: 1.0.2-dev
[root@microshift runtime]# containerd-shim-kata-v2 --version
Kata Containers containerd shim: id: "io.containerd.kata.v2", version: 3.1.0-alpha0, commit: 9bde32daa102368b9dbc27a6c03ed2e3e87d65e1

Kata creates a VM in which to run one or more containers by launching a hypervisor. Kata supports multiple hypervisors. We use QEMU. The hypervisor needs two assets for this task: a Linux kernel and a small root filesystem image to boot the VM. The guest kernel is passed to the hypervisor and used to boot the VM. The default kernel provided in Kata Containers is highly optimized for kernel boot time and minimal memory footprint, providing only those services required by a container workload. The hypervisor uses an image file which provides a minimal root filesystem used by the guest kernel to boot the VM and host the Kata Container. Kata Containers supports both initrd and rootfs based minimal guest images. The default packages provide both an image and an initrd, both of which are created using the osbuilder tool.

The initrd image is a compressed cpio(1) archive, created from a rootfs which is loaded into memory and used as part of the Linux startup process. During startup, the kernel unpacks it into a special instance of a tmpfs mount that becomes the initial root filesystem.

  • The runtime will launch the configured hypervisor.
  • The hypervisor will boot the mini-OS image using the guest kernel.
  • The kernel will start the init daemon as PID 1 (the agent) inside the VM root environment.
  • The agent will create a new container environment, setting its root filesystem to that requested by the user.
  • The agent will then execute the command inside the new container.

The rootfs image, sometimes referred to as the mini O/S, is a highly optimized container bootstrap system. With this,

  • The runtime will launch the configured hypervisor.
  • The hypervisor will boot the mini-OS image using the guest kernel.
  • The kernel will start the init daemon as PID 1 (systemd) inside the VM root environment.
  • systemd, running inside the mini-OS context, will launch the agent in the root context of the VM.
  • The agent will create a new container environment, setting its root filesystem to that requested by the user.
  • The agent will then execute the command inside the new container.

We will see the output of the qemu commands later.

Configure to use initrd image

Since, Kata containers can run with either an initrd image or a rootfs image, we will build both images but initially use the initrd. We will switch to rootfs in later section. So, make sure you add initrd = /usr/share/kata-containers/kata-containers-initrd.img in the configuration file /usr/share/defaults/kata-containers/configuration.toml and comment out the default image line with the following:

sudo mkdir -p /etc/kata-containers/
sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
sudo sed -i 's/^\(image =.*\)/# \1/g' /etc/kata-containers/configuration.toml

The initrd line is not added by default, so add the initrd line in /etc/kata-containers/configuration.toml so that it looks as follows:

# image = "/usr/share/kata-containers/kata-containers.img"
initrd = "/usr/share/kata-containers/kata-containers-initrd.img"

Next, we create the initrd image and the rootfs image. One of the initrd and image options in Kata runtime config file must be set, but not both. The main difference between the options is that the size of initrd (10MB+) is significantly smaller than rootfs image (100MB+).

Create an initrd image

Create a local rootfs for initrd image

yum -y install podman buildah skopeo # If not already installed
sudo ln -s `which podman` /usr/bin/docker # or run later commands with USE_PODMAN instead of USE_DOCKER
export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs"
sudo rm -rf ${ROOTFS_DIR}
cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
./rootfs.sh -l

Output:

[root@microshift runtime]# cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
[root@microshift rootfs-builder]# ./rootfs.sh -l
alpine
centos
clearlinux
debian
ubuntu

Let’s use the distro=ubuntu

export distro=ubuntu
script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_PODMAN=true ./rootfs.sh ${distro}'

This will install rust in the container, download and compile numerous crates using cargo and build the kata-agent (in Rust). When complete, you will see the rootfs directory.

[root@microshift rootfs-builder]# ls rootfs
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

If you get errors such as following, just run the above command again after some time.

Could not connect to ports.ubuntu.com:80 (185.125.190.39), connection timed out

Build an initrd image

cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder
script -fec 'sudo -E AGENT_INIT=yes USE_PODMAN=true ./initrd_builder.sh ${ROOTFS_DIR}'

Output:

[root@microshift rootfs-builder]# cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder
[root@microshift initrd-builder]# script -fec 'sudo -E AGENT_INIT=yes USE_PODMAN=true ./initrd_builder.sh ${ROOTFS_DIR}'
Script started, output log file is 'typescript'.
[OK] init is installed
[OK] Agent is installed
INFO: Creating /root/go/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder/kata-containers-initrd.img based on rootfs at /root/go/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
136212 blocks
Script done.

Install the initrd image

commit=$(git log --format=%h -1 HEAD)
date=$(date +%Y-%m-%d-%T.%N%z)
image="kata-containers-initrd-${date}-${commit}"
sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}"
(cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)

Output:

[root@microshift initrd-builder]# commit=$(git log --format=%h -1 HEAD)
[root@microshift initrd-builder]# date=$(date +%Y-%m-%d-%T.%N%z)
[root@microshift initrd-builder]# image="kata-containers-initrd-${date}-${commit}"
[root@microshift initrd-builder]# sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}"
[root@microshift initrd-builder]# (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)

Create a rootfs image

Create a local rootfs for rootfs image

export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs
sudo rm -rf ${ROOTFS_DIR}
cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder
script -fec 'sudo -E GOPATH=$GOPATH USE_PODMAN=true ./rootfs.sh ${distro}'

Build a rootfs image

cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/image-builder
script -fec 'sudo -E USE_PODMAN=true ./image_builder.sh ${ROOTFS_DIR}'

Install the rootfs image

commit=$(git log --format=%h -1 HEAD)
date=$(date +%Y-%m-%d-%T.%N%z)
image="kata-containers-${date}-${commit}"
sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}"
(cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)

Build Kata Containers Kernel

The process to build a kernel for Kata Containers is automated.

yum -y install flex bison bc
go env -w GO111MODULE=auto
go get github.com/kata-containers/packaging
cd $GOPATH/src/github.com/kata-containers/packaging/kernel

The script ./build-kernel.sh tries to apply the patches from ${GOPATH}/src/github.com/kata-containers/packaging/kernel/patches/ when it sets up a kernel. If you want to add a source modification, add a patch on this directory. The script also copies or generates a kernel config file from ${GOPATH}/src/github.com/kata-containers/packaging/kernel/configs/ to .config in the kernel source code. You can modify it as needed. We will use the defaults.

./build-kernel.sh setup

Output:

[root@microshift image-builder]# go env -w GO111MODULE=auto
[root@microshift image-builder]# go get github.com/kata-containers/packaging
package github.com/kata-containers/packaging: no Go files in /root/go/src/github.com/kata-containers/packaging
[root@microshift image-builder]# cd $GOPATH/src/github.com/kata-containers/packaging/kernel
[root@microshift kernel]# ./build-kernel.sh setup
package github.com/kata-containers/tests: no Go files in /root/go/src/github.com/kata-containers/tests
~/go/src/github.com/kata-containers/tests ~/go/src/github.com/kata-containers/packaging/kernel
~/go/src/github.com/kata-containers/packaging/kernel
versions file (versions-master.yaml) does not exist
Download from https://raw.githubusercontent.com/kata-containers/runtime/master/versions.yaml
INFO: Config version: 92
INFO: Kernel version: 5.4.60
INFO: kernel path does not exist, will download kernel
INFO: Download kernel checksum file: sha256sums.asc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  266k  100  266k    0     0   321k      0 --:--:-- --:--:-- --:--:--  321k
INFO: Download kernel version 5.4.60
INFO: Download kernel
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   162  100   162    0     0    207      0 --:--:-- --:--:-- --:--:--   207
100  104M  100  104M    0     0  7745k      0  0:00:13  0:00:13 --:--:-- 9237k
linux-5.4.60.tar.xz: OK
INFO: Apply patches from /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x
INFO: Found 7 patches
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0001-9p-retrieve-fid-from-file-when-file-instance-exist.patch
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0001-NO-UPSTREAM-9P-always-use-cached-inode-to-fill-in-v9.patch
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0002-ACPI-Always-build-evged-in.patch
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0002-net-virtio_vsock-Fix-race-condition-between-bind-and.patch
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0003-arm-arm64-Provide-a-wrapper-for-SMCCC-1.1-calls.patch
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0004-arm-arm64-smccc-psci-add-arm_smccc_1_1_get_conduit.patch
INFO: Apply /root/go/src/github.com/kata-containers/packaging/kernel/patches//5.4.x/0006-arm64-mm-Enable-memory-hot-remove.patch
ls: cannot access '/root/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/arm64/../common/experimental/*.conf': No such file or directory
INFO: Constructing config from fragments: /root/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/arm64/.config
INFO: Some CONFIG elements failed to make the final .config:
INFO: Value requested for CONFIG_MEMORY_HOTREMOVE not in final .config
Value requested for CONFIG_ZONE_DEVICE not in final .config
Value requested for CONFIG_DEV_PAGEMAP_OPS not in final .config
Value requested for CONFIG_ND_PFN not in final .config
Value requested for CONFIG_NVDIMM_PFN not in final .config
Value requested for CONFIG_NVDIMM_DAX not in final .config
INFO: Generated config file can be found in /root/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/arm64/.config
ERROR: Failed to construct requested .config file
ERROR: failed to find default config

After the kernel source code is ready, we build the kernel

cp /root/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/arm64/.config kata-linux-5.4.60-92/.config
./build-kernel.sh build

Output:

[root@microshift kernel]# cp /root/go/src/github.com/kata-containers/packaging/kernel/configs/fragments/arm64/.config kata-linux-5.4.60-92/.config
[root@microshift kernel]# cd kata-linux-5.4.60-92
[root@microshift kata-linux-5.4.60-92]# make oldconfig
scripts/kconfig/conf  --oldconfig Kconfig
#
# No change to .config
#
[root@microshift kata-linux-5.4.60-92]# cd .. 
[root@microshift kernel]# ./build-kernel.sh build
…

Install the kernel to the default Kata containers path (/usr/share/kata-containers/)

./build-kernel.sh install

Output:

[root@microshift kernel]# ./build-kernel.sh install
package github.com/kata-containers/tests: no Go files in /root/go/src/github.com/kata-containers/tests
~/go/src/github.com/kata-containers/tests ~/go/src/github.com/kata-containers/packaging/kernel
~/go/src/github.com/kata-containers/packaging/kernel
INFO: Config version: 92
INFO: Kernel version: 5.4.60
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
lrwxrwxrwx. 1 root root 17 Nov 24 15:17 /usr/share/kata-containers/vmlinux.container -> vmlinux-5.4.60-92
lrwxrwxrwx. 1 root root 17 Nov 24 15:17 /usr/share/kata-containers/vmlinuz.container -> vmlinuz-5.4.60-92

The /etc/kata-containers/configuration.toml has the following:

# Path to vhost-user-fs daemon.
virtio_fs_daemon = "/usr/libexec/virtiofsd"

Check the output kata-runtime:

[root@microshift kernel]# kata-runtime check --verbose
ERRO[0000] /etc/kata-containers/configuration.toml: file /usr/bin/qemu-system-aarch64 does not exist  arch=arm64 name=kata-runtime pid=205921 source=runtime
/etc/kata-containers/configuration.toml: file /usr/bin/qemu-system-aarch64 does not exist 

Let’s fix this with:

ln -s /usr/libexec/qemu-kvm /usr/bin/qemu-system-aarch64 

Output:

[root@microshift kernel]# ln -s /usr/libexec/qemu-kvm /usr/bin/qemu-system-aarch64
[root@microshift kernel]# kata-runtime check --verbose
WARN[0000] Not running network checks as super user      arch=arm64 name=kata-runtime pid=206229 source=runtime
INFO[0000] Unable to know if the system is running inside a VM  arch=arm64 source=virtcontainers/hypervisor
INFO[0000] kernel property found                         arch=arm64 description="Kernel-based Virtual Machine" name=kvm pid=206229 source=runtime type=module
INFO[0000] kernel property found                         arch=arm64 description="Host kernel accelerator for virtio" name=vhost pid=206229 source=runtime type=module
INFO[0000] kernel property found                         arch=arm64 description="Host kernel accelerator for virtio network" name=vhost_net pid=206229 source=runtime type=module
INFO[0000] kernel property found                         arch=arm64 description="Host Support for Linux VM Sockets" name=vhost_vsock pid=206229 source=runtime type=module
System is capable of running Kata Containers
INFO[0000] device available                              arch=arm64 check-type=full device=/dev/kvm name=kata-runtime pid=206229 source=runtime
INFO[0000] feature available                             arch=arm64 check-type=full feature=create-vm name=kata-runtime pid=206229 source=runtime
INFO[0000] kvm extension is supported                    arch=arm64 description="Maximum IPA shift supported by the host" id=165 name=KVM_CAP_ARM_VM_IPA_SIZE pid=206229 source=runtime type="kvm extension"
INFO[0000] IPA limit size: 44 bits.                      arch=arm64 name=KVM_CAP_ARM_VM_IPA_SIZE pid=206229 source=runtime type="kvm extension"
System can currently create Kata Containers

Check the hypervisor.qemu section in configuration.toml:

[root@microshift kernel]# cat /etc/kata-containers/configuration.toml | awk -v RS= '/\[hypervisor.qemu\]/'
[hypervisor.qemu]
path = "/usr/bin/qemu-system-aarch64"
kernel = "/usr/share/kata-containers/vmlinux.container"
# image = "/usr/share/kata-containers/kata-containers.img"
initrd = "/usr/share/kata-containers/kata-containers-initrd.img"
machine_type = "virt"

Check the initrd image (kata-containers-initrd.img), the rootfs image (kata-containers.img), and the kernel in the /usr/share/kata-containers directory:

[root@microshift kernel]# ls -las /usr/share/kata-containers
total 171696
     4 drwxr-xr-x.   2 root root      4096 Nov 24 15:17 .
     4 drwxr-xr-x. 128 root root      4096 Nov 24 14:20 ..
    68 -rw-r--r--.   1 root root     68536 Nov 24 15:17 config-5.4.60
131072 -rw-r-----.   1 root root 134217728 Nov 24 14:20 kata-containers-2022-11-24-14:19:58.932179446+0000-9bde32daa
 26144 -rw-r-----.   1 root root  26770481 Nov 24 12:48 kata-containers-initrd-2022-11-24-12:48:10.440333043+0000-9bde32daa
     4 lrwxrwxrwx.   1 root root        67 Nov 24 12:48 kata-containers-initrd.img -> kata-containers-initrd-2022-11-24-12:48:10.440333043+0000-9bde32daa
     4 lrwxrwxrwx.   1 root root        60 Nov 24 14:20 kata-containers.img -> kata-containers-2022-11-24-14:19:58.932179446+0000-9bde32daa
  9820 -rw-r--r--.   1 root root  10246656 Nov 24 15:17 vmlinux-5.4.60-92
     0 lrwxrwxrwx.   1 root root        17 Nov 24 15:17 vmlinux.container -> vmlinux-5.4.60-92
  4576 -rw-r--r--.   1 root root   4684131 Nov 24 15:17 vmlinuz-5.4.60-92
     0 lrwxrwxrwx.   1 root root        17 Nov 24 15:17 vmlinuz.container -> vmlinuz-5.4.60-92

The kernel file is called vmlinuz-version. vmlinuz is the name of the Linux kernel executable. vmlinuz is a compressed Linux kernel, and it can load the operating system into memory so that the computer becomes usable and application programs can be run. When virtual memory was developed for easier multitasking abilities, “vm” was put at the front of the file to show that the kernel supports virtual memory. For a while the Linux kernel was called vmlinux, but the kernel grew too large to fit in the available boot memory, so the kernel image was compressed, and the ending x was changed to a z to show it was compressed with zlib compression. This same compression isn’t always used, often replaced with LZMA or BZIP2, and some kernels are simply called zImage.

At the head of this kernel image (vmlinuz) is a routine that does some minimal amount of hardware setup and then decompresses the kernel contained within the kernel image and places it into high memory. If an initial RAM disk image (initrd) is present, this routine moves it into memory (or we can say extract the compressed ramdisk image into the real memory) and notes it for later use. The routine then calls the kernel, and the kernel boot begins. The initial RAM disk (initrd) is an initial root file system that is mounted prior to when the real rootfile system is available. The initrd is bound to the kernel and loaded as part of the kernel boot procedure. The kernel then mounts this initrd as part of the two-stage boot process to load the modules to make the real file systems available and get at the real root file system. The initrd contains a minimal set of directories and executables to achieve this, such as the insmod tool to install kernel modules into the kernel.

Many Linux distributions ship a single, generic Linux kernel image – one that the distribution's developers create specifically to boot on a wide variety of hardware. The device drivers for this generic kernel image are included as loadable kernel modules because statically compiling many drivers into one kernel causes the kernel image to be much larger, perhaps too large to boot on computers with limited memory.

Create the file /etc/crio/crio.conf.d/50-kata

cat > /etc/crio/crio.conf.d/50-kata << EOF
[crio.runtime.runtimes.kata]
  runtime_path = "/usr/local/bin/containerd-shim-kata-v2"
  runtime_root = "/run/vc"
  runtime_type = "vm"
  privileged_without_host_devices = true
EOF

Restart crio and start microshift

systemctl restart crio

systemctl start microshift
export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig

Running some Kata samples

After MicroShift is started, you can apply the kata runtimeclass and run the samples.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/kata/
oc apply -f kata-runtimeclass.yaml
# Start three kata pods
oc apply -f kata-nginx.yaml -f kata-alpine.yaml  -f kata-busybox.yaml

watch "oc get nodes;oc get pods -A;crictl stats -a"

Output:

NAME                     STATUS   ROLES    AGE   VERSION
microshift.example.com   Ready    <none>   19h   v1.21.0

NAMESPACE                       NAME                                  READY   STATUS    RESTARTS   AGE
default                         busybox-1                             1/1     Running   0          94s
default                         kata-alpine                           1/1     Running   0          94s
default                         kata-nginx                            1/1     Running   0          94s
kube-system                     console-deployment-768d7dc869-98r5k   1/1     Running   0          17h
kube-system                     kube-flannel-ds-s4hxb                 1/1     Running   0          19h
kubevirt-hostpath-provisioner   kubevirt-hostpath-provisioner-l8vtl   1/1     Running   0          19h
openshift-dns                   dns-default-dx677                     2/2     Running   0          19h
openshift-dns                   node-resolver-5kprs                   1/1     Running   0          19h
openshift-ingress               router-default-85bcfdd948-72llb       1/1     Running   0          19h
openshift-service-ca            service-ca-7764c85869-phtkw           1/1     Running   0          19h

CONTAINER           CPU %               MEM                 DISK                INODES
003c4997d33d9       0.00                458.8kB             12B                 16
0488ddd1ab858       0.00                0B                  12B                 7
1c91ca2f48c0b       0.00                0B                  13.35kB             22
55d22d7d816d0       0.00                0B                  6.969kB             11
88705e9cce543       0.00                0B                  0B                  4
b4cf5fdc56077       1.26                704.5kB             89B                 8
d0733e7bd2a99       0.00                0B                  12B                 18
e353d5e805e36       0.00                0B                  0B                  3
eca8f7f3ae8a2       0.00                0B                  138B                15
f10185631fc64       0.00                2.449MB             1.225kB             21
f6b155e057a37       0.00                0B                  12B                 19

We can see that the kernel used in the kata containers (when we created the initrd image is 5.4.60) is different from that host 5.15.74 on Host

[root@microshift kata]# kata-runtime kata-env | grep "\[Kernel\]\|\[Host\]" -A 2
[Kernel]
  Path = "/usr/share/kata-containers/vmlinux-5.4.60-92"
  Parameters = "scsi_mod.scan=none"
--
[Host]
  Kernel = "5.15.74-v8.1.el9"
  Architecture = "arm64" 

[root@microshift kata]# oc exec -it kata-nginx -- uname -a
Linux kata-nginx 5.4.60 #1 SMP Thu Nov 24 14:33:21 UTC 2022 aarch64 GNU/Linux
[root@microshift kata]# oc exec -it kata-alpine -- uname -a
Linux kata-alpine 5.4.60 #1 SMP Thu Nov 24 14:33:21 UTC 2022 aarch64 Linux
[root@microshift kata]# oc exec -it busybox-1 -- uname -a
Linux busybox-1 5.4.60 #1 SMP Thu Nov 24 14:33:21 UTC 2022 aarch64 GNU/Linux

[root@microshift kata]# uname -a
Linux microshift.example.com 5.15.74-v8.1.el9 #1 SMP PREEMPT Thu Oct 27 08:34:02 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux 

Try to run ping from the kata-alpine container, it does not work because the capabilities are not set

[root@microshift kata]# oc exec -it kata-alpine -- ping -c2 google.com
PING google.com (142.251.40.206): 56 data bytes
ping: permission denied (are you root?)
command terminated with exit code 1 
[root@rpi kata]# oc exec -it busybox-1 -- ping 10-42-0-6.default.pod.cluster.local
PING 10-42-0-6.default.pod.cluster.local (10.42.0.6): 56 data bytes
ping: permission denied (are you root?)
command terminated with exit code 1

Add the registries line under [crio.image] for default registry to download an image from and the NET_RAW to the default_capabilities to /etc/crio/crio.conf so that we can run ping from within the containers. Delete the kata containers, restart crio and microshift and recreate the kata containers.

[crio]
[crio.image]
registries=["quay.io","docker.io"]

# The crio.runtime table contains settings pertaining to the OCI runtime used
# and options for how to set up and manage the OCI runtime.
[crio.runtime]

default_capabilities = [
        "CHOWN",
        "DAC_OVERRIDE",
        "FSETID",
        "FOWNER",
        "SETGID",
        "SETUID",
        "SETPCAP",
        "NET_BIND_SERVICE",
        "KILL",
        "NET_RAW",
]

# If true, SELinux will be used for pod separation on the host.
selinux = true

Check that we can ping from the kata-alpine container if you set the default_capabilities "NET_RAW" in /etc/crio/crio.conf as shown in Part 23

[root@microshift kata]# oc delete -f kata-alpine.yaml -f kata-busybox.yaml -f kata-nginx.yaml
pod "kata-alpine" deleted
pod "busybox-1" deleted
pod "kata-nginx" deleted 
[root@microshift kata]# systemctl restart crio 
[root@microshift kata]# systemctl restart microshift
[root@microshift kata]# watch "oc get nodes;oc get pods -A;crictl stats -a" 
[root@microshift kata]# oc apply -f kata-alpine.yaml 
pod/kata-alpine created 
[root@microshift kata]# oc exec -it kata-alpine -- ping -c2 google.com
PING google.com (142.250.176.206): 56 data bytes
64 bytes from 142.250.176.206: seq=0 ttl=59 time=5.930 ms
64 bytes from 142.250.176.206: seq=1 ttl=59 time=4.972 ms

--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 4.972/5.451/5.930 ms

When done, you may delete the sample deployments

[root@microshift kata]# oc delete -f kata-alpine.yaml
pod "kata-alpine" deleted 

Influxdb Sample

We execute the runall-balena-dynamic.sh for AlmaLinux 9 after updating the deployment yamls to use the runtimeclass: kata.

cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/influxdb/

Update the influxdb-deployment.yaml, telegraf-deployment.yaml and grafana/grafana-deployment.yaml to use the runtimeClassName: kata. With Kata containers, we do not directly get access to the host devices. So, we run the measure container as a runc pod. In runc, '--privileged' for a container means all the /dev/* block devices from the host are mounted into the guest. This will allow the privileged container to gain access to mount any block device from the host.

sed -i '/^    spec:/a \ \ \ \ \ \ runtimeClassName: kata' influxdb-deployment.yaml telegraf-deployment.yaml grafana/grafana-deployment.yaml

Now, get the nodename

[root@microshift influxdb]# oc get nodes
NAME                     STATUS   ROLES    AGE   VERSION
microshift.example.com   Ready    <none>   19h   v1.21.0

Replace the annotation kubevirt.io/provisionOnNode with the above nodename rpi.example.com and execute the runall-balena-dynamic.sh. This will create a new project influxdb.

nodename=microshift.example.com
sed -i "s|kubevirt.io/provisionOnNode:.*| kubevirt.io/provisionOnNode: $nodename|" influxdb-data-dynamic.yaml
sed -i "s| kubevirt.io/provisionOnNode:.*| kubevirt.io/provisionOnNode: $nodename|" grafana/grafana-data-dynamic.yaml

./runall-balena-dynamic.sh

Let’s watch the stats (CPU%, Memory, Disk and Inodes) of the kata container pods:

watch "oc get nodes;oc get pods;crictl stats"

Output:

NAME                     STATUS   ROLES    AGE   VERSION
microshift.example.com   Ready    <none>   20h   v1.21.0

NAME                                   READY   STATUS    RESTARTS   AGE
grafana-855ffb48d8-z9m4q               1/1     Running   0          16m
influxdb-deployment-6d898b7b7b-tts8x   1/1     Running   0          17m
measure-deployment-58cddb5745-22mm2    1/1     Running   2          17m
telegraf-deployment-d746f5c6-rfvcp     1/1     Running   0          16m

CONTAINER           CPU %               MEM                 DISK                INODES
160167964a6bb       0.05                24.35MB             4.026MB             70
1f3424266a3bf       0.10                12.08MB             186kB               11
ed155feea1586       0.79                20.58MB             265B                11

We can look at the RUNTIME_CLASS using custom columns:

[root@microshift influxdb]# oc get pods -o custom-columns=NAME:metadata.name,STATUS:.status.phase,RUNTIME_CLASS:.spec.runtimeClassName,IP:.status.podIP,IMAGE:.status.containerStatuses[].image -A
NAME                                   STATUS    RUNTIME_CLASS   IP              IMAGE
grafana-855ffb48d8-z9m4q               Running   kata            10.42.0.16      docker.io/grafana/grafana:5.4.3
influxdb-deployment-6d898b7b7b-tts8x   Running   kata            10.42.0.13      docker.io/library/influxdb:1.7.4
measure-deployment-58cddb5745-22mm2    Running   <none>          10.42.0.14      docker.io/karve/measure:latest
telegraf-deployment-d746f5c6-rfvcp     Running   kata            10.42.0.15      docker.io/library/telegraf:1.10.0
console-deployment-768d7dc869-98r5k    Running   <none>          10.42.0.5       docker.io/karve/console:latest
kube-flannel-ds-s4hxb                  Running   <none>          192.168.1.209   quay.io/microshift/flannel:4.8.0-0.okd-2021-10-10-030117
kubevirt-hostpath-provisioner-l8vtl    Running   <none>          10.42.0.2       quay.io/microshift/hostpath-provisioner:4.8.0-0.okd-2021-10-10-030117
dns-default-dx677                      Running   <none>          10.42.0.3       quay.io/microshift/coredns:4.8.0-0.okd-2021-10-10-030117
node-resolver-5kprs                    Running   <none>          192.168.1.209   quay.io/microshift/cli:4.8.0-0.okd-2021-10-10-030117
router-default-85bcfdd948-72llb        Running   <none>          192.168.1.209   quay.io/microshift/haproxy-router:4.8.0-0.okd-2021-10-10-030117
service-ca-7764c85869-phtkw            Running   <none>          10.42.0.4       quay.io/microshift/service-ca-operator:4.8.0-0.okd-2021-10-10-030117

Check the qemu process. We used the initrd image and you can see that in the parameters:

ps -ef | grep qemu | grep initrd

Output:

root      222931       1  2 16:18 ?        00:00:30 /usr/libexec/qemu-kvm -name sandbox-1158f4bbe993c6de6e620ba6eb0e1e485441ac329c823990b04ba86865b78147 -uuid 292b9c8e-04cc-4741-917a-09881a010bdd -machine virt,usb=off,accel=kvm,gic-version=host -cpu host,pmu=off -qmp unix:/run/vc/vm/1158f4bbe993c6de6e620ba6eb0e1e485441ac329c823990b04ba86865b78147/qmp.sock,server=on,wait=off -m 2048M,slots=10,maxmem=7812M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=off,addr=2,io-reserve=4k,mem-reserve=1m,pref64-reserve=1m -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/1158f4bbe993c6de6e620ba6eb0e1e485441ac329c823990b04ba86865b78147/console.sock,server=on,wait=off -device virtio-scsi-pci,id=scsi0,disable-modern=false -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-3224903307,guest-cid=3224903307 -chardev socket,id=char-1312d5ca71445b71,path=/run/vc/vm/1158f4bbe993c6de6e620ba6eb0e1e485441ac329c823990b04ba86865b78147/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-1312d5ca71445b71,tag=kataShared,queue-size=1024 -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=26:b4:88:ca:fd:5a,disable-modern=false,mq=on,vectors=4 -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinux-5.4.60-92 -initrd /usr/share/kata-containers/kata-containers-initrd-2022-11-24-12:48:10.440333043+0000-9bde32daa -append iommu.passthrough=0 console=hvc0 console=hvc1 quiet panic=1 nr_cpus=4 scsi_mod.scan=none -pidfile /run/vc/vm/1158f4bbe993c6de6e620ba6eb0e1e485441ac329c823990b04ba86865b78147/pid -smp 1,cores=1,threads=1,sockets=4,maxcpus=4

Add the "<RaspberryPiIPAddress> grafana-service-influxdb.cluster.local" to /etc/hosts on your laptop and login to http://grafana-service-influxdb.cluster.local/login using admin/admin. You may change the password on first login or click skip. Go to the Dashboards list (left menu > Dashboards > Manage). Open the Analysis Server dashboard to display monitoring information for MicroShift. Open the Balena Sense dashboard to show the temperature, pressure, and humidity from SenseHat.

Finally, after you are done working with this sample, you can run the deleteall-balena-dynamic.sh

./deleteall-balena-dynamic.sh

Deleting the persistent volume claims automatically deletes the persistent volumes.

Configure to use the rootfs image

We have been using the initrd image when running the samples above, now let’s switch to the rootfs image instead of using initrd by changing the following lines in /etc/kata-containers/configuration.toml

image = "/usr/share/kata-containers/kata-containers.img"
#initrd = "/usr/share/kata-containers/kata-containers-initrd.img"

Also disable the image nvdimm by setting the following:

disable_image_nvdimm = true # Default is false

Restart crio and test with the kata-alpine sample

systemctl restart crio
cd ~/microshift/raspberry-pi/kata/
oc apply -f kata-alpine.yaml

Output of qemu process when we use rootfs image with disable_image_nvdimm=true

root      233569       1 28 16:44 ?        00:00:05 /usr/libexec/qemu-kvm -name sandbox-e8806511f0f6564d56417fb96a17f354946ce0f5fc783aeb38e09030124d0d7b -uuid b07b2f1d-0f7a-4d96-b712-2142ca0bb0bf -machine virt,usb=off,accel=kvm,gic-version=host -cpu host,pmu=off -qmp unix:/run/vc/vm/e8806511f0f6564d56417fb96a17f354946ce0f5fc783aeb38e09030124d0d7b/qmp.sock,server=on,wait=off -m 2048M,slots=10,maxmem=7812M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=off,addr=2,io-reserve=4k,mem-reserve=1m,pref64-reserve=1m -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/e8806511f0f6564d56417fb96a17f354946ce0f5fc783aeb38e09030124d0d7b/console.sock,server=on,wait=off -device virtio-blk-pci,disable-modern=false,drive=image-91f5e6bf52b9981d,scsi=off,config-wce=off,share-rw=on,serial=image-91f5e6bf52b9981d -drive id=image-91f5e6bf52b9981d,file=/usr/share/kata-containers/kata-containers-2022-11-24-14:19:58.932179446+0000-9bde32daa,aio=threads,format=raw,if=none,readonly=on -device virtio-scsi-pci,id=scsi0,disable-modern=false -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-699490259,guest-cid=699490259 -chardev socket,id=char-d491e2de85516dea,path=/run/vc/vm/e8806511f0f6564d56417fb96a17f354946ce0f5fc783aeb38e09030124d0d7b/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-d491e2de85516dea,tag=kataShared,queue-size=1024 -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=2e:ed:25:a4:da:c2,disable-modern=false,mq=on,vectors=4 -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinux-5.4.60-92 -append iommu.passthrough=0 root=/dev/vda1 rootflags=data=ordered,errors=remount-ro ro rootfstype=ext4 console=hvc0 console=hvc1 quiet systemd.show_status=false panic=1 nr_cpus=4 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none -pidfile /run/vc/vm/e8806511f0f6564d56417fb96a17f354946ce0f5fc783aeb38e09030124d0d7b/pid -smp 1,cores=1,threads=1,sockets=4,maxcpus=4

We can also run MicroShift Containerized as shown in Part 18 and execute the Jupyter Notebook samples for Digit Recognition, Object Detection and License Plate Recognition with Kata containers as shown in Part 23.

Error Messages in MicroShift Logs

Message: failed to fetch hugetlb info

The following journalctl command will continuously show warning messages “failed to fetch hugetlb info”. The default kernel for the Raspberry Pi OS does not support HugeTLB hugepages.

journalctl -u microshift -f

Output:

...
May 23 11:49:20 rpi.example.com microshift[3112]: W0523 11:49:20.347604    3112 container.go:586] Failed to update stats for container "/system.slice/crio-00c0b63eeee509979e7652cca8b91a1e9e900c1989b7fe54f6a05f6591de0108.scope": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
May 23 11:49:20 rpi.example.com microshift[3112]: failed to fetch hugetlb info
...

To remove these messages, we can recompile the microshift binary using the changes from hugetlb.go.

dnf -y install wget pkg-config

# Install golang 1.17.2 (Do not use 1.18.x or 1.19.x)
wget https://golang.org/dl/go1.17.2.linux-arm64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.2.linux-arm64.tar.gz
rm -f go1.17.2.linux-arm64.tar.gz
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/root/go
cat << EOF >> /root/.bashrc
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/root/go
EOF
mkdir $GOPATH

git clone https://github.com/thinkahead/microshift.git
cd microshift

Edit the file vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go and remove the return on err.

func statHugeTlb(dirPath string, stats *cgroups.Stats) error {
        hugePageSizes, _ := cgroups.GetHugePageSize()
        //hugePageSizes, err := cgroups.GetHugePageSize()
        //if err != nil {
        //        return errors.Wrap(err, "failed to fetch hugetlb info")
        //}
        hugetlbStats := cgroups.HugetlbStats{}
Build and replace the microshift binary. Restart MicroShift.
make
# Check which binary the /usr/lib/systemd/system/microshift.service points to
mv microshift /usr/bin/.
#mv microshift /usr/local/bin/.
restorecon -R -v /usr/bin/microshift # See below
systemctl restart microshift

Permission denied in journalctl microshift logs when you restart microshift

You will see the following errors when you copy the microshift binary over to /usr/bin

[root@microshift microshift]# systemctl restart microshift 
[root@microshift microshift]# journalctl -u microshift -f
Nov 23 21:05:20 microshift.example.com systemd[1]: Started MicroShift.
Nov 23 21:05:20 microshift.example.com systemd[53453]: microshift.service: Failed to locate executable /usr/bin/microshift: Permission denied
Nov 23 21:05:20 microshift.example.com systemd[53453]: microshift.service: Failed at step EXEC spawning /usr/bin/microshift: Permission denied 

Fix it by restoring the default context of /usr/bin/microshift

[root@microshift microshift]# ls -Z /usr/bin/microshift
unconfined_u:object_r:user_tmp_t:s0 /usr/bin/microshift
[root@microshift microshift]# restorecon -R -v /usr/bin/microshift
Relabeled /usr/bin/microshift from unconfined_u:object_r:user_tmp_t:s0 to unconfined_u:object_r:container_runtime_exec_t:s0
[root@microshift microshift]# ls -Z /usr/bin/microshift
unconfined_u:object_r:container_runtime_exec_t:s0 /usr/bin/microshift

Unable to fetch pod log stats

Deleted pods are not properly reaped on nodes. The observed behavior is that after deletion, we see the pods added to a list of recurring events that keeps spamming the logs. This happens with CRI-O >= 1.22.

Nov 23 22:55:51 microshift.example.com microshift[54264]: E1123 22:55:51.898173   54264 cadvisor_stats_provider.go:147] "Unable to fetch pod log stats" err="open /var/log/pods/kubevirt_kubevirt-3993aff5e8e705c86a9c9145f9953a43e32e0468-jobzz9pgwd7g5_9c9d77aa-5feb-45e1-9364-6aac7357b337: no such file or directory" pod="kubevirt/kubevirt-3993aff5e8e705c86a9c9145f9953a43e32e0468-jobzz9pgwd7g5"
Nov 23 22:55:51 microshift.example.com microshift[54264]: E1123 22:55:51.898695   54264 cadvisor_stats_provider.go:147] "Unable to fetch pod log stats" err="open /var/log/pods/influxdb_measure-deployment-58cddb5745-7nkmv_46948320-778e-4f3c-8246-20dba7c6bd44: no such file or directory" pod="influxdb/measure-deployment-58cddb5745-7nkmv"
Nov 23 22:55:51 microshift.example.com microshift[54264]: E1123 22:55:51.899081   54264 cadvisor_stats_provider.go:147] "Unable to fetch pod log stats" err="open /var/log/pods/kubevirt_virt-operator-849f75c557-c5njq_4e482dc8-6997-44d5-880e-c2bf0bf499ab: no such file or directory" pod="kubevirt/virt-operator-849f75c557-c5njq"

The fix will require making changes as mentioned at https://github.com/kubernetes/kubernetes/pull/108855/files and rebuilding the microshift binary.

Conclusion

In this Part 26, we saw multiple options to run MicroShift on the Raspberry Pi 4 with the AlmaLinux 9 (64 bit). We used dynamic persistent volumes to install InfluxDB/Telegraf/Grafana with a dashboard to show SenseHat sensor data. We ran samples that used the Sense Hat/USB camera and worked with a sample that sent the pictures and web socket messages to Node Red when a person was detected. We ran a sample with MongoDB. We created Virtual Machines, installed and used CDI to upload images, installed the OKD Web Console and saw how to connect to a Virtual Machine Instance using KubeVirt. Finally, we built and ran Kata containers. In the next Part 27 we will work with Oracle Linux 9.

Hope you have enjoyed the article. Share your thoughts in the comments or engage in the conversation with me on Twitter @aakarve. I look forward to hearing about your use of MicroShift on ARM devices and if you would like to see something covered in more detail.

References


#MicroShift#Openshift​​#containers#crio#Edge#raspberry-pi#almalinux #mongodb #cdi

​​​ ​​
0 comments
19 views

Permalink