MicroShift and KubeVirt on Raspberry Pi 4 with Oracle Linux 8.5
Introduction
MicroShift is a research project that is exploring how OpenShift OKD Kubernetes distribution can be optimized for small form factor devices and edge computing. In Part 1 we looked at multiple ways to run MicroShift on a MacBook Pro. In Part 4, we ran MicroShift on the Raspberry Pi 4 with the Raspberry Pi OS (64 bit) and further in Part 9, we looked at Virtualization with MicroShift on Raspberry Pi 4 with Raspberry Pi OS (64 bit). In Part 5, we saw multiple options to build and run MicroShift on the Raspberry Pi 4 with the CentOS 8 Stream (64 bit). In Part 6, we deployed MicroShift on the Raspberry Pi 4 with Ubuntu 20.04 (64 bit) and in Part 13 with Ubuntu 22.04 (64 bit). In Part 8, we looked at the All-In-One install of MicroShift on balenaOS. In Part 10, Part 11, and Part 12, we deployed MicroShift and KubeVirt on Fedora IoT, Fedora Server and Fedora CoreOS respectively, Part 14 on Rocky Linux and Part 15 on openSUSE. In this Part 16, we will work with MicroShift on Oracle Linux. We will run an object detection sample and send messages to Node Red installed on MicroShift. Further, we will setup KubeVirt and the OKD Web Console and run Virtual Machine Instances in MicroShift.
Oracle provides an Oracle Linux (aarch64) installation image as a technology preview for developer use only, that is specifically designed to run on a variety of Raspberry Pi models. The installation image that is provided is a default installation of Oracle Linux (aarch64) into a raw disk image that can be cloned, block-by-block, to an SD Card for an immediate boot. Btrfs is the default file system that is used in the image.
Setting up the Raspberry Pi 4 with Oracle Linux
Run the following steps to download the Oracle Linux image and setup the Raspberry Pi 4.
1. Download the Oracle Linux 8.5 64-bit Arm (aarch64) for use with RPi 4 and RPi 3 from https://www.oracle.com/linux/downloads/linux-arm-downloads.html
2. Write to Microsdxc card using balenaEtcher or the Raspberry Pi Imager
3. Optionally, have a Keyboard and Monitor connected to the Raspberry Pi 4
4. Insert Microsdxc into Raspberry Pi4 and poweron
5. Find the ethernet dhcp ipaddress of your Raspberry Pi 4 by running the nmap on your Macbook with your subnet
$ sudo nmap -sn 192.168.1.0/24
Nmap scan report for 192.168.1.227
Host is up (0.010s latency).
MAC Address: E4:5F:01:2E:D8:95 (Raspberry Pi Trading)
6. Login using root/oracle to ipaddress above. You will need to change your password on first login, then ssh again with the new password.
ssh root@$ipaddress
7. Extend the partition to maximize disk usage
fdisk -lu
growpart /dev/mmcblk1 3
btrfs filesystem resize max /
lsblk
Output:
[root@rpi ~]# fdisk -lu
Disk /dev/mmcblk1: 58.2 GiB, 62534975488 bytes, 122138624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc51af36c
Device Boot Start End Sectors Size Id Type
/dev/mmcblk1p1 * 2048 264191 262144 128M 6 FAT16
/dev/mmcblk1p2 264192 788479 524288 256M 82 Linux swap / Solaris
/dev/mmcblk1p3 788480 7604223 6815744 3.3G 83 Linux
[root@rpi ~]# growpart /dev/mmcblk1 3
CHANGED: partition=3 start=788480 old: size=6815744 end=7604224 new: size=121350111 end=122138591
[root@rpi ~]# btrfs filesystem resize max /
Resize device id 1 (/dev/mmcblk1p3) from 3.25GiB to max
[root@rpi ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk1 179:0 0 58.2G 0 disk
├─mmcblk1p1 179:1 0 128M 0 part /boot/efi
├─mmcblk1p2 179:2 0 256M 0 part [SWAP]
└─mmcblk1p3 179:3 0 57.9G 0 part /boot
[root@rpi ~]# fdisk -lu
Disk /dev/mmcblk1: 58.2 GiB, 62534975488 bytes, 122138624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc51af36c
Device Boot Start End Sectors Size Id Type
/dev/mmcblk1p1 * 2048 264191 262144 128M 6 FAT16
/dev/mmcblk1p2 264192 788479 524288 256M 82 Linux swap / Solaris
/dev/mmcblk1p3 788480 122138590 121350111 57.9G 83 Linux
8. Update and set the hostname with a domain and add the ipv4 address to /etc/hosts
dnf update -y
hostnamectl set-hostname rpi.example.com
echo "$ipaddress rpi rpi.example.com" >> /etc/hosts
9. Optionally, enable wifi
dnf -y install wpa_supplicant
# Update with your NETWORK_SSID and NETWORK_PASSWORD/PSA
cat << EOF > /etc/wpa_supplicant/wpa_supplicant.conf
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=wheel
country=US
p2p_disabled=1
network={
scan_ssid=1
ssid="<NETWORK_SSID>"
psk="<NETWORK_PASSWORD>"
}
EOF
For some reason the modprobe command does not work in the Service, so I had to run it manually
rmmod brcmfmac
modprobe -vvv brcmfmac
Output:
[root@rpi ~]# rmmod brcmfmac
[root@rpi ~]# modprobe -vvv brcmfmac
modprobe: INFO: custom logging function 0xaaac75debf80 registered
insmod /lib/modules/5.4.17-2136.307.3.1.el8uek.aarch64/kernel/drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko.xz
modprobe: INFO: context 0xaaaca47904e0 released
Create and start the wlan0 service
cat << EOF > /etc/systemd/system/network-wireless@.service
[Unit]
Description=Wireless network connectivity (%i)
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device
[Service]
Type=oneshot
RemainAfterExit=yes
#ExecStart=modprobe -vvv -r brcmfmac
#ExecStart=modprobe -vvv brcmfmac
ExecStart=/usr/sbin/ip link set dev %i up
ExecStart=/usr/sbin/wpa_supplicant -B -i %i -c /etc/wpa_supplicant/wpa_supplicant.conf
ExecStart=/usr/sbin/dhclient -v %i
ExecStop=/usr/sbin/ip link set dev %i down
[Install]
WantedBy=multi-user.target
EOF
ln -s /etc/systemd/system/network-wireless@.service /etc/systemd/system/multi-user.target.wants/network-wireless@wlan0.service
systemctl daemon-reload
systemctl start network-wireless@wlan0.service
If starting without the service, you can run
modprobe -vvv -r brcmfmac
modprobe -vvv brcmfmac
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
#dhclient -v -r wlan0
dhclient -v wlan0
Output:
[root@rpi ~]# modprobe -vvv -r brcmfmac
modprobe: INFO: custom logging function 0xaaac6879bf80 registered
rmmod brcmfmac
rmmod cfg80211
rmmod brcmutil
modprobe: INFO: context 0xaaac83b104d0 released
[root@rpi ~]# modprobe -vvv brcmfmac
modprobe: INFO: custom logging function 0xaaaab377bf80 registered
insmod /lib/modules/5.4.17-2136.300.7.el8uek.aarch64/kernel/net/wireless/cfg80211.ko.xz
insmod /lib/modules/5.4.17-2136.300.7.el8uek.aarch64/kernel/drivers/net/wireless/broadcom/brcm80211/brcmutil/brcmutil.ko.xz
insmod /lib/modules/5.4.17-2136.300.7.el8uek.aarch64/kernel/drivers/net/wireless/broadcom/brcm80211/brcmfmac/brcmfmac.ko.xz
modprobe: INFO: context 0xaaaace3104d0 released
[root@rpi ~]# wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
Successfully initialized wpa_supplicant
If you do not update the /etc/wpa_supplicant/wpa_supplicant.conf with the SSID, you can use wpa_cli to set the ssid manually.
wpa_cli
#wpa_cli -i wlan0 status
We get interfaces p2p-dev-wlan0 and wlan0. Use the wlan0.
[root@rpi ~]# wpa_cli
wpa_cli v2.9
Copyright (c) 2004-2019, Jouni Malinen <j@w1.fi> and contributors
This software may be distributed under the terms of the BSD license.
See README for more details.
Selected interface 'p2p-dev-wlan0'
Interactive mode
> interface
Available interfaces:
p2p-dev-wlan0
wlan0
> interface wlan0
Connected to interface 'wlan0.
> scan
OK
<3>CTRL-EVENT-SCAN-STARTED
> add_network
0
> set_network 0 ssid "<network_ssid>"
OK
> set_network 0 psk "<network_ssid>"
OK
> enable_network 0
OK
<3>CTRL-EVENT-SCAN-STARTED
<3>WPS-AP-AVAILABLE
…
> quit
[root@rpi ~]# The -v option shows information on screen about dhcp server and obtained lease
[root@rpi ~]# dhclient -v -r wlan0 # Release
[root@rpi ~]# dhclient -v wlan0 # Get new
[root@rpi ~]# pgrep -a wpa_supplicant
998 wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf
10. Check the release and file system type. Observe also the ssd mount option for btrfs filesystem for the “/” and “/boot”. Btrfs is SSD-aware and exploits TRIM/Discard to allow the file system to report unused blocks to the storage device for reuse. On SSDs, Btrfs avoids unnecessary seek optimization and aggressively sends writes in clusters, even if they are from unrelated files. This results in larger write operations and faster write throughput, albeit at the expense of more seeks later.
cat /etc/os-release
lsblk
df -Th
Output:
[root@rpi ~]# cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="8.5"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.5"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:5:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"
ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.5
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.5
[root@rpi ~]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
mmcblk1
├─mmcblk1p1 vfat C9C2-8A9D /boot/efi
├─mmcblk1p2 swap 73c0a84a-2edd-4815-8a54-37f23cf2a26a [SWAP]
└─mmcblk1p3 btrfs ol 789c4af1-dc46-4fae-92da-8b72c1f29df0 /var/lib/containers/storage/overlay
[root@rpi ~]# df -Th | grep "^/dev"
/dev/mmcblk1p3 btrfs 58G 12G 44G 21% /
/dev/mmcblk1p3 btrfs 58G 12G 44G 21% /boot
/dev/mmcblk1p1 vfat 128M 33M 96M 26% /boot/efi
[root@rpi ~]# mount | egrep 'btrfs|ext4'
/dev/mmcblk1p3 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=258,subvol=/root)
/dev/mmcblk1p3 on /boot type btrfs (rw,relatime,ssd,space_cache,subvolid=256,subvol=/boot)
11. Check the cgroup - A control group (cgroup) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, and so on) of a collection of processes. Cgroups are a key component of containers because there are often multiple processes running in a container that you need to control together. In Microshift, cgroups are used to implement resource requests and limits and corresponding QoS classes at the pod level.
ssh root@$ipaddress
mount | grep cgroup
cat /proc/cgroups | column -t # Check that memory and cpuset are present
ls -l /sys/fs/cgroup/cpu/cpu.cfs_quota_us # This needs to be present for MicroShift to work
Output:
[root@rpi ~]# mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
[root@rpi ~]# cat /proc/cgroups | column -t # Check that memory and cpuset are present
#subsys_name hierarchy num_cgroups enabled
cpuset 6 1 1
cpu 2 1 1
cpuacct 2 1 1
blkio 4 1 1
memory 10 90 1
devices 7 47 1
freezer 3 1 1
net_cls 5 1 1
perf_event 8 1 1
net_prio 5 1 1
hugetlb 12 1 1
pids 9 69 1
rdma 11 1 1
[root@rpi ~]# ls -l /sys/fs/cgroup/cpu/cpu.cfs_quota_us # This needs to be present for MicroShift to work
-rw-r--r-- 1 root root 0 May 12 14:39 /sys/fs/cgroup/cpu/cpu.cfs_quota_us
Install sense_hat and RTIMULib on Oracle Linux
The Sense HAT is an add-on board for the Raspberry Pi. The Sense HAT has an 8 × 8 RGB LED matrix, a five – button joystick and includes the following sensors: Inertial Measurement Unit (Accelerometer, Gyroscope, Magnetometer), Temperature, Barometric pressure, Humidity. If you have the Sense HAT attached, install the libraries.
Install sensehat
dnf -y install zlib zlib-devel libjpeg-devel gcc gcc-c++ i2c-tools python3-devel python3 python3-pip cmake
pip3 install Cython Pillow numpy sense_hat
Check the Sense Hat with i2cdetect
i2cdetect -y 1
[root@rpi ~]# i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- UU -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- 46 -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- UU -- -- 5f
60: -- -- -- -- -- -- -- -- -- -- 6a -- -- -- -- --
70: -- -- -- -- -- -- -- --
Add the i2c-dev line to /etc/modules to load the kernel module automatically on boot.
cat << EOF >> /etc/modules
i2c-dev
EOF
Create the file /etc/udev/rules.d/99-i2c.rules with the following contents:
cat << EOF >> /etc/udev/rules.d/99-i2c.rules
KERNEL=="i2c-[0-7]",MODE="0666"
EOF
The Raspberry Pi build comes with the Industrial I/O modules preloaded. We get initialization errors on some of the sensors because the Industrial I/O modules grab on to the i2c sensors on the Sense HAT and refuse to let them go or allow them to be read correctly. Check this with “lsmod | grep st_”.
[root@rpi ~]# lsmod | grep st_
st_pressure_spi 262144 0
st_magn_spi 262144 0
st_sensors_spi 262144 2 st_pressure_spi,st_magn_spi
st_magn_i2c 262144 0
st_pressure_i2c 262144 0
st_pressure 262144 2 st_pressure_i2c,st_pressure_spi
st_magn 262144 2 st_magn_i2c,st_magn_spi
st_sensors_i2c 262144 2 st_pressure_i2c,st_magn_i2c
st_sensors 327680 6 st_pressure,st_pressure_i2c,st_magn_i2c,st_pressure_spi,st_magn,st_magn_spi
industrialio_triggered_buffer 262144 2 st_pressure,st_magn
industrialio 327680 9 st_pressure,industrialio_triggered_buffer,st_sensors,st_pressure_i2c,kfifo_buf,st_magn_i2c,st_pressure_spi,st_magn,st_magn_spi
We need to blacklist the modules and reboot to take effect
cat << EOF > /etc/modprobe.d/blacklist-industialio.conf
blacklist st_magn_spi
blacklist st_pressure_spi
blacklist st_sensors_spi
blacklist st_pressure_i2c
blacklist st_magn_i2c
blacklist st_pressure
blacklist st_magn
blacklist st_sensors_i2c
blacklist st_sensors
blacklist industrialio_triggered_buffer
blacklist industrialio
EOF
reboot
Install RTIMULib
dnf -y install git
git clone https://github.com/RPi-Distro/RTIMULib.git
cd RTIMULib/
cd Linux/python
python3 setup.py build
python3 setup.py install
cd ../..
cd RTIMULib
mkdir build
cd build
cmake ..
make -j4
make install
ldconfig
# Optional test the sensors
cd /root/RTIMULib/Linux/RTIMULibDrive11
make -j4
make install
RTIMULibDrive11 # Ctrl-C to break
cd /root/RTIMULib/Linux/RTIMULibDrive10
make -j4
make install
RTIMULibDrive10 # Ctrl-C to break
# Optional
dnf -y install qt5-qtbase-devel
cd /root/RTIMULib/Linux/RTIMULibDemoGL
qmake-qt5
make -j4
make install
Check the Sense Hat with i2cdetect and that the i2c sensors are no longer being held.
i2cdetect -y 1
lsmod | grep st_
[root@rpi ~]# i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- 1c -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- 46 -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- 5c -- -- 5f
60: -- -- -- -- -- -- -- -- -- -- 6a -- -- -- -- --
70: -- -- -- -- -- -- -- --
[root@rpi sensehat-fedora-iot]# lsmod | grep st_
empty
Replace the sense_hat.py with the new file that uses SMBus as shown below and test the SenseHat samples for the Sense Hat's LED matrix and sensors.
cd ~
git clone https://github.com/thinkahead/microshift.git
cd microshift
cd raspberry-pi/sensehat-fedora-iot
pip3 install smbus
# Update the python package to use the i2cbus
cp -f sense_hat.py.new /usr/local/lib/python3.6/site-packages/sense_hat/sense_hat.py
# Enable random LEDs
python3 sparkles.py # Ctrl-C to interrupt
# Show multiple screens to test LEDs
python3 rainbow.py # Ctrl-C to interrupt
# First time you run the temperature.py, you may see “Temperature: 0 C”. Just run it again.
python3 temperature.py
# Show the Temperature, Pressure and Humidity
python3 testsensehat.py # Ctrl-C to interrupt
# Show two digits for multiple numbers
python3 digits.py
# Use the new get_state method from sense_hat.py
python3 joystick.py # U=Up D=Down L=Left R=Right M=Press
# When a magnet gets close to SenseHAT, the LEDs will all turn red for 1/5 of a second
python3 magnetometer.py
# Find Magnetic North
python3 compass.py
Install MicroShift on the Raspberry Pi 4 Oracle Linux host
Setup EPEL
tee /etc/yum.repos.d/ol8-epel.repo<<EOF
[ol8_developer_EPEL]
name= Oracle Linux \$releasever EPEL (\$basearch)
baseurl=https://yum.oracle.com/repo/OracleLinux/OL8/developer/EPEL/\$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
EOF
dnf makecache
Setup crio and MicroShift Nightly EPEL Stream 8 aarch64
rpm -qi selinux-policy # selinux-policy-3.14.3-80.el8_5.2
dnf -y install 'dnf-command(copr)'
curl https://copr.fedorainfracloud.org/coprs/g/redhat-et/microshift-nightly/repo/epel-8/group_redhat-et-microshift-nightly-epel-8.repo -o /etc/yum.repos.d/microshift-nightly-epel-8.repo
cat /etc/yum.repos.d/microshift-nightly-epel-8.repo
VERSION=1.22
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_8/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo
cat /etc/yum.repos.d/devel\:kubic\:libcontainers\:stable\:cri-o\:${VERSION}.repo
dnf -y install firewalld cri-o cri-tools microshift
Install KVM on the host and validate the Host Virtualization Setup. The virt-host-validate command validates that the host is configured in a suitable way to run libvirt hypervisor driver qemu.
dnf -y install libvirt-client libvirt-nss qemu-kvm virt-manager virt-install virt-viewer
# Works with nftables on Fedora Server and Fedora IoT, Oracle Linux
# vi /etc/firewalld/firewalld.conf # FirewallBackend=iptables
systemctl enable --now libvirtd
virt-host-validate qemu
Check that cni plugins are present
ls /opt/cni/bin/ # empty
ls /usr/libexec/cni # cni plugins
We will have systemd start and manage MicroShift. Refer to the microshift service for the three approaches.
systemctl enable --now crio microshift
# Copy flannel
cp /opt/cni/bin/flannel /usr/libexec/cni/.
You may read about selecting zones for your interfaces.
systemctl enable firewalld --now
firewall-cmd --zone=trusted --add-source=10.42.0.0/16 --permanent
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --zone=public --add-port=5353/udp --permanent
firewall-cmd --reload
Additional ports may need to be opened. For external access to run kubectl or oc commands against MicroShift, add the 6443 port:
firewall-cmd --zone=public --permanent --add-port=6443/tcp
For access to services through NodePort, add the port range 30000-32767:
firewall-cmd --zone=public --permanent --add-port=30000-32767/tcp
firewall-cmd --reload
firewall-cmd --list-all --zone=public
firewall-cmd --get-default-zone
#firewall-cmd --set-default-zone=public
#firewall-cmd --get-active-zones
firewall-cmd --list-all
Check the microshift and crio logs
journalctl -u microshift -f
journalctl -u crio -f
The microshift service references the microshift binary in the /usr/bin directory
[root@rpi sensehat-fedora-iot]# cat /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift
Wants=network-online.target crio.service
After=network-online.target crio.service
[Service]
WorkingDirectory=/usr/bin/
ExecStart=/usr/bin/microshift run
Restart=always
User=root
[Install]
WantedBy=multi-user.target
Install the kubectl and the openshift oc client
ARCH=arm64
cd /tmp
dnf -y install tar
export OCP_VERSION=4.9.11 && \
curl -o oc.tar.gz https://mirror2.openshift.com/pub/openshift-v4/$ARCH/clients/ocp/$OCP_VERSION/openshift-client-linux-$OCP_VERSION.tar.gz && \
tar -xzvf oc.tar.gz && \
rm -f oc.tar.gz && \
install -t /usr/local/bin {kubectl,oc} && \
rm -f {README.md,kubectl,oc}
It will take around 3 minutes for all pods to start. Check the status of node and pods using kubectl or oc client.
export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
#watch "kubectl get nodes;kubectl get pods -A;crictl pods;crictl images"
watch "oc get nodes;oc get pods -A;crictl pods;crictl images"
Install podman - We will use podman for containerized deployment of MicroShift and building images for the samples.
dnf -y install podman
Samples to run on MicroShift
We will run samples that will show the use of dynamic persistent volume, SenseHat and the USB camera.
1. InfluxDB/Telegraf/Grafana with dynamic persistent volumes
The source code is available for this influxdb sample in github.
cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/influxdb
If you want to run all the steps in a single command, get the nodename.
oc get nodes
Output:
[root@rpi influxdb]# oc get nodes
NAME STATUS ROLES AGE VERSION
rpi.example.com Ready <none> 3m36s v1.21.0
Replace the annotation kubevirt.io/provisionOnNode with the above nodename and execute the runall-fedora-dynamic.sh. This will create a new project influxdb. Note that the node name is different when running MicroShift with the all-in-one containerized approach. So, you will use the the node name in the container microshift.example.com instead of the rpi.example.com.
sed -i "s|coreos|rpi.example.com|" influxdb-data-dynamic.yaml
sed -i "s|coreos|rpi.example.com|" grafana/grafana-data-dynamic.yaml
./runall-fedora-dynamic.sh
We create and push the “measure-fedora:latest” image using the Dockerfile that uses SMBus. The script will create a new project influxdb for this sample, install InfluxDB, install the pod for SenseHat measurements, install Telegraf and check the measurements for the telegraf database in InfluxDB. Finally, it will install Grafana.
This script will allocate dynamic persistent volumes using influxdb-data-dynamic.yaml and grafana-data-dynamic.yaml. The annotation provisionOnNode and the storageClassName are required for dynamic PV.
annotations:
kubevirt.io/provisionOnNode: rpi.example.com
spec:
storageClassName: kubevirt-hostpath-provisioner
Persistent Volumes and Claims Output:
[root@rpi raspberry-pi]# oc get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-92ab8559-9998-4abf-8e7e-a68833cf3748 57Gi RWO Delete Bound influxdb/influxdb-data kubevirt-hostpath-provisioner 13m
persistentvolume/pvc-c8530d30-0282-4898-a963-c01cf589d57d 57Gi RWO Delete Bound influxdb/grafana-data kubevirt-hostpath-provisioner 9m57s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/grafana-data Bound pvc-c8530d30-0282-4898-a963-c01cf589d57d 57Gi RWO kubevirt-hostpath-provisioner 9m58s
persistentvolumeclaim/influxdb-data Bound pvc-92ab8559-9998-4abf-8e7e-a68833cf3748 57Gi RWO kubevirt-hostpath-provisioner 13m
Add the "<RaspberryPiIPAddress> grafana-service-influxdb.cluster.local" to /etc/hosts on your laptop and login to http://grafana-service-influxdb.cluster.local/login using admin/admin. You will need to change the password on first login. Go to the Dashboards list (left menu > Dashboards > Manage). Open the Analysis Server dashboard to display monitoring information for MicroShift. Open the Balena Sense dashboard to show the temperature, pressure, and humidity from SenseHat.
Finally, after you are done working with this sample, you can run the deleteall-fedora-dynamic.sh
./deleteall-fedora-dynamic.sh
Deleting the persistent volume claims automatically deletes the persistent volumes.
2. Node Red live data dashboard with SenseHat sensor charts
We will install Node Red on the ARM device as a deployment within MicroShift, add the dashboard and view the gauges for temperature/pressure/humidity data from SenseHat on the dashboard.
cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/nodered
Build and push the arm64v8 image “karve/nodered-fedora:arm64”
cd docker-custom/
./docker-debianonfedora.sh
podman push docker.io/karve/nodered-fedora:arm64
cd ..
Deploy Node Red with persistent volume for /data within the node red container
mkdir /var/hpvolumes/nodered
restorecon -R -v "/var/hpvolumes/*"
rm -rf /var/hpvolumes/nodered/*;cp -r nodered-volume/* /var/hpvolumes/nodered/.
oc new-project nodered
oc apply -f noderedpv.yaml -f noderedpvc.yaml -f nodered2.yaml -f noderedroute.yaml
oc get routes
oc -n nodered wait deployment nodered-deployment --for condition=Available --timeout=300s
oc logs deployment/nodered-deployment -f
Add the ipaddress of the Raspberry Pi 4 device for nodered-svc-nodered.cluster.local to /etc/hosts on your Laptop and browse to http://nodered-svc-nodered.cluster.local/
The following modules required for the dashboard have been preinstalled node-red-dashboard, node-red-node-smooth, node-red-node-pi-sense-hat. These can be seen under “Manage Palette - Install”. The Flow 1 or Flow 2 have already been imported from the nodered sample. This import to the Node Red can be done manually under “Import Nodes” and then click “Deploy”. The node-red-node-pi-sense-hat module require a change in the sensehat.py in order to use the sense_hat.py.new that uses smbus and new function for joystick. This change is accomplished by overwriting with the modified sensehat.py in Dockerfile.debianonfedora (docker.io/karve/nodered-fedora:arm6 built using docker-debianonfedora.sh) and further copied from /tmp directory to the correct volume when the pod starts in nodered2.yaml.
Double click the Sense HAT input node and make sure that all the events are checked. Select the Dashboard. Click on the outward arrow in the tabs to view the sensor charts. You will see the Home by Default. You can see the state of the Joystick Up, Down, Left, Right or Pressed. Click on the Hamburger Menu (3 lines) and select PiSenseHAT.
If you selected the Flow 1, you could click on the Input for the Timestamp under “Dot Matrix” to see the “Alarm” message scroll on the SenseHat LED.
We can continue running the next sample that will reuse this Node Red deployment. If the Node Red Deployment is no longer required, we can delete it as follows:
cd ~/microshift/raspberry-pi/nodered
oc delete -f noderedpv.yaml -f noderedpvc.yaml -f nodered2.yaml -f noderedroute.yaml -n nodered
3. TensorFlow Lite Python object detection example in MicroShift with SenseHat and Node Red
This example requires the same Node Red setup as in the previous Sample 2.
cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/object-detection
We will build the image for object detection send pictures and web socket chat messages to Node Red when a person is detected using a pod in microshift.
cp ../sensehat-fedora-iot/sense_hat.py.new .
podman build -f Dockerfile.fedora -t docker.io/karve/object-detection-raspberrypi4-fedora . # Select the docker.io/balenalib/raspberrypi4-64-debian:latest as the base image
podman push docker.io/karve/object-detection-raspberrypi4-fedora:latest
Update the env WebSocketURL and ImageUploadURL as shown below. Also update the hostAliases in object-detection-fedora.yaml to point to your raspberry pi 4 ip address (192.168.1.227 shown below).
env:
- name: WebSocketURL
value: "ws://nodered-svc-nodered.cluster.local/ws/chat"
- name: ImageUploadURL
value: http://nodered-svc-nodered.cluster.local/upload
hostAliases:
- hostnames:
- nodered-svc-nodered.cluster.local
ip: 192.168.1.227
oc project default
oc apply -f object-detection-fedora.yaml
We will see pictures being sent to Node Red when a person is detected at http://nodered-svc-nodered.cluster.local/#flow/3e30dc50ae28f61f and chat messages at http://nodered-svc-nodered.cluster.local/chat. When we are done testing, we can delete the deployment.
oc delete -f object-detection-fedora.yaml
4. Running a Virtual Machine Instance on MicroShift
Find the latest version of the KubeVirt Operator.
LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest-arm64)
echo $LATEST
I used the following version:
LATEST=20220513 # If the latest version does not work
oc apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml
oc apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml
oc adm policy add-scc-to-user privileged -n kubevirt -z kubevirt-operator
# The .status.phase will show Deploying multiple times and finally Deployed
oc get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}" -w # Ctrl-C to break
oc -n kubevirt wait kv kubevirt --for condition=Available --timeout=300s
oc get pods -n kubevirt
We can build the OKD Web Console (Codename: “bridge”) from the source as mentioned in Part 9. We will run the “bridge” as a container image that we run within MicroShift.
cd /root/microshift/raspberry-pi/console
oc create serviceaccount console -n kube-system
oc create clusterrolebinding console --clusterrole=cluster-admin --serviceaccount=kube-system:console -n kube-system
sleep 5
oc get serviceaccount console --namespace=kube-system -o jsonpath='{.secrets[0].name}'
oc get serviceaccount console --namespace=kube-system -o jsonpath='{.secrets[1].name}'
Replace BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT value https://192.168.1.209:6443 with your raspberry pi 4's ip address, and secretRef token with the console-token-* from above two secret names for BRIDGE_K8S_AUTH_BEARER_TOKEN in okd-web-console-install.yaml. Then apply/create the okd-web-console-install.yaml.
oc apply -f okd-web-console-install.yaml
oc expose svc console-np-service -n kube-system
oc get routes -n kube-system
oc -n kube-system wait deployment console-deployment --for condition=Available --timeout=300s
oc logs deployment/console-deployment -f -n kube-system
Add the Raspberry Pi IP address to /etc/hosts on your Macbook Pro to resolve console-np-service-kube-system.cluster.local. Now you can access the OKD Web Console from your Laptop http://console-np-service-kube-system.cluster.local/
We can optionally preload the fedora image into crio (if using the all-in-one containerized approach, this needs to be run within the microshift pod running in podman)
crictl pull quay.io/kubevirt/fedora-cloud-container-disk-demo:20210811_9fec1f849-arm64
Now let’s create a Fedora Virtual Machine Instance using the vmi-fedora.yaml.
cd /root/microshift/raspberry-pi/vmi
oc apply -f vmi-fedora.yaml
watch oc get vmi,pods
The output for the virtualmachineinstance PHASE goes from “Scheduling” to “Scheduled” to “Running” after the virt-launcher-vmi-fedora pod STATUS goes from “Init” to “Running”. Note down the ip address of the vmi-fedora Virtual Machine Instance. Directly connect to the VMI from the Raspberry Pi 4 with fedora as the password. Note that it will take another minute after the VMI goes to Running state to ssh to the instance.
Output:
[root@rpi vmi]# oc get vmi
NAME AGE PHASE IP NODENAME READY
vmi-fedora 36m Running 10.42.0.31 rpi.example.com True
[root@rpi vmi]# ssh fedora@10.42.0.31
The authenticity of host '10.42.0.31 (10.42.0.31)' can't be established.
ECDSA key fingerprint is SHA256:go9hFGRHMLKW/PBgimj93wnRTTkYY4iJpbiexMUIQWc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.42.0.31' (ECDSA) to the list of known hosts.
fedora@10.42.0.31's password:
[fedora@vmi-fedora ~]$ ping -c 2 google.com
PING google.com (142.250.176.206) 56(84) bytes of data.
64 bytes from lga34s37-in-f14.1e100.net (142.250.176.206): icmp_seq=1 ttl=116 time=3.18 ms
64 bytes from lga34s37-in-f14.1e100.net (142.250.176.206): icmp_seq=2 ttl=116 time=2.99 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.992/3.088/3.184/0.096 ms
[fedora@vmi-fedora ~]$ exit
logout
Connection to 10.42.0.31 closed.
Alternatively, a second way is to create a Pod to run the ssh client and connect to the Fedora VM from this pod. Let’s create that openssh-client pod:
oc run alpine --privileged --rm -ti --image=alpine -- /bin/sh
apk update && apk add --no-cache openssh-client
or
oc run sshclient --privileged --rm -ti --image=karve/alpine-sshclient:arm64 -- /bin/sh
#oc attach sshclient -c sshclient -i -t
Then, ssh to the Fedora VMI from this openssh-client container.
Output:
[root@rpi vmi]# oc run sshclient --privileged --rm -ti --image=karve/alpine-sshclient:arm64 -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # ssh fedora@10.42.0.31 "bash -c \"ping -c 2 google.com\""
The authenticity of host '10.42.0.31 (10.42.0.31)' can't be established.
ED25519 key fingerprint is SHA256:AXd6hQWbiFrau3pYPQiTRJvLLHCV1BJpU8cgEg2ZMTg.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.42.0.31' (ED25519) to the list of known hosts.
fedora@10.42.0.31's password:
PING google.com (142.250.176.206) 56(84) bytes of data.
64 bytes from lga34s37-in-f14.1e100.net (142.250.176.206): icmp_seq=1 ttl=116 time=3.12 ms
64 bytes from lga34s37-in-f14.1e100.net (142.250.176.206): icmp_seq=2 ttl=116 time=3.02 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 3.017/3.070/3.124/0.053 ms
/ # exit
Session ended, resume using 'oc attach sshclient -c sshclient -i -t' command when the pod is running
pod "sshclient" deleted
A third way to connect to the VM is to use the virtctl console. You can compile your own virtctl as was described in Part 9. To simplify, we copy virtctl arm64 binary from prebuilt container image to /usr/local/bin on the Raspberry Pi 4 and connect to the VMI using “virtctl console” command.
id=$(podman create docker.io/karve/kubevirt:arm64)
podman cp $id:_out/cmd/virtctl/virtctl /usr/local/bin
podman rm -v $id
virtctl console vmi-fedora
Output:
[root@rpi nodered]# id=$(podman create docker.io/karve/kubevirt:arm64)
Trying to pull docker.io/karve/kubevirt:arm64...
Getting image source signatures
Copying blob 7065f6098427 done
Copying config 1c7a5aa443 done
Writing manifest to image destination
Storing signatures
[root@rpi nodered]# podman cp $id:_out/cmd/virtctl/virtctl /usr/local/bin
[root@rpi nodered]# podman rm -v $id
37acc1660b7684e7d98d0873df70b98fad502056be825238f39335ceea1ed5aa
[root@rpi nodered]# virtctl console vmi-fedora
Successfully connected to vmi-fedora console. The escape sequence is ^]
vmi-fedora login: fedora
Password:
Last login: Fri May 13 16:37:58 from 10.42.0.1
[fedora@vmi-fedora ~]$ ping -c 2 google.com
PING google.com (142.250.176.206) 56(84) bytes of data.
64 bytes from lga34s37-in-f14.1e100.net (142.250.176.206): icmp_seq=1 ttl=116 time=3.28 ms
64 bytes from lga34s37-in-f14.1e100.net (142.250.176.206): icmp_seq=2 ttl=116 time=2.91 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.908/3.095/3.282/0.187 ms
[fedora@vmi-fedora ~]$ # Ctrl-] to detach
[root@rpi nodered]#
When done, we can delete the VMI
oc delete -f vmi-fedora.yaml
We can run other VM and VMI samples for alpine, cirros and fedora images as in Part 9. When done, you may delete kubevirt operator.
oc delete -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml
oc delete -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml
5. Install Metrics Server
This will enable us to run the “kubectl top” and “oc adm top” commands.
dnf install -y wget jq
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server-components.yaml
oc apply -f metrics-server-components.yaml
# Wait for the metrics-server to start in the kube-system namespace
oc get deployment metrics-server -n kube-system
oc get events -n kube-system
# Wait for a couple of minutes for metrics to be collected
oc get --raw /apis/metrics.k8s.io/v1beta1/nodes
oc get --raw /apis/metrics.k8s.io/v1beta1/pods
oc get --raw /api/v1/nodes/$(kubectl get nodes -o json | jq -r '.items[0].metadata.name')/proxy/stats/summary
watch "kubectl top nodes;kubectl top pods -A"
watch "oc adm top nodes;oc adm top pods -A"
Output:
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
rpi.example.com 317m 7% 2572Mi 33%
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system metrics-server-64cf6869bd-ptwrt 5m 18Mi
kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-kw7cg 1m 11Mi
openshift-dns dns-default-6x7hd 4m 29Mi
openshift-dns node-resolver-px52p 7m 16Mi
openshift-ingress router-default-85bcfdd948-rmjjt 2m 35Mi
openshift-service-ca service-ca-7764c85869-wrlw4 9m 27Mi
We can delete the metrics server using
oc delete -f metrics-server-components.yaml
6. Run a jupyter notebook sample for license plate recognition (RPi with 8GB RAM)
We will run the sample described at the Red Hat OpenShift Data Science Workshop License plate recognition. The Dockerfile uses the arm64 Jupyter Notebook base image: scipy-notebook. Since we do not have a tensorflow arm64 image, we install it as described at Qengineering.
cd ~
git clone https://github.com/thinkahead/microshift.git
cd ~/microshift/raspberry-pi/tensorflow-notebook
oc apply -f notebook.yaml
oc -n default wait pod notebook --for condition=Ready --timeout=300s
oc get routes
Output:
[root@rpi tensorflow-notebook]# oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
flask-route flask-route-default.cluster.local notebook-svc 5000 None
notebook-route notebook-route-default.cluster.local notebook-svc 5001 None
Add the ipaddress of the Raspberry Pi 4 device for notebook-route-default.cluster.local to /etc/hosts on your Laptop and browse to http://notebook-route-default.cluster.local/tree?. Login with the default password mysecretpassword. Go the work folder and select and run the License-plate-recognition notebook at http://notebook-route-default.cluster.local/notebooks/work/02_Licence-plate-recognition.ipynb
We can also run it as an application and test it using the corresponding notebooks. Run the http://notebook-route-default.cluster.local/notebooks/work/03_LPR_run_application.ipynb
Wait for the following to appear.
Instructions for updating:
non-resource variables are not supported in the long term
Model Loaded successfully...
Model Loaded successfully...
[INFO] Model loaded successfully...
[INFO] Labels loaded successfully...
Then run http://notebook-route-default.cluster.local/notebooks/work/04_LPR_test_application.ipynb
We can experiment with a custom image. Let’s download the image to the pod and run the cells again with the new image and check the prediction.
oc exec -it notebook -- bash -c "wget \"https://unsplash.com/photos/MgfKoRdI948/download?force=true&ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjUyNDY4Mjcz\" -O /tmp/3183KND.jpg"
Then, run the http://notebook-route-default.cluster.local/notebooks/work/05_Send_image.ipynb
Add the cell with the following code:
my_image = 'https://unsplash.com/photos/MgfKoRdI948/download?force=true&ixid=MnwxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNjUyNDY4Mjcz'
from PIL import Image
import requests
from io import BytesIO
response = requests.get(my_image)
img = BytesIO(response.content).read()
import base64
import requests
from json import dumps
encoded_image = base64.b64encode(img).decode('utf-8')
content = {"image": encoded_image}
json_data = dumps(content)
headers = {"Content-Type" : "application/json"}
r = requests.post(my_route + '/predictions', data=json_data, headers=headers)
print(r.content)
from IPython.display import Image
from IPython.core.display import HTML
Image(url=my_image)
When we are done working with the license plate recognition sample notebook, we can delete it as follows:
oc delete -f notebook.yaml
Cleanup MicroShift
We can use the cleanup.sh script available on github to cleanup the pods and images. If you already cloned the microshift repo from github, you have the script in the ~/microshift/hack directory.
cd ~/microshift/hack
./cleanup.sh
Containerized MicroShift on Oracle Linux (64 bit)
We can run MicroShift within containers in two ways:
- MicroShift Containerized – The MicroShift binary runs in a Podman container, CRI-O Systemd service runs directly on the host and data is stored in a podman volume
- MicroShift Containerized All-In-One – The MicroShift binary and CRI-O service run within a container and data is stored in a podman volume, microshift-data. This should be used for “Testing and Development” only
Microshift Containerized
If you did not already install podman, you can do it now.
dnf install -y podman
We will use a new microshift.service that runs microshift in a pod using the prebuilt image and uses a podman volume. Rest of the pods run using crio on the host.
cat << EOF > /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift Containerized
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target crio.service
After=network-online.target crio.service
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/usr/bin/mkdir -p /var/lib/kubelet ; /usr/bin/mkdir -p /var/hpvolumes
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/bin/podman run \
--cidfile=%t/%n.ctr-id \
--cgroups=no-conmon \
--rm \
--replace \
--sdnotify=container \
--label io.containers.autoupdate=registry \
--network=host \
--privileged \
-d \
--name microshift \
-v /var/hpvolumes:/var/hpvolumes:z,rw,rshared \
-v /var/run/crio/crio.sock:/var/run/crio/crio.sock:rw,rshared \
-v microshift-data:/var/lib/microshift:rw,rshared \
-v /var/lib/kubelet:/var/lib/kubelet:z,rw,rshared \
-v /var/log:/var/log \
-v /etc:/etc quay.io/microshift/microshift:latest
ExecStop=/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target default.target
EOF
systemctl daemon-reload
systemctl enable --now crio microshift
podman ps -a
podman volume inspect microshift-data # Get the Mountpoint where kubeconfig is located
export KUBECONFIG=/var/lib/containers/storage/volumes/microshift-data/_data/resources/kubeadmin/kubeconfig
watch "oc get nodes;oc get pods -A;crictl pods;crictl images"
Now that microshift is started, we can run the samples shown earlier.
After we are done, we can delete the microshift container. The --rm we used in the podman run will delete the container when we stop it.
podman stop microshift && podman volume rm microshift-data
After it is stopped, we can run the cleanup.sh to delete the pods and images from crio.
MicroShift Containerized All-In-One
Let’s stop the crio on the host, we will be creating an all-in-one container in podman that will run crio within the container.
systemctl stop crio
systemctl disable crio
mkdir /var/hpvolumes
We will run the all-in-one microshift in podman using prebuilt images (replace the image in the podman run command below with the latest image). Normally you would run the following to start the all-in-one microshift, but it does not work:
setsebool -P container_manage_cgroup true
podman volume rm microshift-data;podman volume create microshift-data
podman run -d --rm --name microshift -h microshift.example.com --privileged -v /lib/modules:/lib/modules -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -p 6443:6443 -p 8080:8080 -p 80:80 quay.io/microshift/microshift-aio:4.8.0-0.microshift-2022-04-20-182108-linux-nft-arm64
The containers will give error when starting within the microshift pod. Since the “setsebool -P container_manage_cgroup true” does not work, we mount the /sys/fs/cgroup into the container using -v /sys/fs/cgroup:/sys/fs/cgroup:ro. This will volume mount /sys/fs/cgroup into the container as read/only, but the subdir/mount points will be mounted in as read/write.
podman run -d --rm --name microshift -h microshift.example.com --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /lib/modules:/lib/modules -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -p 6443:6443 -p 8080:8080 -p 80:80 quay.io/microshift/microshift-aio:4.8.0-0.microshift-2022-04-20-182108-linux-nft-arm64
Now that you know the podman command to start the microshift all-in-one, you may alternatively use the following microshift service.
cat << EOF > /usr/lib/systemd/system/microshift.service
[Unit]
Description=MicroShift all-in-one
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --sdnotify=conmon --cgroups=no-conmon --rm --replace -d --name microshift -h microshift.example.com --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -v /lib/modules:/lib/modules --label io.containers.autoupdate=registry -p 6443:6443 -p 80:80 quay.io/microshift/microshift-aio:latest
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target default.target
EOF
systemctl daemon-reload
systemctl start microshift
On the host Raspberry Pi 4, we set KUBECONFIG to point to the kubeconfig on the data volume at the Mountpoint from above.
export KUBECONFIG=/var/lib/containers/storage/volumes/microshift-data/_data/microshift/resources/kubeadmin/kubeconfig
# crio on host is stopped, so we do not run crictl commands on host
watch "oc get nodes;oc get pods -A;podman exec -it microshift crictl ps -a"
The crio service is stopped on the Raspberry Pi, so crictl command will not work directly on the Pi. The crictl commands will work within the microshift container in podman as shown in the watch command above.
To run the Virtual Machine examples in the all-in-one MicroShift, we need to execute the mount with --make-shared as follows in the microshift container to prevent the “Error: path "/var/run/kubevirt" is mounted on "/" but it is not a shared mount” event from virt-handler.
podman exec -it microshift mount --make-shared /
We may also preload the virtual machine images using "crictl pull".
podman exec -it microshift crictl pull quay.io/kubevirt/fedora-cloud-container-disk-demo:20210811_9fec1f849-arm64
Now, we can run the samples shown earlier.
For the Virtual Machine Instance Sample 4, we can connect to the vmi-fedora by exposing the ssh port for the Virtual Machine Instance as a NodePort Service after the instance is started. This NodePort is within the all-in-one pod that is running in podman. The ip address of the all-in-one microshift podman container is 10.88.0.3. We expose the target port 22 on the VM as a service on port 22 that is in turn exposed on the microshift container with allocated port 30585 as seen below. We run and exec into a new pod called ssh-proxy, install the openssh-client on the ssh-proxy and ssh to the port 30585 on the all-in-one microshift container. This takes us to the VMI port 22 as shown below:
[root@rpi vmi]# oc get vmi,pods
NAME AGE PHASE IP NODENAME READY
virtualmachineinstance.kubevirt.io/vmi-fedora 11m Running 10.42.0.14 microshift.example.com True
NAME READY STATUS RESTARTS AGE
pod/virt-launcher-vmi-fedora-w97wd 2/2 Running 0 11m
[root@rpi vmi]# virtctl expose vmi vmi-fedora --port=22 --target-port=22 --name=vmi-fedora-ssh --type=NodePort
Service vmi-fedora-ssh successfully exposed for vmi vmi-fedora
[root@rpi vmi]# oc get svc vmi-fedora-ssh
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vmi-fedora-ssh NodePort 10.43.27.32 <none> 22:30585/TCP 11s
[root@rpi vmi]# podman inspect --format "{{.NetworkSettings.IPAddress}}" microshift
10.88.0.3
[root@rpi vmi]# oc run -i --tty ssh-proxy --rm --image=ubuntu --restart=Never -- /bin/sh -c "apt-get update;apt-get -y install openssh-client;ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null fedora@10.88.0.3 -p 30585" If you don't see a command prompt, try pressing enter.
…
The following additional packages will be installed:
libbsd0 libcbor0.8 libedit2 libfido2-1 libmd0 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 xauth
Suggested packages:
keychain libpam-ssh monkeysphere ssh-askpass
The following NEW packages will be installed:
libbsd0 libcbor0.8 libedit2 libfido2-1 libmd0 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1 openssh-client xauth
0 upgraded, 14 newly installed, 0 to remove and 3 not upgraded.
…
Warning: Permanently added '[10.88.0.3]:30585' (ED25519) to the list of known hosts.
fedora@10.88.0.3's password:
[fedora@vmi-fedora ~]$ sudo dnf install -y qemu-guest-agent
…
Complete!
[fedora@vmi-fedora ~]$ sudo systemctl enable --now qemu-guest-agent
[fedora@vmi-fedora ~]$ ping -c 2 google.com
PING google.com (142.250.80.46) 56(84) bytes of data.
64 bytes from lga34s34-in-f14.1e100.net (142.250.80.46): icmp_seq=1 ttl=115 time=4.15 ms
64 bytes from lga34s34-in-f14.1e100.net (142.250.80.46): icmp_seq=2 ttl=115 time=4.04 ms
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.044/4.099/4.154/0.055 ms
[fedora@vmi-fedora ~]$ exit
logout
Connection to 10.88.0.3 closed.
pod "ssh-proxy" deleted
The QEMU guest agent that we installed is a daemon that runs on the virtual machine and passes information to the host about the virtual machine, users, file systems, and secondary networks.
After we are done, we can delete the all-in-one microshift container.
podman rm -f microshift && podman volume rm microshift-data
or if started using systemd, then
systemctl stop microshift
Conclusion
In this Part 16, we saw multiple options to run MicroShift on the Raspberry Pi 4 with the Oracle Linux 8.5 (64 bit). We used dynamic persistent volumes to install InfluxDB/Telegraf/Grafana with a dashboard to show SenseHat sensor data. We ran samples that used the Sense Hat/USB camera and worked with a sample that sent the pictures and web socket messages to Node Red when a person was detected. We installed the OKD Web Console and saw how to connect to a Virtual Machine Instance using KubeVirt on MicroShift with Oracle Linux 8.5. Finally we saw how to run jupyter notebooks with the license plate recognition demo. We will work with MicroShift with KubeVirt and Kata Containers on Raspberry Pi 4 with Oracle Linux 9 in Part 27. We will look at AlmaLinux 8.5 next in Part 17.
Hope you have enjoyed the article. Share your thoughts in the comments or engage in the conversation with me on Twitter @aakarve. I look forward to hearing about your use of MicroShift on ARM devices and if you would like to see something covered in more detail.
References
#MicroShift#Openshift#containers#crio#Edge#raspberry-pi
#oraclelinux