Infrastructure as a Service

 View Only

MicroShift – Part 11: Raspberry Pi 4 with Fedora 35 Server

By Alexei Karve posted Mon March 14, 2022 04:39 PM


MicroShift and KubeVirt on Raspberry Pi 4 with Fedora Server


MicroShift is a research project that is exploring how OpenShift OKD Kubernetes distribution can be optimized for small form factor devices and edge computing. In Part 1 we looked at multiple ways to run MicroShift on a MacBook Pro. In Part 4, we ran MicroShift on the Raspberry Pi 4 with the Raspberry Pi OS (64 bit) and further in Part 9, we looked at Virtualization with MicroShift on Raspberry Pi 4 with Raspberry Pi OS (64 bit). In Part 5, we saw multiple options to build and run MicroShift on the Raspberry Pi 4 with the CentOS 8 Stream (64 bit). In Part 6, we deployed MicroShift on the Raspberry Pi 4 with Ubuntu 20.04 (64 bit). In Part 8, we looked at the All-In-One install of MicroShift on balenaOS. In Part 10, we deployed MicroShift and KubeVirt on Fedora IoT. In this Part 11, we will deploy MicroShift on Fedora Server. We will deploy Node Red on MicroShift with a dashboard to show gauges for SenseHat sensor data. We will also run an object detection sample and send messages to Node Red deployment. Further, we will setup KubeVirt and the OKD Web Console and run Virtual Machine Instances in MicroShift. Finally, we will install InfluxDB/Telegraf/Grafana with a dashboard that will show SenseHat sensor data.

Fedora distributes a generic Fedora Server Edition for Single Board Computers (SBC) for Raspberry Pi. Fedora on SBC hardware uses UEFI as boot system. It creates an efi partition and a small /boot partition, used by grub2 bootloader. Thereafter, it creates another partition including one volume group (VG), which provides a logical volume with XFS file system for the operating system and its software. The rest is left free for other logical volumes that are to hold user data. Separation of system and user data eases system administration, increases security, and decreases error-proneness.

Setting up the Raspberry Pi 4 with Fedora Server

Run the following steps to download the Fedora IoT image and setup the Raspberry Pi 4. Have a display and keyboard attached to the Raspberry Pi 4. For the first boot, these are required to perform an initial bare minimum configuration. Afterwards you will perform everything using either ssh or more comfortably using Cockpit, a graphical, web-based user interface, which is preinstalled and activated by default.

1. Download the latest version of the Fedora IoT Fedora: Raw Image for aarch64 onto your MacBook Pro
wget -O Fedora-Server-35-1.2.aarch64.raw.xz
2. Write to Microsdxc card using Balena Etcher, insert Microsdxc into Raspberry Pi4 and poweron. I used the 64GB Microsdxc card.
3. After the initial boot, Fedora allows you to directly customize the basic settings that will help you to create a user, configure the network and more. The customization is available on a display connected to the Raspberry Pi and not via SSH. If you have a DHCP server on your LAN, the only “strictly necessary” action is to configure the User. Press 5 and then 1 for User Creation, 2 to set the name to fedora, 5 to set the password, 6 for Administrator access and finally c twice to continue.
4. We can login to the cockpit at https://$ipaddress:9090 and configure the storage. We will instead ssh to the VM and perform the necessary steps.

5. Login using fedora user and password created above and enlarge partition and volume group to fill the disk space.

ssh fedora@$ipaddress
sudo su –

sudo fdisk -lu
sudo growpart /dev/mmcblk0 3
sudo fdisk -lu /dev/mmcblk0
sudo pvresize /dev/mmcblk0p3
sudo lvextend -l +100%FREE /dev/mapper/fedora_fedora/root
sudo xfs_growfs -d /

If you used the USB flash drive instead of the Microsdxc, switch the above to use /dev/sda instead of /dev/mmcblk0.

[root@fedora ~]# fdisk -lu
Disk /dev/mmcblk0: 59.48 GiB, 63864569856 bytes, 124735488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8f4e3975

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       2048  1230847  1228800  600M  6 FAT16
/dev/mmcblk0p2      1230848  3327999  2097152    1G 83 Linux
/dev/mmcblk0p3      3328000 14680063 11352064  5.4G 8e Linux LVM

Disk /dev/mapper/fedora_fedora-root: 5.41 GiB, 5809111040 bytes, 11345920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/zram0: 7.64 GiB, 8205107200 bytes, 2003200 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

[root@fedora ~]# growpart /dev/mmcblk0 3
CHANGED: partition=3 start=3328000 old: size=11352064 end=14680064 new: size=121407455 end=124735455

[root@fedora ~]# fdisk -lu /dev/mmcblk0
Disk /dev/mmcblk0: 59.48 GiB, 63864569856 bytes, 124735488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8f4e3975

Device         Boot   Start       End   Sectors  Size Id Type
/dev/mmcblk0p1 *       2048   1230847   1228800  600M  6 FAT16
/dev/mmcblk0p2      1230848   3327999   2097152    1G 83 Linux
/dev/mmcblk0p3      3328000 124735454 121407455 57.9G 8e Linux LVM

[root@fedora ~]# sudo pvresize /dev/mmcblk0p3
  Physical volume "/dev/mmcblk0p3" changed
  1 physical volume(s) resized or updated / 0 physical volume(s) not resized

[root@fedora ~]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        3.8G     0  3.8G   0% /dev
tmpfs                           3.9G     0  3.9G   0% /dev/shm
tmpfs                           1.6G  9.3M  1.6G   1% /run
/dev/mapper/fedora_fedora-root  5.5G  2.5G  3.0G  45% /
tmpfs                           3.9G   20K  3.9G   1% /tmp
/dev/mmcblk0p2                 1014M  154M  861M  16% /boot
/dev/mmcblk0p1                  599M   31M  569M   6% /boot/efi
tmpfs                           783M     0  783M   0% /run/user/1000

[root@fedora ~]# sudo lvextend -l +100%FREE /dev/mapper/fedora_fedora/root
  Size of logical volume fedora_fedora/root changed from 5.41 GiB (1385 extents) to <57.89 GiB (14819 extents).
  Logical volume fedora_fedora/root successfully resized.

[root@fedora ~]# sudo xfs_growfs -d /
meta-data=/dev/mapper/fedora_fedora-root isize=512    agcount=4, agsize=354560 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=1418240, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 1418240 to 15174656

6. Enable wifi (optional)

nmcli device wifi list # Note your ssid
nmcli device wifi connect $ssid --ask

7. Set the hostname with a domain (update with your raspberry pi ipaddress below)

sudo dnf update -y
hostnamectl hostname
echo "$ipaddress fedora" >> /etc/hosts


8. Check the release

[root@fedora ~]# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
[root@fedora ~]# cat /proc/cgroups | column -t # Check that memory and cpuset are present
#subsys_name  hierarchy  num_cgroups  enabled
cpuset        0          93           1
cpu           0          93           1
cpuacct       0          93           1
blkio         0          93           1
memory        0          93           1
devices       0          93           1
freezer       0          93           1
net_cls       0          93           1
perf_event    0          93           1
net_prio      0          93           1
pids          0          93           1
misc          0          93           1
[root@fedora ~]# cat /etc/redhat-release
Fedora release 35 (Thirty Five)
[root@fedora ~]# uname -a
Linux 5.16.12-200.fc35.aarch64 #1 SMP Wed Mar 2 18:49:17 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
[fedora@fedora ~]$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="35 (Server Edition)"
PRETTY_NAME="Fedora Linux 35 (Server Edition)"
VARIANT="Server Edition"

Install the dependencies for MicroShift and SenseHat

Dependencies to build RTIMULib

sudo dnf -y install git cmake zlib zlib-devel libjpeg-devel gcc gcc-c++ i2c-tools python3-devel python3 python3-pip

Dependencies for MicroShift. [Use cri-o:1.23 for Fedora 36 and cri-o:1.24 for Fedora Rawhide]

sudo dnf module enable -y cri-o:1.21
sudo dnf install -y cri-o cri-tools
sudo systemctl enable crio --now
sudo dnf copr enable -y @redhat-et/microshift
sudo dnf install -y microshift-selinux
ls /opt/cni/bin/ # empty
ls /usr/libexec/cni # cni plugins

Setting up libvirtd on the host

sudo dnf -y install libvirt-client libvirt-nss qemu-system-aarch64 virt-manager virt-install virt-viewer
# Works with nftables on Fedora Server and Fedora IoT
# vi /etc/firewalld/firewalld.conf # FirewallBackend=iptables
systemctl enable --now libvirtd

Installing sense_hat and RTIMULib on Fedora IoT

The Sense HAT is an add-on board for the Raspberry Pi. The Sense HAT has an 8 × 8 RGB LED matrix, a five – button joystick and includes the following sensors: Inertial Measurement Unit (Accelerometer, Gyroscope, Magnetometer), Temperature, Barometric pressure, Humidity. If you have the Sense HAT attached, install the libraries. We will install the default libraries; then overwrite the /usr/local/lib/python3.10/site-packages/sense_hat-2.2.0-py3.10.egg/sense_hat/ to use the smbus after installing RTIMULib in a few steps below.

0x5c: LPS25H Pressure
0x1c: LSM9DS1 9-axis iNEMO inertial module (IMU): 3D magnetometer, 3D accelerometer, 3D gyroscope
0x5f: HTS221 Humidity and Temperature
0x46: LED2472G 24-Channels LED driver with LED error detection and gain control
0x6a: LSM9DS1 Accelerometer Gyro Magnetometer

Install sensehat

i2cget -y 1 0x6A 0x75
i2cget -y 1 0x5f 0xf
i2cdetect -y 1
lsmod | grep st_
pip3 install Cython Pillow numpy sense_hat smbus


[root@fedora ~]# i2cget -y 1 0x6A 0x75
[root@fedora ~]# i2cget -y 1 0x5f 0xf
[root@fedora ~]# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- UU -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- 46 -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- UU -- -- 5f
60: -- -- -- -- -- -- -- -- -- -- 6a -- -- -- -- --
70: -- -- -- -- -- -- -- --
[root@fedora ~]# lsmod | grep st_
st_magn_spi            16384  0
st_pressure_spi        16384  0
st_magn_i2c            16384  0
st_sensors_spi         16384  2 st_pressure_spi,st_magn_spi
regmap_spi             16384  1 st_sensors_spi
st_magn                20480  2 st_magn_i2c,st_magn_spi
st_pressure_i2c        16384  0
st_pressure            16384  2 st_pressure_i2c,st_pressure_spi
st_sensors_i2c         16384  2 st_pressure_i2c,st_magn_i2c
st_sensors             28672  6 st_pressure,st_pressure_i2c,st_magn_i2c,st_pressure_spi,st_magn,st_magn_spi
industrialio_triggered_buffer    16384  2 st_pressure,st_magn
industrialio           98304  9 st_pressure,industrialio_triggered_buffer,st_sensors,st_pressure_i2c,kfifo_buf,st_magn_i2c,st_pressure_spi,st_magn,st_magn_spi

We need to blacklist the Industrial I/O modules.

cat << EOF > /etc/modprobe.d/blacklist-industialio.conf
blacklist st_magn_spi
blacklist st_pressure_spi
blacklist st_sensors_spi
blacklist st_pressure_i2c
blacklist st_magn_i2c
blacklist st_pressure
blacklist st_magn
blacklist st_sensors_i2c
blacklist st_sensors
blacklist industrialio_triggered_buffer
blacklist industrialio


Check the Sense Hat with i2cdetect
After the reboot, the 1C and 5C show up.

ssh root@$ipaddress
i2cdetect -y 1


[root@microshift ~]# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- 1c -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- 46 -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- 5c -- -- 5f
60: -- -- -- -- -- -- -- -- -- -- 6a -- -- -- -- --
70: -- -- -- -- -- -- -- --

Install RTIMULib

git clone
cd RTIMULib/
cd Linux/python
python3 build
python3 install
cd ../..
mkdir build
cd build
cmake ..
make -j4
make install
cd /root/RTIMULib/Linux/RTIMULibDrive11
make -j4
make install
RTIMULibDrive11 # Ctrl-C to break
cd /root/RTIMULib/Linux/RTIMULibDrive10
make -j4
make install
RTIMULibDrive10 # Ctrl-C to break

Replace the with the new file that uses SMBus

cd ~
git clone
cd microshift
cd raspberry-pi/sensehat-fedora-iot

pip3 install Cython Pillow numpy sense_hat smbus

# Update the python package to use the i2cbus
cp -f /usr/local/lib/python3.10/site-packages/sense_hat/

Test the SenseHat samples for the Sense Hat's LED matrix

If you have the in local directory, you will get the Error: "OSError: /var/roothome/microshift/raspberry-pi/sensehat-fedora-iot/sense_hat_text.png not found"

# Enable random LEDs
python3 # Ctrl-C to interrupt

# Show multiple screens to test LEDs
python3 # Ctrl-C to interrupt

# Show the Temperature, Pressure and Humidity
python3 # Ctrl-C to interrupt

# Show two digits for multiple numbers

# First time you run the, you may see “Temperature: 0 C”. Just run it again.

# Use the new get_state method from
python3 # U=Up D=Down L=Left R=Right M=Press

Test the USB camera

Install the latest pygame. Note that pygame 1.9.6 will throw “SystemError: set_controls() method: bad call flags”. So, you need to upgrade pygame to 2.1.0.

pip3 install pygame --upgrade
python3 # It will create a file 101.bmp

Install the oc and kubectl client

export OCP_VERSION=4.9.11 && \
    curl -o oc.tar.gz$ARCH/clients/ocp/$OCP_VERSION/openshift-client-linux-$OCP_VERSION.tar.gz && \
    tar -xzvf oc.tar.gz && \
    rm -f oc.tar.gz && \
    install -t /usr/local/bin {kubectl,oc}

Start MicroShift

We will have systemd start and manage MicroShift on this rpm-based host. You may read about selecting zones for your interfaces.

sudo dnf install -y microshift firewalld
sudo systemctl enable firewalld --now
sudo firewall-cmd --zone=trusted --add-source= --permanent
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=443/tcp --permanent
sudo firewall-cmd --zone=public --add-port=5353/udp --permanent
sudo firewall-cmd --reload

For Fedora 36 and Rawhide, replace the microshift binary, see Part 21 for details.

curl -L > /usr/local/bin/microshift
chmod +x /usr/local/bin/microshift
cp /usr/lib/systemd/system/microshift.service /etc/systemd/system/microshift.service
sed -i "s|/usr/bin|/usr/local/bin|" /etc/systemd/system/microshift.service
systemctl daemon-reload
sudo systemctl enable microshift --now

Additional ports may need to be opened. For external access to run kubectl or oc commands against MicroShift, add the 6443 port:

sudo firewall-cmd --zone=public --permanent --add-port=6443/tcp

For access to services through NodePort, add the port range 30000-32767:

sudo firewall-cmd --zone=public --permanent --add-port=30000-32767/tcp

sudo firewall-cmd --reload
firewall-cmd --list-all --zone=public
firewall-cmd --get-default-zone
firewall-cmd --set-default-zone=public
firewall-cmd --get-active-zones

Check the microshift and crio logs

journalctl -u microshift -f
journalctl -u crio -f

It will take around 3 minutes for all pods to start. Check the status of node and pods using kubectl or oc client.

echo "export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig" >> ~/.bash_profile
export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
watch "oc get nodes;oc get pods -A;crictl pods;crictl images"

The microshift service references the microshift binary in the /usr/bin directory

[root@microshift ~]# cat /usr/lib/systemd/system/microshift.service
Description=MicroShift crio.service crio.service

ExecStart=microshift run


If you make changes to the above microshift.service, you need to run the following to take changed configurations from filesystem and regenerate dependency trees.

systemctl daemon-reload

Samples to run on MicroShift

We will run a few samples that will show the use of persistent volume, SenseHat and the USB camera.

1. Postgresql database server

The source code is in github

cd ~
git clone
cd ~/microshift/raspberry-pi/pg

Create a new project pg. Create the configmap, pv, pvc and deployment for PostgreSQL

oc new-project pg
mkdir -p /var/hpvolumes/pg

If you have the selinux set to Enforcing, run the

restorecon -R -v "/var/hpvolumes/*"

oc apply -f hostpathpvc.yaml -f hostpathpv.yaml -f pg-configmap.yaml -f pg.yaml

First time we start postgresql, the logs will show that it creates the database

oc logs deployment.apps/pg-deployment

Connect to the database

oc exec -it deployment.apps/pg-deployment -- bash
psql --host localhost --port 5432 --user postgresadmin --dbname postgresdb # test123 as password

Create a TABLE cities and insert a couple of rows

CREATE TABLE cities (name varchar(80), location point);
INSERT INTO cities VALUES ('Madison', '(89.40, 43.07)'),('San Francisco', '(-122.43,37.78)');
SELECT * from cities;

After we are done, we can delete the deployment and project

oc delete -f pg.yaml -f pg-configmap.yaml -f hostpathpvc.yaml -f hostpathpv.yaml
oc project default
oc delete project pg
rm -rf /var/hpvolumes/pg/

2. Node Red live data dashboard with SenseHat sensor charts

In this sample, we will install Node Red on the ARM device as a deployment within MicroShift, add the dashboard and view the guages for temperature/pressure/humidity data from SenseHat on the dashboard. This Node Red deployment will be used in the next two samples.

cd ~
git clone
cd ~/microshift/raspberry-pi/nodered

Build and push the arm64v8 image “karve/nodered:arm64”

cd docker-custom/
docker push karve/nodered-fedora:arm64
cd ..

Deploy Node Red with persistent volume for /data within the node red container

mkdir /var/hpvolumes/nodered
restorecon -R -v "/var/hpvolumes/*"
rm -rf /var/hpvolumes/nodered/*;cp -r nodered-volume/* /var/hpvolumes/nodered/.
oc new-project nodered
oc apply -f noderedpv.yaml -f noderedpvc.yaml -f nodered2.yaml -f noderedroute.yaml
oc get routes
oc logs deployment/nodered-deployment -f


[root@fedora nodered]# oc logs deployment/nodered-deployment -f
Installing Nodes

> node-red-node-pi-sense-hat@0.1.1 postinstall /data/node_modules/node-red-node-pi-sense-hat
> scripts/

Sense HAT python library is installed
added 138 packages from 209 contributors and audited 138 packages in 27.725s

3 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Starting NodeRed

> node-red-docker@2.1.4 start /usr/src/node-red
> node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"

14 Mar 17:25:17 - [info]

Welcome to Node-RED

14 Mar 17:25:17 - [info] Node-RED version: v2.1.4
14 Mar 17:25:17 - [info] Node.js  version: v14.19.0
14 Mar 17:25:17 - [info] Linux 5.16.12-200.fc35.aarch64 arm64 LE
14 Mar 17:25:18 - [info] Loading palette nodes
14 Mar 17:25:21 - [info] Dashboard version 3.1.6 started at /ui
14 Mar 17:25:21 - [info] Settings file  : /data/settings.js
14 Mar 17:25:21 - [info] Context store  : 'default' [module=memory]
14 Mar 17:25:21 - [info] User directory : /data
14 Mar 17:25:21 - [warn] Projects disabled : editorTheme.projects.enabled=false
14 Mar 17:25:21 - [info] Flows file     : /data/flows.json
14 Mar 17:25:21 - [info] Server now running at
14 Mar 17:25:22 - [warn]

Your flow credentials file is encrypted using a system-generated key.

If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.

You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.

14 Mar 17:25:22 - [info] Starting flows
14 Mar 17:25:22 - [info] Started flows

Add the ipaddress of the Raspberry Pi 4 device for nodered-svc-nodered.cluster.local to /etc/hosts on your Laptop and browse to http://nodered-svc-nodered.cluster.local/

The following modules required for the dashboard have been preinstalled node-red-dashboard, node-red-node-smooth, node-red-node-pi-sense-hat. These can be seen under “Manage Palette - Install”. The Flow 1 or Flow 2 have already been imported from the nodered sample. This import to the Node Red can be done manually under “Import Nodes” and then click “Deploy”. The node-red-node-pi-sense-hat module require a change in the in order to use the that uses smbus and new function for joystick. This change is accomplished by overwriting with the modified in Dockerfile.debianonfedora ( built using and further copied from /tmp directory to the correct volume when the pod starts in nodered2.yaml.

Double click the Sense HAT input node and make sure that all the events are checked. Select the Dashboard. Click on the outward arrow in the tabs to view the sensor charts. If you selected the Flow 1, you could click on the Input for the Timestamp under “Dot Matrix” to see the “Alarm” message scroll on the SenseHat LED.

Environment Control and Temperature Graph

For Flow 2, you can see the state of the Joystick Up, Down, Left, Right or Pressed.

Temperature, Humidity, Pressure and Joystick

We can continue running the next two samples that will reuse this Node Red deployment. If the Node Red Deployment is no longer required, we can delete it as follows:

cd ~/microshift/raspberry-pi/nodered
oc delete -f noderedpv.yaml -f noderedpvc.yaml -f nodered.yaml -f noderedroute.yaml -n nodered


[root@fedora object-detection]# cd ~/microshift/raspberry-pi/nodered
[root@fedora nodered]# oc delete -f noderedpv.yaml -f noderedpvc.yaml -f nodered.yaml -f noderedroute.yaml -n nodered
warning: deleting cluster-scoped resources, not scoped to the provided namespace
persistentvolume "noderedpv" deleted
persistentvolumeclaim "noderedpvc" deleted
deployment.apps "nodered-deployment" deleted
service "nodered-svc" deleted "nodered-route" deleted

3. Sense Hat and USB camera sending messages to Node Red

We will use Node Red to show pictures and chat messages sent from the Raspberry Pi 4 when a person is detected. The Node Red deployment earlier had preinstalled the node-red-contrib-image-tools, node-red-contrib-image-output, and node-red-node-base64 and imported the Chat flow and the Picture (Image) display flow. The Chat can be seen at http://nodered-svc-nodered.cluster.local/ and the images at http://nodered-svc-nodered.cluster.local/#flow/3e30dc50ae28f61f. On the Image flow, click on the square box to the right of image preview or viewer to Deactivate and Activate the Node. You will be able to see the picture when you Activate the Node and run sample below.

cd ~/microshift/raspberry-pi/sensehat

We will use the image by building from the Dockerfile.fedora that used the base image built from Dockerfile. The base image for arm32v7 is from and the wheel for pip from The RTIMULib binaries are from

podman build -f Dockerfile.fedora -t .
podman push

We send pictures and web socket chat messages to Node Red using a pod in MicroShift. Update the URL to point to nodered-svc-nodered.cluster.local and the ip address in hostAliases to your Raspberry Pi 4 ip address in the sensehat-fedora.yaml.

          - name: WebSocketURL
            value: "ws://nodered-svc-nodered.cluster.local/ws/chat"
          - name: VideoSource
            value: "/dev/video0"
          - name: ImageUploadURL
            value: http://nodered-svc-nodered.cluster.local/upload

      - hostnames:
        - nodered-svc-nodered.cluster.local
oc project default
oc apply -f sensehat-fedora.yaml

When we are done, we can delete the deployment

oc delete -f sensehat-fedora.yaml

4. TensorFlow Lite Python object detection example in MicroShift with SenseHat and Node Red

This example requires the same Node Red setup as in the previous Sample 4.

cd ~
git clone
cd ~/microshift/raspberry-pi/object-detection

We will build the image for object detection send pictures and web socket chat messages to Node Red when a person is detected using a pod in microshift.

podman build -f Dockerfile.fedora -t .
podman push

Update the env WebSocketURL and ImageUploadURL as shown below. Also update the hostAliases in object-detection-fedora.yaml to point to your raspberry pi 4 ip address.

          - name: WebSocketURL
            value: "ws://nodered-svc-nodered.cluster.local/ws/chat"
          - name: ImageUploadURL
            value: http://nodered-svc-nodered.cluster.local/upload

      - hostnames:
        - nodered-svc-nodered.cluster.local
oc project default
oc apply -f object-detection-fedora.yaml

We will see pictures being sent to Node Red with a person is detected. When we are done testing, we can delete the deployment

oc delete -f object-detection-fedora.yaml

5. Running a Virtual Machine Instance on MicroShift

We first install the kubevirt operator.

virt-host-validate qemu
LATEST=$(curl -L
echo $LATEST
oc apply -f${LATEST}/kubevirt-operator-arm64.yaml
oc apply -f${LATEST}/kubevirt-cr-arm64.yaml
oc adm policy add-scc-to-user privileged -n kubevirt -z kubevirt-operator
# The .status.phase will show Deploying multiple times and finally Deployed
# We wait till the virt-operator, virt-api, virt-controller and virt-handler pods are started, this can take upto 5 minutes
oc get -n kubevirt -o=jsonpath="{.status.phase}" -w # Ctrl-C to break
oc -n kubevirt wait kv kubevirt --for condition=Available --timeout=300s
oc get pods -n kubevirt

We can build the OKD Web Console (Codename: “bridge”) from the source as mentioned in Part-9. We will run the “bridge” as a container image that we run within MicroShift.

cd /root/microshift/raspberry-pi/console
oc create serviceaccount console -n kube-system
oc create clusterrolebinding console --clusterrole=cluster-admin --serviceaccount=kube-system:console -n kube-system
sleep 5
oc get serviceaccount console --namespace=kube-system -o jsonpath='{.secrets[0].name}'
oc get serviceaccount console --namespace=kube-system -o jsonpath='{.secrets[1].name}'

Replace BRIDGE_K8S_MODE_OFF_CLUSTER_ENDPOINT in okd-web-console-install.yaml with your Raspberry Pi 4 ip address. Also replace the secretRef token for BRIDGE_K8S_AUTH_BEARER_TOKEN from the console-token secret name from one of secrets names from the two outputs above.

oc apply -f okd-web-console-install.yaml
oc expose svc console-np-service -n kube-system
oc logs deployment/console-deployment -f -n kube-system
oc get routes -n kube-system

Add the Raspberry Pi IP address to /etc/hosts on your Macbook Pro to resolve console-np-service-kube-system.cluster.local. Now you can access the OKD Web Console from your Laptop http://console-np-service-kube-system.cluster.local/

We can optionally preload the fedora image into crio.

crictl pull

Now let’s create a Fedora Virtual Machine Instance using the vmi-fedora.yaml.

cd /root/microshift/raspberry-pi/vmi
oc apply -f vmi-fedora.yaml
watch oc get vmi,pods

The Output for the virtualmachineinstance PHASE goes from “Scheduling” to “Scheduled” to “Running” after the virt-launcher-vmi-fedora pod STATUS goes from “Init” to “Running”. Note down the ip address of the vmi-fedora Virtual Machine Instance. Now we create a Pod to run ssh client and connect to the Fedora VM from this pod.

kubectl run alpine --privileged --rm -ti --image=alpine -- /bin/sh
apk update && apk add --no-cache openssh-client


kubectl run sshclient --privileged --rm -ti --image=karve/alpine-sshclient:arm64 -- /bin/sh
#kubectl attach sshclient -c sshclient -i -t

From the sshclient pod, ssh to the Fedora VMI from this sshclient container.

[root@fedora vmi]# oc get vmi,pods
NAME                                            AGE   PHASE     IP           NODENAME             READY   99s   Running   True

NAME                                      READY   STATUS    RESTARTS   AGE
pod/nodered-deployment-5bdddb5d94-szw4p   1/1     Running   0          35m
pod/virt-launcher-vmi-fedora-h77q8        2/2     Running   0          99s
[root@fedora vmi]# kubectl run alpine --privileged --rm -ti --image=alpine -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # apk update && apk add --no-cache openssh-client
v3.15.0-341-gbacd756db7 []
v3.15.0-340-g4ed6115e99 []
OK: 15734 distinct packages available
(1/6) Installing openssh-keygen (8.8_p1-r1)
(2/6) Installing ncurses-terminfo-base (6.3_p20211120-r0)
(3/6) Installing ncurses-libs (6.3_p20211120-r0)
(4/6) Installing libedit (20210910.3.1-r0)
(5/6) Installing openssh-client-common (8.8_p1-r1)
(6/6) Installing openssh-client-default (8.8_p1-r1)
Executing busybox-1.34.1-r3.trigger
OK: 10 MiB in 20 packages
/ # ssh fedora@
The authenticity of host ' (' can't be established.
ED25519 key fingerprint is SHA256:wpD6lU+d2T5R++UiRfuu5UyhR7VqCIy7F7kIRBabs6Q.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '' (ED25519) to the list of known hosts.
fedora@'s password:
[fedora@vmi-fedora ~]$ ping
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=117 time=4.35 ms
64 bytes from ( icmp_seq=2 ttl=117 time=3.99 ms
--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 3.994/4.171/4.348/0.177 ms
[fedora@vmi-fedora ~]$ exit
Connection to closed.
/ # exit
Session ended, resume using 'kubectl attach alpine -c alpine -i -t' command when the pod is running
pod "alpine" deleted
[root@fedora vmi]# oc delete -f vmi-fedora.yaml "vmi-fedora" deleted
[root@fedora vmi]#

6. Running a Virtual Machine on MicroShift

This meeds the kubevirt-operator-arm64 installed as in “4. Running a Virtual Machine Instance on MicroShift” above.

Copy virtctl binary from prebuilt image to /usr/local/bin

id=$(podman create
podman cp $id:_out/cmd/virtctl/virtctl /usr/local/bin
podman rm -v $id


[root@fedora vmi]# id=$(podman create
Trying to pull
Getting image source signatures
Copying blob 7065f6098427 done
Copying config 1c7a5aa443 done
Writing manifest to image destination
Storing signatures
[root@fedora vmi]# podman cp $id:_out/cmd/virtctl/virtctl /usr/local/bin
[root@fedora vmi]# podman rm -v $id

The virtctl binary can also be built as shown in the “Building KubeVirt” section.

Create the Virtual Machine using the vm-fedora.yaml. The Virtual Machine will show Stopped Status.

cd /root/microshift/raspberry-pi/vmi
oc apply -f vm-fedora.yaml
oc get pods,vm,vmi -n default


[root@fedora vmi]# oc apply -f vm-fedora.yaml created
[root@fedora vmi]# oc get vm,vmi,pods -n default
NAME                                    AGE   STATUS    READY   33s   Stopped   False

Start the virtual machine

virtctl start vm-example -n default


kubectl patch virtualmachine vm-example --type merge -p  '{"spec":{"running":true}}' -n default
oc get pods,vm,vmi -n default


[root@fedora vmi]# virtctl start vm-example -n default
VM vm-example was scheduled to start
[root@fedora vmi]# oc get vm,vmi,pods -n default
NAME                                    AGE   STATUS     READY   70s   Starting   False

NAME                                            AGE   PHASE       IP    NODENAME             READY   7s    Scheduled   False

NAME                                 READY   STATUS    RESTARTS   AGE
pod/virt-launcher-vm-example-6h6s9   2/2     Running   0          7s

[root@fedora vmi]# oc get vm,vmi,pods -n default
NAME                                    AGE     STATUS    READY   7m54s   Running   True

NAME                                            AGE     PHASE     IP           NODENAME             READY   6m51s   Running   True

NAME                                 READY   STATUS    RESTARTS   AGE
pod/virt-launcher-vm-example-6h6s9   2/2     Running   0          6m52s

We can use the virtctl to connect to the VMI vm-example VMI with “virtctl console”.

virtctl console vm-example
# Login as fedora/fedora


[root@fedora vmi]# virtctl console vm-example -n default
Successfully connected to vm-example console. The escape sequence is ^]

vm-example login: fedora
ping [fedora@vm-example ~]$ ping
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=116 time=4.90 ms
64 bytes from ( icmp_seq=2 ttl=116 time=4.00 ms
64 bytes from ( icmp_seq=3 ttl=116 time=4.33 ms

--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.996/4.408/4.900/0.373 ms
[fedora@vm-example ~]$ exit

Fedora 32 (Cloud Edition)
Kernel 5.6.6-300.fc32.aarch64 on an aarch64 (ttyAMA0)

SSH host key: SHA256:tdv1ws2EChY3geGQMc8Hc54fpMsaE7badmW7gJci7AI (RSA)
SSH host key: SHA256:431Is8lmx16VHkN5fatO1ZKB2RpvtO1nZ4I/S4GOSV4 (ECDSA)
SSH host key: SHA256:oxxHYVY++9z6TBJ4GH4am49Gl++VQ7ao9ZHhr0iYMj8 (ED25519)
eth0: fe80::5054:ff:fe8d:6116
vm-example login: [root@fedora vmi]# 

Alternatively note down the ip address of the vm-example and ssh to it from a sshclient container as shown in previous sample.

We can access the OKD Web Console and run Actions (Restart, Stop, Pause etc) on the VM http://console-np-service-kube-system.cluster.local/

Stop the virtual machine using virtctl:

virtctl stop vm-example -n default

or the kubectl patch command:

kubectl patch virtualmachine vm-example --type merge -p '{"spec":{"running":false}}' -n default


[root@fedora vmi]# oc get vm,vmi,pods -n default
NAME                                    AGE   STATUS     READY   61m   Stopping   False

NAME                                            AGE   PHASE     IP           NODENAME             READY   60m   Running   False

NAME                                 READY   STATUS        RESTARTS   AGE
pod/virt-launcher-vm-example-6h6s9   2/2     Terminating   0          60m

7. Install Metrics Server

This will enable us to run the “kubectl top” and “oc adm top” commands.

curl -L > metrics-server-components.yaml
oc apply -f metrics-server-components.yaml

# Wait for the metrics-server to start in the kube-system namespace
oc get deployment metrics-server -n kube-system
oc get events -n kube-system
oc logs deployment/metrics-server -n kube-system


[root@fedora vmi]# oc adm top pods -A
NAMESPACE                       NAME                                  CPU(cores)   MEMORY(bytes)
kube-system                     console-deployment-77dbf58b88-2jvdp   2m           16Mi
kube-system                     kube-flannel-ds-68rgv                 7m           10Mi
kube-system                     metrics-server-64cf6869bd-n6ghb       17m          15Mi
kubevirt                        virt-api-7c66dc7874-c29t8             8m           85Mi
kubevirt                        virt-api-7c66dc7874-lnxmb             6m           85Mi
kubevirt                        virt-controller-8546dcf8c-4hh82       10m          85Mi
kubevirt                        virt-controller-8546dcf8c-vm76z       11m          82Mi
kubevirt                        virt-handler-q7n27                    3m           95Mi
kubevirt                        virt-operator-6d4bff4fd4-bpw9w        10m          104Mi
kubevirt                        virt-operator-6d4bff4fd4-n46hq        6m           93Mi
kubevirt-hostpath-provisioner   kubevirt-hostpath-provisioner-xbqs4   2m           6Mi
nodered                         nodered-deployment-5bdddb5d94-szw4p   1m           83Mi
openshift-dns                   dns-default-jxt49                     7m           19Mi
openshift-dns                   node-resolver-s7p2t                   0m           4Mi
openshift-ingress               router-default-85bcfdd948-bsd44       3m           38Mi
openshift-service-ca            service-ca-7764c85869-nqnxc           13m          33Mi
[root@fedora vmi]# oc adm top nodes
NAME                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   626m         15%    4004Mi          51%

8. InfluxDB/Telegraf/Grafana

The source code is available for this influxdb sample in github.

cd ~
git clone
cd microshift/raspberry-pi/influxdb

We create and push the “measure-fedora:latest” image using the Dockerfile. If you want to run all the steps in a single command, just execute the

oc get route grafana-service -o jsonpath --template="http://{}/login{'\n'}"


[root@fedora influxdb]# oc get route grafana-service -o jsonpath --template="http://{}/login{'\n'}"

The script will create a new project influxdb for this sample, install InfluxDB, install the pod for SenseHat measurements, install Telegraf and check the measurements for the telegraf database in InfluxDB. Finally, it will install Grafana. Add the "<RaspberryPiIPAddress> grafana-service-influxdb.cluster.local" to /etc/hosts on your laptop and login to http://grafana-service-influxdb.cluster.local/login using admin/admin. You will need to change the password on first login. Go to the Dashboards list (left menu > Dashboards > Manage).

Open the Analysis Server dashboard to display monitoring information for MicroShift.

Analysis Server

Open the Balena Sense dashboard to show the temperature, pressure and humidity from SenseHat.

SenseHat Gauges

Finally, after you are done working with this sample, you can run the


Cleanup MicroShift

We can use the ~/microshift/hack/ script to remove the pods and images.


[root@fedora vmi]# cd ~/microshift/hack 
[root@fedora hack]# ./
DATA LOSS WARNING: Do you wish to stop and cleanup ALL MicroShift data AND cri-o container workloads?
1) Yes
2) No
#? 1
Stopping microshift
Removing crio pods
Removing crio containers
Removing crio images
Killing conmon, pause processes
Removing /var/lib/microshift
Cleanup succeeded

Containerized MicroShift on Fedora IoT

We can run MicroShift within containers in two ways:

  1. MicroShift Containerized – The MicroShift binary runs in a Podman container, CRI-O Systemd service runs directly on the host and data is stored at /var/lib/microshift and /var/lib/kubelet on the host VM.
  1. MicroShift Containerized All-In-One – The MicroShift binary and CRI-O service run within a Docker container and data is stored in a docker volume, microshift-data. This should be used for “Testing and Development” only.

Microshift Containerized

We can use the latest prebuilt image.
podman pull $IMAGE

podman run --rm --ipc=host --network=host --privileged -d --name microshift -v /var/run:/var/run -v /sys:/sys:ro -v /var/lib:/var/lib:rw,rshared -v /lib/modules:/lib/modules -v /etc:/etc -v /run/containers:/run/containers -v /var/log:/var/log -v /var/hpvolumes:/var/hpvolumes -e KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig $IMAGE
export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
watch "podman ps;oc get nodes;oc get pods -A;crictl pods"

The microshift container will run using podman and the rest of the pods in crio. Now, we can run the samples shown earlier.

After we are done, we can delete the microshift container. The --rm we used in the podman run will delete the container when we stop it.

podman stop microshift

After it is stopped, we can run the to delete the pods and images from crio.

MicroShift Containerized All-In-One

Let’s stop the crio on the host, we will be creating an all-in-one container in docker that will have crio within the container.

systemctl stop crio
systemctl disable crio

We will run the all-in-one microshift in podman using prebuilt images. I had to mount the /sys/fs/cgroup in the podman run command.  The “sudo setsebool -P container_manage_cgroup true” did not work. We can just volume mount /sys/fs/cgroup into the container using -v /sys/fs/cgroup:/sys/fs/cgroup:ro. This will volume mount in /sys/fs/cgroup into the container as read/only, but the subdir/mount points will be mounted in as read/write.

podman volume rm microshift-data;podman volume create microshift-data
podman run -d --rm --name microshift -h --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /lib/modules:/lib/modules -v microshift-data:/var/lib -v /var/hpvolumes:/var/hpvolumes -p 6443:6443 -p 8080:8080 -p 80:80

We can inspect the microshift-data volume to find the path

[root@fedora hack]# podman volume inspect microshift-data
        "Name": "microshift-data",
        "Driver": "local",
        "Mountpoint": "/var/lib/containers/storage/volumes/microshift-data/_data",
        "CreatedAt": "2022-03-13T10:51:41.468851891-04:00",
        "Labels": {},
        "Scope": "local",
        "Options": {}

On the host raspberry pi, we set KUBECONFIG to point to the kubeconfig on the data volume at the Mountpoint from above

export KUBECONFIG=/var/lib/containers/storage/volumes/microshift-data/_data/microshift/resources/kubeadmin/kubeconfig
# crio on host is stopped, so we do not run crictl commands on host
watch "oc get nodes;oc get pods -A"

The crio service is stopped on the Raspberry Pi, so crictl command will not work directly on the Pi. The crictl commands will work within the microshift container in podman.

podman exec -it microshift crictl ps -a

We can run the samples through the crio within the all-in-one podman container. To run the Virtual Machine examples in the all-in-one MicroShift, we need to execute the mount with --make-shared as follows in the microshift container. We may also preload the virtual machine images.

mount --make-shared /


[root@fedora kubevirt]# podman exec -it microshift bash
[root@fedora /]# mount --make-shared /
[root@fedora /]# crictl pull
Image is up to date for
[root@fedora /]# exit

Building KubeVirt

We can build the KubeVirt binaries from source.

git clone
cd kubevirt
vi ~/kubevirt/hack/

Hardcode the podman in the determine_cri_bin() as shown below to avoid the error “no working container runtime found. Neither docker nor podman seems to work.”

determine_cri_bin() {
    echo podman

Run the make command below to build KubeVirt

make bazel-build

After the build is complete, we can view the kubevirt-bazel-server container

podman exec kubevirt-bazel-server ls _out/cmd/virtctl
podman stop kubevirt-bazel-server


[root@fedora vmi]# podman exec kubevirt-bazel-server ls _out/cmd/virtctl

Copy the virtctl binary to /usr/local/bin

cp _out/cmd/virtctl/virtctl /usr/local/bin
cd ..
rm -rf kubevirt


1. OSError: Cannot detect RPi-Sense FB device

[root@fedora ~]# python3
Traceback (most recent call last):
  File "", line 4, in <module>
    sense = SenseHat()
  File "/usr/local/lib/python3.6/site-packages/sense_hat/", line 39, in __init__
    raise OSError('Cannot detect %s device' % self.SENSE_HAT_FB_NAME)
OSError: Cannot detect RPi-Sense FB device

To solve this, use the new that uses smbus.

2. CrashloopBackoff: dns-default and service-ca

The following patch may be required if the dns-default pod in the openshift-dns namespace keeps restarting.

oc patch daemonset/dns-default -n openshift-dns -p '{"spec": {"template": {"spec": {"containers": [{"name": "dns","resources": {"requests": {"cpu": "80m","memory": "90Mi"}}}]}}}}'

You may also need to patch the service-ca deployment if it keeps restarting:

oc patch deployments/service-ca -n openshift-service-ca -p '{"spec": {"template": {"spec": {"containers": [{"name": "service-ca-controller","args": ["-v=4"]}]}}}}'


In this Part 11, we saw multiple options to run MicroShift on the Raspberry Pi 4 with the Fedora 35 Server. We ran samples that used persistent volume for postgresql, Sense Hat, and USB camera. We saw an object detection sample that sent pictures and web socket messages to Node Red when a person was detected. Next, we installed the OKD Web Console and saw how to manage a VM using KubeVirt on MicroShift. Finally, we installed InfluxDB/Telegraf/Grafana with a dashboard to show SenseHat sensor data. We will work with Kata Containers in Part 23. In the next Part 12, we will work with MicroShift on Raspberry Pi 4 with Fedora CoreOS.

Hope you have enjoyed the article. Share your thoughts in the comments or engage in the conversation with me on Twitter @aakarve. I look forward to hearing about your use of MicroShift on ARM devices and if you would like to see something covered in more detail.


​ ​