Microshift on Raspberry Pi 4 with Raspberry Pi OS (64 bit)
Introduction
MicroShift is a research project that is exploring how OpenShift OKD Kubernetes distribution can be optimized for small form factor devices and edge computing. In the previous Parts (Part 1, Part 2 and Part 3) of this series, we looked at multiple ways to run MicroShift on a MacBook Pro and the Jetson Nano. In this Part 4, we will set up and deploy MicroShift on Raspberry Pi OS (64 bit). In Part 5 and Part 6 we will continue working with the Raspberry Pi 4. In Part 5, we will install Microshift on CentOS Stream 8 (64 bit) and in Part 6 on Ubuntu 20.04 (64 bit).
Raspbian is a 32-bit fork of Debian Linux that's optimized for the Pi. The official operating system for the Raspberry Pi changed its name from Raspbian to Raspberry Pi OS. The new 64-bit operating system doesn't use software from the Raspbian project. It now offers 64-bit support to go together with the new SBCs shipping with 8GB RAM.
Setting up the Raspberry Pi 4 with Raspberry Pi OS (64 bit)
We download the Raspberry Pi OS image zip and write to Microsdxc card, setup the sshd and hostname.
- Download the image from https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2021-11-08/2021-10-30-raspios-bullseye-arm64.zip
- Write to Microsdxc card
- Have a Keyboard and Monitor connected to the Raspberry Pi
- Insert Microsdxc into Raspberry Pi4 and poweron, it will reboot after the file system is resized
- After first boot, you can change the pi password
- Enable ssh using sudo raspi-config -> “3 Interface Options” -> “P2 SSH”
- Set the hostname raspberrypi.example.com in “1 System Options” -> “S4 Hostname”
We login to the Raspberry Pi 4
ssh pi@$ipaddress
sudo su –
Check the release and check that the SELinux status is disabled
cat /etc/os-release
sestatus
root@raspberrypi:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@raspberrypi:/etc# sestatus
SELinux status: disabled
Control groups allow for allocating resources among user-defined processes on a system. Control groups is a Linux kernel feature that enables organization of processes into hierarchically ordered groups - cgroups. The hierarchy (control groups tree) is defined by providing structure to cgroups virtual file system, mounted by default on the /sys/fs/cgroup/ directory by creating and removing sub-directories
Update cgroup kernel parameters: concatenate onto the end of the existing line (do not add a new line) in /boot/cmdline.txt and reboot
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
The Sense HAT “Hardware attached on top” is an add-on board for the Raspberry Pi. The Sense HAT has an 8 × 8 RGB LED matrix, a five – button joystick and includes the following sensors: Inertial Measurement Unit (Accelerometer, Gyroscope, Magnetometer), Temperature, Barometric pressure, Humidity. If you have the Sense HAT attached, test it with i2cdetect. The 46 output for the LED matrix shows UU, that is fine.
root@raspberrypi:~# i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- 1c -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- UU -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- 5c -- -- 5f
60: -- -- -- -- -- -- -- -- -- -- 6a -- -- -- -- --
70: -- -- -- -- -- -- -- --
0x5c: LPS25H Pressure
0x1c: LSM9DS1 9-axis iNEMO inertial module (IMU): 3D magnetometer, 3D accelerometer, 3D gyroscope
0x5f: HTS221 Humidity and Temperature
0x46: LED2472G 24-Channels LED driver with LED error detection and gain control
0x6a: LSM9DS1 Accelerometer Gyro Magnetometer
Test the Sense Hat's LED matrix - Run the sparkles.py and check the pixels changing color
cat << EOF > sparkles.py
from sense_hat import SenseHat
from random import randint
from time import sleep
sense = SenseHat()
while True:
x = randint(0, 7)
y = randint(0, 7)
r = randint(0, 255)
g = randint(0, 255)
b = randint(0, 255)
sense.set_pixel(x, y, r, g, b)
sleep(0.1)
EOF
python3 sparkles.py # Ctrl-C to interrupt
Test the temperature sensor - Create the temperature.py and check the temperature reading
cat << EOF > temperature.py
from sense_hat import SenseHat
sense = SenseHat()
temp = sense.get_temperature_from_pressure()
#temp = sense.get_temperature_from_humidity()
print("Temperature: %s C" % temp)
sense.show_message("{:.1f} C".format(temp))
EOF
python3 temperature.py
Test the USB camera
Create the testcam.py and test the USB camera - Install the latest pygame. Note that pygame 1.9.6 will throw “SystemError: set_controls() method: bad call flags”. So, we need to upgrade pygame to 2.1.0.
pip3 install pygame --upgrade
cat << EOF > testcam.py
import pygame, sys
from pygame.locals import *
import pygame.camera
pygame.init()
pygame.camera.init()
cam = pygame.camera.Camera("/dev/video0",(352,288))
cam.start()
image= cam.get_image()
pygame.image.save(image,'101.bmp')
cam.stop()
EOF
python3 testcam.py # It will create a file 101.bmp
Install MicroShift
Clone the microshift repo so we can run the install.sh script
sudo su –
git clone https://github.com/thinkahead/microshift.git
Hardcode the DISTRO=ubuntu and OS_VERSION=20.04
root@raspberrypi:~/microshift# git diff install.sh
diff --git a/install.sh b/install.sh
index 9e21c619..a0721e94 100755
--- a/install.sh
+++ b/install.sh
@@ -17,7 +17,8 @@ CONFIG_ENV_ONLY=${CONFIG_ENV_ONLY:=false}
# Function to get Linux distribution
get_distro() {
DISTRO=$(egrep '^(ID)=' /etc/os-release| sed 's/"//g' | cut -f2 -d"=")
- if [[ $DISTRO != @(rhel|fedora|centos|ubuntu) ]]
+ DISTRO=ubuntu
+ if [[ $DISTRO != @(rhel|fedora|centos|ubuntu|debian) ]]
then
echo "This Linux distro is not supported by the install script"
exit 1
@@ -37,6 +38,7 @@ get_arch() {
# Function to get OS version
get_os_version() {
OS_VERSION=$(egrep '^(VERSION_ID)=' /etc/os-release | sed 's/"//g' | cut -f2 -d"=")
+ OS_VERSION=20.04
}
You could use the https://github.com/thinkahead/microshift/blob/main/install-raspberry-pi-os.sh where the changes are already made. You will need another change as mentioned in the Errors section below if it does not find crio 1.21.
Set the hostname with domain (if not already set)
hostnamectl set-hostname raspberrypi.example.com
# the host needs a fqdn domain for MicroShift to work well
Run the install script
./install.sh
We can get more details about the microshift service with
systemctl show microshift.service
To check the microshift systemd service, check the file /lib/systemd/system/microshift.service. It shows that the microshift binary is in /usr/local/bin/ directory.
root@raspberrypi:# cat /lib/systemd/system/microshift.service
[Unit]
Description=MicroShift
After=crio.service
[Service]
WorkingDirectory=/usr/local/bin/
ExecStart=microshift run
Restart=always
User=root
[Install]
WantedBy=multi-user.target
To start microshift and check the status and logs, we can run
systemctl start microshift
systemctl status microshift
journalctl -u microshift -f
Install the oc client
wget https://mirror.openshift.com/pub/openshift-v4/arm64/clients/ocp/candidate/openshift-client-linux.tar.gz
mkdir tmp;cd tmp
tar -zxvf ../openshift-client-linux.tar.gz
mv -f oc /usr/local/bin
cd ..;rm -rf tmp
It will take around 3 minutes for all pods to start. Check the status of node and pods using kubectl or oc client.
export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
watch "kubectl get nodes;kubectl get pods -A;crictl pods;crictl images"
#watch "oc get nodes;oc get pods -A;crictl pods;crictl images"
Output when microshift is started
Every 2.0s: kubectl get nodes;kubectl get pods -A;crictl pods;crictl images raspberrypi.example.com: Fri Dec 3 07:53:09 2021
NAME STATUS ROLES AGE VERSION
raspberrypi.example.com Ready <none> 3m2s v1.21.0
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-5mbjn 1/1 Running 0 2m59s
kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-prk4t 1/1 Running 0 3m1s
openshift-dns dns-default-rhrt6 2/2 Running 0 3m
openshift-dns node-resolver-h5dvr 1/1 Running 0 3m
openshift-ingress router-default-85bcfdd948-6v4g8 1/1 Running 0 3m1s
openshift-service-ca service-ca-76674bfb58-f2rmd 1/1 Running 0 3m3s
POD ID CREATED STATE NAME NAMESPACE ATTEMPT R
UNTIME
798d692c1c73f 40 seconds ago Ready dns-default-rhrt6 openshift-dns 0 (
default)
325066912f355 41 seconds ago Ready router-default-85bcfdd948-6v4g8 openshift-ingress 0 (
default)
e854d516b8ec9 2 minutes ago Ready service-ca-76674bfb58-f2rmd openshift-service-ca 0 (
default)
b22a6d7c46d80 2 minutes ago Ready kubevirt-hostpath-provisioner-prk4t kubevirt-hostpath-provisioner 0 (
default)
bde7383c29095 2 minutes ago Ready kube-flannel-ds-5mbjn kube-system 0 (
default)
93228e1009894 2 minutes ago Ready node-resolver-h5dvr openshift-dns 0 (
default)
IMAGE TAG IMAGE ID SIZE
k8s.gcr.io/pause 3.2 2a060e2e7101d 489kB
quay.io/microshift/cli 4.8.0-0.okd-2021-10-10-030117 33a276ba2a973 205MB
quay.io/microshift/coredns 4.8.0-0.okd-2021-10-10-030117 67a95c8f15902 265MB
quay.io/microshift/flannel-cni 4.8.0-0.okd-2021-10-10-030117 0e66d6f50c694 8.78MB
quay.io/microshift/flannel 4.8.0-0.okd-2021-10-10-030117 85fc911ceba5a 68.1MB
quay.io/microshift/haproxy-router 4.8.0-0.okd-2021-10-10-030117 37292c44812e7 225MB
quay.io/microshift/hostpath-provisioner 4.8.0-0.okd-2021-10-10-030117 fdef3dc1264ad 39.3MB
quay.io/microshift/kube-rbac-proxy 4.8.0-0.okd-2021-10-10-030117 7f149e453e908 41.5MB
quay.io/microshift/service-ca-operator 4.8.0-0.okd-2021-10-10-030117 0d3ab44356260 276MB
Build the MicroShift binary for arm64 on Raspberry Pi OS (64 bit)
We can replace the microshift binary that was download from the install.sh script with our own. Let’s build the microshift binary from scratch. Clone the microshift repository from github, install golang, run make and finally move the microshift binary to /usr/local/bin.
sudo su -
apt -y install build-essential curl libgpgme-dev pkg-config libseccomp-dev
# Install golang
wget https://golang.org/dl/go1.17.2.linux-arm64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.2.linux-arm64.tar.gz
rm -f go1.17.2.linux-arm64.tar.gz
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/root/go
cat << EOF >> /root/.bashrc
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/root/go
EOF
mkdir $GOPATH
git clone https://github.com/thinkahead/microshift.git
cd microshift
make
./microshift version
ls -las microshift # binary in current directory /root/microshift
mv microshift /usr/local/bin/microshift
systemctl restart microshift
Output:
root@raspberrypi:~# git clone https://github.com/thinkahead/microshift.git
root@raspberrypi:~# cd microshift
root@raspberrypi:~/microshift# make
go build -mod=vendor -tags 'include_gcs include_oss containers_image_openpgp gssapi providerless netgo osusergo' -ldflags "-X k8s.io/component-base/version.gitMajor=1 -X k8s.io/component-base/version.gitMajor=1 -X k8s.io/component-base/version.gitMinor=21 -X k8s.io/component-base/version.gitVersion=v1.21.0 -X k8s.io/component-base/version.gitCommit=c3b9e07a -X k8s.io/component-base/version.gitTreeState=clean -X k8s.io/component-base/version.buildDate=2021-12-03T11:20:58Z -X k8s.io/client-go/pkg/version.gitMajor=1 -X k8s.io/client-go/pkg/version.gitMinor=21 -X k8s.io/client-go/pkg/version.gitVersion=v1.21.1 -X k8s.io/client-go/pkg/version.gitCommit=b09a9ce3 -X k8s.io/client-go/pkg/version.gitTreeState=clean -X k8s.io/client-go/pkg/version.buildDate=2021-12-03T11:20:58Z -X github.com/openshift/microshift/pkg/version.versionFromGit=4.8.0-0.microshift-unknown -X github.com/openshift/microshift/pkg/version.commitFromGit=63e378fb -X github.com/openshift/microshift/pkg/version.gitTreeState=dirty -X github.com/openshift/microshift/pkg/version.buildDate=2021-12-03T11:20:58Z -s -w" github.com/openshift/microshift/cmd/microshift
root@raspberrypi:~/microshift# ./microshift version
MicroShift Version: 4.8.0-0.microshift-unknown
Base OKD Version: 4.8.0-0.okd-2021-10-10-030117
root@raspberrypi:~/microshift# ls -las microshift
147672 -rwxr-xr-x 1 root root 151211021 Dec 3 06:30 microshift
root@raspberrypi:~/microshift# mv microshift /usr/local/bin/microshift
We may also download the latest microshift binary from github as follows:
ARCH=arm64
export VERSION=$(curl -s https://api.github.com/repos/redhat-et/microshift/releases | grep tag_name | head -n 1 | cut -d '"' -f 4) && \
curl -LO https://github.com/redhat-et/microshift/releases/download/$VERSION/microshift-linux-${ARCH}
chmod +x microshift-linux-${ARCH}
mv microshift-linux-${ARCH} /usr/local/bin/microshift
systemctl restart microshift
Install Docker on Raspberry Pi OS
Although we do not need docker for microshift, we will build images using docker and test some containers. So, let’s install docker.
apt-get install -y ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io
Samples to run on MicroShift
We will run a few samples that will show the use of helm, persistent volume, SenseHat and the USB camera. The first two samples are the same as in Part 3 that we ran on the Jetson Nano.
1. Mysql database server
Download helm and run the mysql server in a container with hostpath persistent volume and a mysql client container
# Install helm
curl -o helm-v3.5.2-linux-arm64.tar.gz https://get.helm.sh/helm-v3.5.2-linux-arm64.tar.gz
tar -zxvf helm-v3.5.2-linux-arm64.tar.gz
cp linux-arm64/helm /usr/local/bin
rm -rf linux-arm64
rm -f helm-v3.5.2-linux-arm64.tar.gz
chmod 600 /var/lib/microshift/resources/kubeadmin/kubeconfig
# Add the repo for mysql helm chart
helm repo add stable https://charts.helm.sh/stable
oc project default
# Install mysql with provided image tag (hacky way to use the sha256 tag for the arm64 image) and my-user as the userid with custom passwords for root and my-user
helm install mysql stable/mysql --set mysqlRootPassword=secretpassword,mysqlUser=my-user,mysqlPassword=my-password,mysqlDatabase=my-database --set persistence.enabled=true --set storageClass=kubevirt-hostpath-provisioner --set image=mysql/mysql-server@sha256 --set imageTag=5e373bcea878b3657937c68cdefa8a1504f53e356ac19a3e51bf515e41e0c48c
helm list
# Remember to delete the /var/hpvolumes/mysql if it already exists (otherwise it will use old password from previous run)
rm -rf /var/hpvolumes/mysql
cat << EOF > hostpathpv.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: hostpath-provisioner
spec:
#storageClassName: "kubevirt-hostpath-provisioner"
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/hpvolumes/mysql"
...
EOF
# Create the persistent volume
kubectl apply -f hostpathpv.yaml
# Wait for the pod to be Running
kubectl get pods -n default -w
kubectl get svc mysql # Note down the CLUSTER-IP
export ipofmysqlserver=
2. Nginx web server
Create the file nginx.yaml
cat << EOF > nginx.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginxinc/nginx-unprivileged:alpine # arm64 image
ports:
- containerPort: 8080
# resource required for hpa
resources:
requests:
memory: 128M
cpu: 125m
limits:
memory: 1024M
cpu: 1000m
---
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
labels:
app: nginx
spec:
type: NodePort
ports:
- port: 8080
nodePort: 30080
selector:
app: nginx
...
EOF
# Create the deployment and service. Test it.
kubectl apply -f nginx.yaml
kubectl get svc nginx-svc # see the port 8080:30080
Now we can access nginx on the NodePort 30080
curl localhost:30080
If we add following the line to /etc/hosts on the Raspberry Pi 4
127.0.0.1 localhost nginx-svc-default.cluster.local
and run
oc expose svc nginx-svc
We can access nginx on port 80 using the route on the Raspberry Pi 4
curl localhost
Finally on your Laptop/MacBook Pro, you can add the line with ipaddress of the Raspberry Pi 4 with nginx-svc-default.cluster.local to /etc/hosts. Then access nginx at http://nginx-svc-default.cluster.local/
We can delete nginx with:
oc delete -f nginx.yaml
oc delete route nginx-svc
3. Sample with Sense Hat and USB camera
Let’s install Node Red on IBM Cloud. We will use Node Red to show pictures and chat messages sent from the Raspberry Pi 4. Alternatively, we can use the Node Red that we deployed as an application in MicroShift on the MacBook Pro in VirtualBox in Part 1.
- Create an IBM Cloud free tier account at https://www.ibm.com/cloud/free and login to Console (top right).
- Create an API Key and save it, Manage->Access->IAM->API Key->Create an IBM Cloud API Key
- Click on Catalog and Search for "Node-Red App", select it and click on "Get Started"
- Give a unique App name, for example xxxxx-node-red and select the region nearest to you
- Select the Pricing Plan Lite, if you already have an existing instance of Cloudant, you may select it in Pricing Plan
- Click Create
- Under Deployment Automation -> Configure Continuous Delivery, click on "Deploy your app"
- Select the deployment target Cloud Foundry that provides a Free-Tier of 256 MB cost-free or Code Engine. The latter has monthly limits and takes more time to deploy. [ Note: Cloud Foundry is deprecated, use the IBM Cloud Code Engine. Any IBM Cloud Foundry application runtime instances running IBM Cloud Foundry applications will be permanently disabled and deprovisioned ]
- Enter the IBM Cloud API Key from Step 2, or click on "New" to create one
- The rest of the fields Region, Organization, Space will automatically get filled up. Use the default 256MB Memory and click "Next"
- In "Configure the DevOps toolchain", click Create
- Wait for 10 minutes for the Node Red instance to start
- Click on the "Visit App URL"
- On the Node Red page, create a new userid and password
- In Manage Palette, install the node-red-contrib-image-tools, node-red-contrib-image-output, and node-red-node-base64
- Import the Chat flow and the Picture (Image) display flow. On the Chat flow, you will need to edit the template node line 35 to use wss:// (on IBM Cloud) instead of ws:// (on your Laptop)
- On another browser tab, start the https://mynodered.mybluemix.net/chat (Replace mynodered with your IBM Cloud Node Red URL)
- On the Image flow, click on the square box to the right of image preview or viewer to Deactivate and Activate the Node. You will be able to see the picture when you Activate the Node and run samples below
cd ~
git clone https://github.com/thinkahead/microshift.git
cd microshift/raspberry-pi/sensehat
3a. Test directly on the Raspberry Pi 4
pip3 install websocket-client
Update the URL to your node red instance and run the python code to send to Node Red
sed -i "s|mynodered.mybluemix.net|yournodered.mybluemix.net|" *.py
python3 sendtonodered.py
Output
root@raspberrypi:~/microshift/raspberry-pi/sensehat# python3 sendtonodered.py
pygame 2.1.0 (SDL 2.0.16, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html
File 101.jpg uploaded
{"user":"raspberrypi4","message":"1638560972: Temperature: 28.572917938232422 C"}
waiting 5 seconds...
File 101.jpg uploaded
{"user":"raspberrypi4","message":"1638560982: Temperature: 28.625 C"}
waiting 5 seconds...
3b. Use a docker container
Check that we can access the Sense Hat and the camera from a container in docker
docker build -t karve/sensehat .
docker push karve/sensehat
docker run --privileged --name sensehat -ti karve/sensehat bash
# Inside the docker container
python sparkles.py # Tests the Sense Hat's LED matrix
python temperature.py # Get the temperature from Sense Hat’s sensor
python testcam.py # Creates 101.bmp
# Update the URL to your node red instance
sed -i "s|mynodered.mybluemix.net|yournodered.mybluemix.net|" *.py
python sendimages1.py # Ctrl-C to stop
python sendtonodered.py # Ctrl-C to stop
exit
docker rm -f sensehat
3c. Run in MicroShift
We will use the above docker image to send pictures and web socket chat messages to Node Red using a pod in MicroShift.
oc apply -f sensehat.yaml
When we are done, we can delete the deployment
oc delete -f sensehat.yaml
Cleanup MicroShift
We can use the script available on github to cleanup the pods and images. If you already cloned the microshift repo from github, you have the script in the ~/microshift/hack directory.
wget https://raw.githubusercontent.com/thinkahead/microshift/main/hack/cleanup.sh
bash ./cleanup.sh
Containerized MicroShift
We can run MicroShift within containers in two ways:
- MicroShift Containerized – The MicroShift binary runs in a Podman container, CRI-O Systemd service runs directly on the host and data is stored at /var/lib/microshift and /var/lib/kubelet on the host VM.
- MicroShift Containerized All-In-One – The MicroShift binary and CRI-O service run within a Podman container and data is stored in a podman volume, microshift-data. This should be used for “Testing and Development” only.
Microshift Containerized
We can either use the prebuilt image or build the image using docker.
To use the prebuilt image, set
IMAGE=quay.io/microshift/microshift:4.8.0-0.microshift-2021-11-19-115908-linux-arm64
docker pull $IMAGE
To build the image, clone the microshift repository from github and build
git clone https://github.com/thinkahead/microshift.git
cd microshift
# Edit the packaging/images/microshift/Dockerfile. Replace the go-toolset with go-toolset:1.16.7-5
-FROM registry.access.redhat.com/ubi8/go-toolset as builder
+FROM registry.access.redhat.com/ubi8/go-toolset:1.16.7-5 as builder
# This will create the image quay.io/microshift/microshift:4.8.0-0.microshift-unknown-linux-arm64
make build-containerized-cross-build-linux-arm64 -e FROM_SOURCE=true
The Dockerfile uses the registry.access.redhat.com/ubi8/go-toolset:1.16.7-5 as builder to build the microshift binary. Then, it copies the binary to the registry.access.redhat.com/ubi8/ubi-minimal:8.4 that is used for the run stage. Output from the build:
root@raspberrypi:~/microshift# make build-containerized-cross-build-linux-arm64 -e FROM_SOURCE=true
make _build_containerized ARCH=arm64
make[1]: Entering directory '/root/microshift'
echo BIN_TIMESTAMP==2021-12-03T21:32:40Z
BIN_TIMESTAMP==2021-12-03T21:32:40Z
/usr/bin/docker build -t quay.io/microshift/microshift:4.8.0-0.microshift-unknown-linux-arm64 \
-f "/root/microshift"/packaging/images/microshift/Dockerfile \
--build-arg SOURCE_GIT_TAG=4.8.0-0.microshift-unknown \
--build-arg BIN_TIMESTAMP=2021-12-03T21:32:40Z \
--build-arg ARCH=arm64 \
--build-arg MAKE_TARGET="cross-build-linux-arm64" \
--build-arg FROM_SOURCE=true \
--platform="linux/arm64" \
.
Sending build context to Docker daemon 563.2MB
Step 1/17 : FROM registry.access.redhat.com/ubi8/go-toolset:1.16.7-5 as builder
1.16.7-5: Pulling from ubi8/go-toolset
…
go build -mod=vendor -tags 'include_gcs include_oss containers_image_openpgp gssapi providerless netgo osusergo' -ldflags "-X k8s.io/component-base/version.gitMajor=1 -X k8s.io/component-base/version.gitMajor=1 -X k8s.io/component-base/version.gitMinor=21 -X k8s.io/component-base/version.gitVersion=v1.21.0 -X k8s.io/component-base/version.gitCommit=c3b9e07a -X k8s.io/component-base/version.gitTreeState=clean -X k8s.io/component-base/version.buildDate=2021-12-03T21:32:40Z -X k8s.io/client-go/pkg/version.gitMajor=1 -X k8s.io/client-go/pkg/version.gitMinor=21 -X k8s.io/client-go/pkg/version.gitVersion=v1.21.1 -X k8s.io/client-go/pkg/version.gitCommit=b09a9ce3 -X k8s.io/client-go/pkg/version.gitTreeState=clean -X k8s.io/client-go/pkg/version.buildDate=2021-12-03T21:32:40Z -X github.com/openshift/microshift/pkg/version.versionFromGit=4.8.0-0.microshift-unknown -X github.com/openshift/microshift/pkg/version.commitFromGit=c5ad62e0 -X github.com/openshift/microshift/pkg/version.gitTreeState=dirty -X github.com/openshift/microshift/pkg/version.buildDate=2021-12-03T21:32:40Z -s -w" -o '_output/bin/linux_arm64/microshift' github.com/openshift/microshift/cmd/microshift
make[1]: Leaving directory '/opt/app-root/src/github.com/redhat-et/microshift'
Removing intermediate container 93ff567ab4e7
---> 2b2897b0aa96
Step 12/17 : FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4
8.4: Pulling from ubi8/ubi-minimal
…
Step 15/17 : COPY --from=builder /opt/app-root/src/github.com/redhat-et/microshift/_output/bin/linux_$ARCH/microshift /usr/bin/microshift
…
Successfully built 6dafad18ab8e
Successfully tagged quay.io/microshift/microshift:4.8.0-0.microshift-unknown-linux-arm64
make[1]: Leaving directory '/root/microshift'
Set the IMAGE to the one we just built above
IMAGE=quay.io/microshift/microshift:4.8.0-0.microshift-unknown-linux-arm64
Run the microshift container
docker run --rm --ipc=host --network=host --privileged -d --name microshift -v /var/run:/var/run -v /sys:/sys:ro -v /var/lib:/var/lib:rw,rshared -v /lib/modules:/lib/modules -v /etc:/etc -v /run/containers:/run/containers -v /var/log:/var/log -e KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig $IMAGE
export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
watch "docker ps;kubectl get nodes;kubectl get pods -A;crictl images"
The output shows the microshift container running within docker and the rest of the pods in crio on the host
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
a0a2b53e07ae quay.io/microshift/microshift:4.8.0-0.microshift-unknown-linux-arm64 "/usr/bin/microshift…" 5 minutes ago Up 5 minutes
microshift
kubectl get nodes
NAME STATUS ROLES AGE VERSION
raspberrypi.example.com Ready <none> 3m43s v1.21.0
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-bq98q 1/1 Running 0 3m40s
kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-frq2q 1/1 Running 0 3m43s
openshift-dns dns-default-sg7m2 2/2 Running 0 3m41s
openshift-dns node-resolver-h6lqj 1/1 Running 0 3m41s
openshift-ingress router-default-85bcfdd948-c96f9 1/1 Running 0 3m42s
openshift-service-ca service-ca-76674bfb58-bqwrz 1/1 Running 0 3m44s
crictl images
IMAGE TAG IMAGE ID SIZE
k8s.gcr.io/pause 3.2 2a060e2e7101d 489kB
quay.io/microshift/cli 4.8.0-0.okd-2021-10-10-030117 33a276ba2a973 205MB
quay.io/microshift/coredns 4.8.0-0.okd-2021-10-10-030117 67a95c8f15902 265MB
quay.io/microshift/flannel-cni 4.8.0-0.okd-2021-10-10-030117 0e66d6f50c694 8.78MB
quay.io/microshift/flannel 4.8.0-0.okd-2021-10-10-030117 85fc911ceba5a 68.1MB
quay.io/microshift/haproxy-router 4.8.0-0.okd-2021-10-10-030117 37292c44812e7 225MB
quay.io/microshift/hostpath-provisioner 4.8.0-0.okd-2021-10-10-030117 fdef3dc1264ad 39.3MB
quay.io/microshift/kube-rbac-proxy 4.8.0-0.okd-2021-10-10-030117 7f149e453e908 41.5MB
quay.io/microshift/service-ca-operator 4.8.0-0.okd-2021-10-10-030117 0d3ab44356260 276MB
The microshift process is running within the microshift container
root@raspberrypi:~/microshift# docker top microshift -o pid,cmd
PID CMD
74205 /usr/bin/microshift run
The rest of the containers run within cri-o on the host:
root@raspberrypi:~/microshift# crictl pods
POD ID CREATED STATE NAME NAMESPACE
f12e4a9d51083 3 minutes ago Ready dns-default-sg7m2 openshift-dns
4b02b06f2e870 3 minutes ago Ready router-default-85bcfdd948-c96f9 openshift-ingress
c9555515314ef 4 minutes ago Ready kube-flannel-ds-bq98q kube-system
f6146be11026f 4 minutes ago Ready node-resolver-h6lqj openshift-dns
381a7c185ccd4 4 minutes ago Ready kubevirt-hostpath-provisioner-frq2q kubevirt-hostpath-provisioner
b6cc1733c42d8 4 minutes ago Ready service-ca-76674bfb58-bqwrz openshift-service-ca
Now, we can run the samples shown in previous section.
After we are done, we can delete the microshift container. The --rm we used in the docker run will delete the container when we stop it.
docker stop microshift
After it is stopped, we can run the cleanup.sh to delete the pods and images from crio.
MicroShift Containerized All-In-One
Let’s stop the crio on the host, we will be creating an all-in-one container in docker that will have crio within the container.
systemctl stop crio
To build the all-in-one image, clone the microshift repository from github and run make
git clone https://github.com/thinkahead/microshift.git
cd microshift
# Edit the packaging/images/microshift-aio/Dockerfile. Replace the go-toolset with go-toolset:1.16.7-5
-FROM registry.access.redhat.com/ubi8/go-toolset as builder
+FROM registry.access.redhat.com/ubi8/go-toolset:1.16.7-5 as builder
# This will create the image quay.io/microshift/microshift-aio:4.8.0-0.microshift-unknown-linux-nft-arm64
make build-containerized-all-in-one-arm64
The Dockerfile uses the registry.access.redhat.com/ubi8/go-toolset:1.16.7-5 as builder to get the microshift binary. Then, it copies the microshift binary, packaging files and downloads kubectl to the registry.access.redhat.com/ubi8/ubi-init:8.4 that is used for the run stage. It finally installs the cri-o and dependencies within the image. Output from the build:
root@raspberrypi:~/microshift# make build-containerized-all-in-one-arm64
make _build_containerized_aio ARCH=arm64
make[1]: Entering directory '/root/microshift'
echo BIN_TIMESTAMP==2021-12-04T13:18:37Z
BIN_TIMESTAMP==2021-12-04T13:18:37Z
/usr/bin/docker build -t quay.io/microshift/microshift-aio:4.8.0-0.microshift-unknown-linux-nft-arm64 \
-f "/root/microshift"/packaging/images/microshift-aio/Dockerfile \
--build-arg SOURCE_GIT_TAG=4.8.0-0.microshift-unknown \
--build-arg BIN_TIMESTAMP=2021-12-04T13:18:37Z \
--build-arg ARCH=arm64 \
--build-arg MAKE_TARGET="cross-build-linux-arm64" \
--build-arg FROM_SOURCE=false \
--build-arg IPTABLES=nft \
--platform="linux/arm64" \
.
Sending build context to Docker daemon 563.3MB
Step 1/29 : ARG IMAGE_NAME=registry.access.redhat.com/ubi8/ubi-init:8.4
Step 2/29 : FROM registry.access.redhat.com/ubi8/go-toolset:1.16.7-5 as builder
…
Step 13/29 : RUN if [ "$FROM_SOURCE" == "true" ]; then make clean $MAKE_TARGET SOURCE_GIT_TAG=$SOURCE_GIT_TAG BIN_TIMESTAMP=$BIN_TIMESTAMP && mv _output/bin/linux_$ARCH/microshift microshift; else export VERSION=$(curl -s https://api.github.com/repos/redhat-et/microshift/releases | grep tag_name | head -n 1 | cut -d '"' -f 4) && curl -LO https://github.com/redhat-et/microshift/releases/download/$VERSION/microshift-linux-$ARCH && mv microshift-linux-$ARCH microshift; fi
…
Step 14/29 : FROM ${IMAGE_NAME}
8.4: Pulling from ubi8/ubi-init
…
Step 17/29 : ENV BUILD_PATH=packaging/images/microshift-aio
---> Running in fa3f914645a6
Removing intermediate container fa3f914645a6
---> 66724c255163
Step 18/29 : COPY --from=builder /opt/app-root/src/github.com/redhat-et/microshift/microshift /usr/local/bin/microshift
---> 865afaa572e7
Step 19/29 : COPY $BUILD_PATH/unit /usr/lib/systemd/system/microshift.service
---> 3e78aa44b735
Step 20/29 : COPY $BUILD_PATH/kubelet-cgroups.conf /etc/systemd/system.conf.d/kubelet-cgroups.conf
---> 3622d488a598
Step 21/29 : COPY $BUILD_PATH/crio-bridge.conf /etc/cni/net.d/100-crio-bridge.conf
---> 33e9a5457057
Step 22/29 : RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/$ARCH/kubectl" && chmod +x ./kubectl && mv ./kubectl /usr/local/bin/kubectl
---> Running in e0eb7c669c2e
…
Step 25/29 : RUN dnf install -y cri-o cri-tools iproute procps-ng && dnf clean all
---> Running in 5598191b9295
…
Successfully built c0850658b7b9
Successfully tagged quay.io/microshift/microshift-aio:4.8.0-0.microshift-unknown-linux-nft-arm64
make[1]: Leaving directory '/root/microshift'
Create a new docker volume microshift-data and run the microshift container
docker volume rm microshift-data;docker volume create microshift-data
docker run -d --rm --name microshift -h microshift.example.com --privileged -v /lib/modules:/lib/modules -v microshift-data:/var/lib -p 6443:6443 quay.io/microshift/microshift-aio:4.8.0-0.microshift-unknown-linux-nft-arm64
# You may also want to mount the -v /var/hpvolumes:/var/hpvolumes
Now login to the microshift container and see the pods created using crio within the container, not directly on the host. It may take upto 6 minutes for all pods to be started.
root@raspberrypi:~/microshift# docker exec -it microshift bash
[root@microshift /]# export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
[root@microshift /]# watch "kubectl get nodes;kubectl get pods -A;crictl images;crictl pods"
NAME STATUS ROLES AGE VERSION
microshift.example.com Ready
We exit back to the host and check that the microshift container is still running within docker
root@raspberrypi:~/microshift# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ac6cc94bc6d5 quay.io/microshift/microshift-aio:4.8.0-0.microshift-unknown-linux-nft-arm64 "/sbin/init" 38 minutes ago Up 37 minutes 0.0.0.0:6443->6443/tcp, :::6443->6443/tcp microshift
We can inspect the microshift-data volume to find the path
root@raspberrypi:~/microshift# docker volume inspect microshift-data
[
{
"CreatedAt": "2021-12-04T08:29:59-05:00",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/microshift-data/_data",
"Name": "microshift-data",
"Options": {},
"Scope": "local"
}
]
On the host, we set KUBECONFIG to point to the kubeconfig on the data volume export KUBECONFIG=/var/lib/docker/volumes/microshift-data/_data/microshift/resources/kubeadmin/kubeconfig
# crio on host is stopped, so we do not run crictl commands on host
watch "kubectl get nodes;kubectl get pods -A"
Output
NAME STATUS ROLES AGE VERSION
microshift.example.com Ready <none> 42m v1.21.0
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-g65bp 1/1 Running 0 42m
kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-9nlh8 1/1 Running 0 41m
openshift-dns dns-default-s6tzp 2/2 Running 0 31m
openshift-dns node-resolver-7lkb7 1/1 Running 0 42m
openshift-ingress router-default-85bcfdd948-bzfkj 1/1 Running 0 42m
openshift-service-ca service-ca-57cbc45559-6vjm9 1/1 Running 0 31m
The following patch may be required if the dns-default pod in the openshift-dns namespace keeps restarting.
oc patch daemonset/dns-default -n openshift-dns -p '{"spec": {"template": {"spec": {"containers": [{"name": "dns","resources": {"requests": {"cpu": "80m","memory": "90Mi"}}}]}}}}'
You may also need to patch the service-ca deployment if it keeps restarting
oc patch deployments/service-ca -n openshift-service-ca -p '{"spec": {"template": {"spec": {"containers": [{"name": "service-ca-controller","args": ["-v=4"]}]}}}}'
Now, we can run the samples shown previously. Note that the /var/hpvolumes is within the microshift container. You will either need to mount it to the host or create any directories required by the samples within the container. For example, in case of the mysql sample, run “mkdir /var/hpvolumes/mysql” within the container.
After we are done, we can delete the microshift container. The --rm we used in the docker run will delete the container when we stop it.
docker stop microshift
After it is stopped, we can run the cleanup.sh to delete the pods and images from crio.
Errors
E: Unable to locate package cri-o
There is a problem that http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.21/xUbuntu_20.04/arm64/ does not have the crio but http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.21:/1.21.6/xUbuntu_20.04/arm64/ does, so you could replace the contents of /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:1.21.list with 1.21:/1.21.6/xUbuntu_20.04 after the crio:/
Another way is to update the CRIOVERSION=1.22 in the install.sh
Conclusion
In this Part 4, we saw multiple options to build and run MicroShift on the Raspberry Pi 4 with the Raspberry Pi OS (64 bit). We ran samples that used helm, persistent volume for mysql, Sense Hat, and USB camera. We saw a sample that sent the pictures and web socket messages to Node Red on IBM Cloud. In Part 5, we will deploy MicroShift on the Raspberry Pi 4 with CentOS 8 Stream. In Part 6, we will deploy MicroShift on the Raspberry Pi 4 with Ubuntu 20.04. We will look at Virtualization with MicroShift using KubeVirt on the Raspberry Pi 4 in Part 9.
Hope you have enjoyed the article. Share your thoughts in the comments or engage in the conversation with me on Twitter @aakarve. I look forward to hearing about your use of MicroShift on ARM devices and if you would like to see something covered in more detail.
References