Mirror and clean up OCP operator images for CPD upgrade in air-gapped cluster

Introduction

Many IBM Cloud Pak for Data (CPD) customers are deploying OpenShift Container Platform (OCP) and CPD software in an air-gapped environment, means the cluster can not access internet directly. So all the contianer images are located in a local private container registry instead of public image registry, such as registry.redhat.io, quay.io, icr.io, etc.

Then during CPD upgrade, users have to update OCP from lower version into the required version including OCP operators, such as OCS, ODF, OADP operators. Sometimes the OCP upgrade involves multiple version upgrade and users have to mirror mutliple versions of OCP operators into private container registry, then clean them up after OCP upgraded into the target version.

This blog introduce you the steps of mirroring OCP operator images into private container registry before OCP and CPD upgrade, then cleaning them up after the upgrade.

1. Prepare a bastion node and install required tools

Bastion node is a RHEL 8 machine which can access both private container registry and internet, and users install the required tools to download images from public image registry, mirror them into private container registry and delete the images after the upgrade. These tools include:

- OpenShift CLI

Users must install a version of the OpenShift CLI that is compatible with your Red Hat OpenShift Container Platform cluster.

Check out the OC CLI

oc version

- podman

Podman (the POD manager) is an open source tool for developing, managing, and running containers on your Linux® systems. Originally developed by Red Hat® engineers along with the open source community, Podman manages the entire container ecosystem using the libpod library. 

Podman’s daemonless and inclusive architecture makes it a more secure and accessible option for container management, and its accompanying tools and features, such as Buildah and Skopeo, allow developers to customize their container environments to best suit their needs. 

Install podman

yum install -y podman

- opm CLI

The opm CLI tool is provided by the Operator Framework for use with the Operator bundle format. This tool allows you to create and maintain catalogs of Operators from a list of bundles, called an index, that are similar to software repositories. The result is a container image, called an index image, which can be stored in a container registry and then installed on a cluster.

An index contains a database of pointers to Operator manifest content that can be queried through an included API that is served when the container image is run. On OpenShift Container Platform, Operator Lifecycle Manager (OLM) can use the index image as a catalog by referencing it in a CatalogSource object, which polls the image at regular intervals to enable frequent updates to installed Operators on the cluster.

Users can download the opm CLI from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp

Install opm, for example for OCP 4.9

wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest-4.9/opm-linux.tar.gz
tar xvf opm-linux.tar.gz
chmod +x opm

- skopeo

Skopeo is a tool for manipulating, inspecting, signing, and transferring container images and image repositories on Linux® systems, Windows and MacOS. Like Podman and Buildah, Skopeo is an open source community-driven project that does not require running a container daemon (https://github.com/containers/skopeo).

With Skopeo, you can inspect images on a remote registry without having to download the entire image with all its layers, making it a lightweight and modular solution for working with container images across different formats, including Open Container Initiative (OCI) and Docker images.

Install skopeo

yum install -y skopeo

2. Mirror required OCP operators to private container registry

Log into redhat registry

podman login registry.redhat.io

Prune the Red Hat OCP operators index image with the required operators such as ODF and OADP operator, then push the index image into private container registry ${LOCAL_REGISTRY}. This example is mirroring OCP version 4.9 operators.
Note: export your private container registry location into environment variable ${LOCAL_REGISTRY}, such as export LOCAL_REGISTRY=sample-registry.ibm.com:5000

opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.9 \
  -p odf-operator,local-storage-operator,ocs-operator,mcg-operator,redhat-oadp-operator \
  -t ${LOCAL_REGISTRY}/olm-mirror/redhat-operator-index:v4.9

podman push ${LOCAL_REGISTRY}/olm-mirror/redhat-operator-index:v4.9

Mirror operators images to private container registry

oc adm catalog mirror \
  ${LOCAL_REGISTRY}/olm-mirror/redhat-operator-index:v4.9 \
  ${LOCAL_REGISTRY}/olm-mirror \
  -a ${LOCAL_SECRET_JSON} \
  --insecure \
  --index-filter-by-os='linux/amd64'

Note:

• You can run above mirroring command with nohup ... & and monitor nohup.out file, and after mirrorring completed, make sure no error in nohup.out
• Export your Red Hat pull secret into environment variable ${LOCAL_SECRET_JSON}

After mirroring, you can inspect the manifests directory that is generated in your current directory, and it is a folder with this format manifests-redhat-operator-index-<random_number>

cd manifests-redhat-operator-index-<random_number>

The manifests directory contains catalogSource.yaml imageContentSourcePolicy.yaml mapping.txt

Edit imageContentSourcePolicy.yaml to give it a unique name, such as redhat-operator-index-4-9

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  labels:
    operators.openshift.org/catalog: "true"
  name: redhat-operator-index-4-9

Apply the imageContentSourcePolicy.yaml

oc create -f imageContentSourcePolicy.yaml

Take a note of the image url from catalogSource.yaml and you need update catalogsource of operators during upgrade OCP operators

3. Upgrade OCP and CPD

Users can follow the process to upgrade your OCP and CPD to the target versions.

4. Clean up OCP operator images from private container registry

Important: Users have to make sure the OCP operator images to be deleted are no longer required for any of your OCP cluster.

After users mirrored each version of OCP operator images in step 2, there is a manifests directory in the format of manifests-redhat-operator-index-<random_number>

cd manifests-redhat-operator-index-<random_number>

There is a mapping.text file which lists all the mirrored operator images in private container registry

Use the following command to delete all these mirrored images from private container registry

for i in $(awk -F"=" '{print $2}' ./mapping.txt) ; do echo $i ; skopeo delete --creds=<username:password> --tls-verify=false docker://$i ; done

Then make garbage-collect to private container registry to delete the images and release storage space

podman exec <private_container_registry_name> bin/registry garbage-collect --delete-untagged /etc/docker/registry/config.yml

You can use podman ps to show the name of private container registry in the machine of private container registry running over.

End of the blog