Overview
In this blog series, we delve into the intricacies of creating multi-arch images that support IBM Power (ppc64le), x86 (amd64), and other hardware architectures. Our primary emphasis in this first blog is on understanding the techniques involved in constructing these versatile images to include ppc64le. In the second blog, we cover automating multi-arch image builds using various Continuous Integration (CI) tools.
By the end of the series, you'll be equipped with the knowledge to efficiently build multi-arch images that cater to different platforms, including IBM Power, while harnessing the agility of automation provided by CI tools.
Why does this matter for IBM Power?
Multi-arch build pipelines can greatly reduce the complexity of supporting multiple operating systems and architectures. Further, multi-arch images are built to run on multiple CPU platforms without the need to explicitly specify which platform it's running on. The build pipeline automatically pulls the correct image at runtime based on the host CPU architecture.
To build a multi-arch container image, you need access to native hardware. If you don’t have access to Power hardware or if adding new hardware to your existing CI setup is too complex, you can use build tools to emulate the target architecture. Keep reading to learn more.
Introduction to build pipelines
Build pipelines are integral components of the software development process, facilitating the creation of software packages and container images for various architectures, including IBM Power. This document explores three distinct types of build pipelines: Native build pipeline, emulation build pipeline, and cross-compilation build pipeline. Each approach has advantages in terms of efficiency, optimization, and cross-platform compatibility and each offers unique strategies for creating multi-arch images that cater to diverse architectures.
Understanding these pipelines empowers you to craft multi-arch images that excel in efficiency, compatibility, and performance across a wide array of platforms.
Native build pipeline
Native build pipelines optimize performance by building directly on target architectures, making it ideal for users who prioritize raw speed and efficiency.
A native build pipeline involves constructing software or container images directly on a designated target architecture, eliminating the need for emulation or cross-compilation. This approach capitalizes on the native tools and resources available on the target platform, resulting in optimized binaries or images tailored to the architecture's specific capabilities and performance traits, ensuring superior execution efficiency. The following architectural diagram depicts the workflow of the native build pipeline.
In this workflow, images are built individually on specific target architectures, such as ppc64le. These images are then pushed to a container image registry, each tagged with a unique tag corresponding to the target architecture. Subsequently, a fat manifest is generated, serving as a high-level reference that associates the different target architecture images. The fat manifest is also pushed to the registry, allowing users to conveniently pull the multi-arch image using a single tag.
Emulation build pipeline
Emulation build pipelines simulate target behavior and is a versatile choice for users who value comprehensive testing and compatibility assurance across various environments.
An emulation build pipeline uses emulation software or virtualization techniques to simulate the behavior of the target architecture. Emulation build pipelines allow you to create a build environment that mimics the target architecture, i.e., ppc64le, even when working on a different architecture, i.e., amd64. This approach allows you to utilize the tools, libraries, and configurations required for the target architecture. The following architectural diagram depicts the workflow of the emulation build pipeline.
In this example, the entire build process occurs on a native amd64 architecture. Target architecture images are built using the QEMU emulator, which mimics the respective target architecture. These architecture-specific images are merged into a single multi-arch image, which is then pushed to the container image registry. Users can access the multi-arch image using a single tag to benefit from the comprehensive simulation offered by the emulation build pipeline.
Cross-compilation build pipeline
Cross-compilation build pipelines leverage cross compilation to produce versatile images, making it the preferred choice for those seeking broad compatibility and scalability across multiple architectures or devices.
This approach involves two stages: The build stage and the runtime stage. During the build stage, code is compiled on the native architecture, covering common instruction layers across various images, ensuring a consistent foundation. The runtime stage utilizes the QEMU emulator to generate target architecture-specific images with architecture-specific instruction layers. The following architectural diagram depicts the workflow of the cross-compilation build pipeline.
Images produced in the runtime stage are architecture-specific and can be run on their respective target architectures.
Container image registry
The final step for each type of build pipeline involves pushing the multi-arch image to your container image registry. This enables you to easily retrieve the multi-arch image using a single tag. In this blog, I use quay.io, but you can use whatever image registry you choose and achieve the same results.
Methods for building multi-arch images
There are multiple methods for building multi-arch images, but for the purposes of this blog, I’ll cover two of the most popular: Using Dockerfile and Using Buildx.
Building multi-arch images using Dockerfile
This section provides step-by-step guidance for building, verifying, and pushing multi-arch images using Dockerfile. A Dockerfile is a text file that contains all the commands needed to build a given image.
Prerequisites
Following are the prerequisites for building multi-arch images using a Dockerfile:
- Docker Engine 20.10 or later.
- Valid login credentials for your image registry. Quay.io is referenced in this blog.
Step 1: Dockerfile setup
Use the following Dockerfile as a foundation for constructing your multi-arch image:
# Dockerfile
# Use the official golang base image
FROM golang:latest AS build
# Set the working directory inside the container
WORKDIR /app
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# Copy the source code into the container
COPY main.go main.go
# Build the Go application
RUN CGO_ENABLED=0 go build -o multiarch-app .
# Set the command to run the executable
CMD ["/app/multiarch-app"]
Step 2: Build the container image for ppc64le and other platforms
Perform the following steps to build, verify, and push your ppc64le container image. Note: the steps are the same for each architecture; simply replace the ppc64le tag with the appropriate architecture tag (i.e., amd64, arm64, etc.).
- Log into the container image registry on the target platform by using the following command.
docker login quay.io -u <REGISTRY_USERNAME> -p <REGISTRY_PASSWORD>
- Build the image.
# docker build -t quay.io/pparab/multiarch:latest-ppc64le
- Verify the image.
# docker inspect quay.io/pparab/multiarch:latest-ppc64le | grep Arch
"Architecture": "ppc64le",
# docker run -t quay.io/pparab/multiarch:latest-ppc64le
Process started.
Running on linux/ppc64le
Process completed.
Welcome to the Multi-Arch World
- Push the image.
# docker push quay.io/pparab/multiarch:latest-ppc64le
The push refers to repository [quay.io/pparab/multiarch]
6074a711fea8: Pushed
54146204a4c3: Pushed
f120bb2df836: Pushed
c69e403eb8ce: Pushed
ede11bda2308: Pushed
9ff37309962c: Pushed
f14dbdd36186: Pushed
b84b2150c255: Pushed
c90391dcee86: Pushed
61587a363581: Pushed
0023d86a8fb0: Pushed
latest-ppc64le: digest: sha256:8901a52a6e0321b5d9a8f0f21b6cbd2c59097ef0c9aec9b8fc939e130e5fc872 size: 2624
Step 3: Create, verify, and push the multi-arch manifest
This process ensures a streamlined consolidation of architecture-specific images, verifies the accuracy of the resulting multi-arch manifest, and facilitates its seamless integration into the container image registry.
- Log into your container image registry (amd64 was used for the purposes of this blog) by using the following command:
docker login quay.io -u <REGISTRY_USERNAME> -p <REGISTRY_PASSWORD>
- Create the manifest.
# docker manifest create --amend quay.io/pparab/multiarch:latest \
quay.io/pparab/multiarch:latest-amd64 quay.io/pparab/multiarch:latest-arm64 quay.io/pparab/multiarch:latest-ppc64le
Created manifest list quay.io/pparab/multiarch:latest
- Verify the manifest.
# docker manifest inspect quay.io/pparab/multiarch:latest
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 2625,
"digest": "sha256:f0bdba9fccb92d0c9171504f4e64d4dccbd2a43626246a464ca0c91f7a5976ed",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 2623,
"digest": "sha256:fda2f5aa4afe1645d25bc487a790e141e85158e5d58b2e18b65b57467922faf7",
"platform": {
"architecture": "arm64",
"os": "linux",
"variant": "v8"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 2624,
"digest": "sha256:8901a52a6e0321b5d9a8f0f21b6cbd2c59097ef0c9aec9b8fc939e130e5fc872",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
}
]
}
- Push the manifest.
# docker manifest push quay.io/pparab/multiarch:latest
sha256:9151747d15064b8a03bdea6ad02424efb61cd44060a044a90d12cdc2fe1e8f44
- Validate the successful push operation by navigating to your registry and clicking on Repository Tags. You should observe the distinct layers of each architecture as integral components of the manifest being pushed.
Step 4: Validate the manifest on ppc64le and other platforms
The final step is to validate the manifest in the registry to ensure image integrity and security across the various platforms. I’ve shown the commands for doing this on ppc64le below however, the commands are the same for each architecture; simply replace the ppc64le tag with the appropriate architecture tag (i.e., amd64, arm64, etc.).
# docker pull quay.io/pparab/multiarch
# docker inspect quay.io/pparab/multiarch | grep Arch
"Architecture": "ppc64le",
# docker run -t quay.io/pparab/multiarch
Process started.
Running on linux/ppc64le
Process completed.
Welcome to the Multi-Arch World
You should now have the knowledge to build your own multi-arch images using a Dockerfile. This expertise will enable you to enhance software compatibility across diverse architectures, contributing to a more inclusive and versatile software ecosystem.
Building multi-arch images using Buildx
Buildx is an extension of the Docker CLI plugin that enhances the capabilities of the docker build command to simplify the creation of Docker images for multiple platforms and architectures.
Buildx seamlessly integrates with QEMU emulation to facilitate building and testing multi-arch images on machines lacking native support for those architectures.
For detailed information on this topic, refer to Cross-compile and build ppc64le image on x86 host using Docker Buildx section in the ‘Build multi-arch images on GitHub Actions with Buildx’ documentation on IBM Power Developer eXchange.
Summary
By the end of this blog, readers are equipped with the knowledge of types of multi-arch build pipelines and the process to build multi-arch images using Dockerfile or Buildx. This expertise enhances software compatibility across diverse architectures, contributing to a more inclusive and versatile software ecosystem.
Continue on to the second blog of this series to learn about automating multi-arch image builds. If you have question or comments, please drop them below.
#Featured-area-1#Featured-area-1-home