Open Source Development

 View Only

Create a containerized, cross-architectural, emulated Linux on Power development environment from just about anywhere!

By Linda Alkire posted Tue November 12, 2024 07:58 AM

  

Original authors: Emma Erickson and Paul Clarke

In the Run a full-system Linux on Power environment from Microsoft Windows tutorial, we run a fully-emulated IBM Power system on an x86 system. This is reasonably intuitive, with QEMU doing everything required to pretend to be a Power system. Given that fully emulated Power system, we simply install an operating system and can then do whatever we can do on a real Power system (with emulation necessarily offering much less performance).

However, there are costs for full virtualization, including most significantly:

  • Additional storage required for the firmware and full operating system
  • Additional memory required for the firmware, kernel, and all tasks and services

All of the firmware, kernel, and basic operating system components that are necessarily present for the emulated system are on top of the real firmware, kernel, and basic operating system components. Running a program generally just requires the program and supporting libraries, and the kernel. We don’t need two operating systems, or two kernels.

Linux Containers allow a program and its dependent libraries to run on any like-architecture Linux-based operating system and kernel. The interface from the program and libraries to the kernel is mapped as needed to the native kernel. This avoids most of the redundancy of running a fully emulated virtual system just to run the program.

If we inject emulation of just the program and its dependencies, and carefully map the kernel interface, we can run a very space-efficient instance of a foreign program on a native operating system.

As a bonus, an application running emulated in a container may run a bit faster than in an emulated full system, as any kernel operations will run natively (non-emulated).

This tutorial explains how to utilize Linux containers to establish a cross-architectural development environment such that it is convenient to port, test, and performance-tune an application in a Linux on Power environment using only an x86 system running Microsoft Windows (with Windows Subsystem for Linux) or a Linux-based operating system.

Prerequisites

To complete this tutorial, make sure that the following prerequisites are fulfilled:

  • A Linux-based system with internet connectivity, command line and superuser access.
  • Or, a Microsoft Windows Subsystem for Linux (WSL) environment with internet connectivity, command line and superuser access.

Some familiarity with Linux command line usage is helpful.

Estimated time

Assuming that you have an internet connection with good speed and your command line proficiency is good, it takes around 30 minutes to complete the steps in this tutorial.

Steps

This section explains how to install the required user-mode (in contrast to full system) emulation components, install the container management software, build and run the foreign (emulated) container, and test it.

Install Windows Subsystem for Linux

The Run a full-system Linux on Power environment from Microsoft Windows tutorial covered installation and configuration of WSL for the purpose of using QEMU full-system emulation. The same steps are needed here, so refer to that tutorial if needed.

Once WSL is ready, it acts just like a native Linux system. The rest of this tutorial applies to both a native Linux-based system as well as WSL.

Install QEMU programs

Running a cross-architectural container requires statically built QEMU user-mode programs. We can install them directly if they are available in the Linux distribution, or we can build them from source. Building from source takes a bit more effort, but may offer later versions of QEMU than what is available in repositories. Later versions of QEMU may have better support for later Power processor generations

Red Hat

Red Hat does not make available foreign architecture QEMU packages. There are at least two choices for installing statically built QEMU user-mode programs: install from equivalent Fedora repositories or build from source.

Installing from Fedora repositories

It is possible to define the Fedora repositories on a Red Hat Enterprise Linux system, and use the normal package management tools to install the qemu-user-static package. The user is cautioned that there could be any number of package conflicts between Fedora and Red Hat, and enabling the Fedora repositories on a Red Hat system is unlikely to be a supported configuration. It may be safe to install only what is needed, then uninstall what is not needed. This is what is described below:

  1. Download and install the Fedora GPG keys for the appropriate release. For Fedora 36:
    $ /usr/bin/sudo dnf --repofrompath=fedora,https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/36/Everything/x86_64/os/ --repo=fedora --nogpgcheck install fedora-gpg-keys
  1. Import the Fedora GPG keys:
    $ /usr/bin/sudo rpmkeys --import /etc/pki/rpm-gpg/RPM-GPG-KEY-36-fedora
  1. Install the statically built QEMU user packages:
    $ /usr/bin/sudo dnf --repofrompath=fedora,https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/36/Everything/x86_64/os/ --repo=fedora install qemu-user-static
  1. Optionally, leave the system in a state closest to how we started:
    • Remove the Fedora GPG keys from the RPM keyring:
      $ for key in $(rpm -qa | grep gpg-pubkey); do
if [[ "$(rpm -q --queryformat="%{SUMMARY}\n" $key)" =~ Fedora.*' public key' ]]; then
/usr/bin/sudo rpm -e $key
fi
done
    • Remove the Fedora GPG keys from the system:
      $ /usr/bin/sudo dnf remove fedora-gpg-keys
  1. Skip ahead to the Install podman section.

Installing from QEMU source

Download QEMU source and prepare to build

  1. Visit the Download QEMU web page and refer to the instructions for downloading the latest release or pulling the latest development version. For the example in this tutorial, we proceed with release 7.0.0.
    $ curl -O https://download.qemu.org/qemu-7.0.0.tar.xz
  1. Extract the contents of the downloaded package.
    $ tar -xf qemu-7.0.0.tar.xz
  1. Make a new directory in which you need to build, and make that the current working directory.
    $ mkdir qemu-build
    $ cd qemu-build
  1. Install the pre-requisites.
    $ /usr/bin/sudo dnf install make ninja-build gcc perl glibc-static pcre-static zlib-static glib2-static

Note that there are many features of QEMU which require additional prerequisites, but the above will provide a working result for our basic usage.

Build and install qemu user-mode ppc64le statically

  1. Build a static QEMU user-mode program.

$ ../qemu-7.0.0/configure --target-list=ppc64le-linux-user --static

Note that there are many, many interesting configuration options for QEMU, but the above will provide a working result for our basic usage.

$ make -j

The newly built program file is qemu-ppc64le, in the current directory.

  1. Install the program to /usr/local, and set a working SELinux context.

$ /usr/bin/sudo make install

$ /usr/bin/sudo chcon --reference /bin/ls /usr/local/bin/qemu-ppc64le

  1. Enable automatic execution of foreign architecture programs.

$ /usr/bin/sudo scripts/qemu-binfmt-conf.sh --persistent yes

Note that this may generate a lot of errors for other architectures, all of which can be ignored.

  1. Skip ahead to the Install podman

SUSE

SUSE does not include a package with QEMU user-mode static programs. There are at least two choices: install from equivalent openSUSE repositories, or build from QEMU source.

Both choices require enabling the openSUSE repositories. As per openSUSE.org, “Leap uses source from SUSE Linux Enterprise”, which leads to the impression that openSUSE and SUSE will be fairly compatible, but caution is still warranted.

Enable the openSUSE Leap repositories

Add the matching openSUSE Leap repository. For example, the openSUSE Leap 15.4 repository is http://download.opensuse.org/distribution/leap/15.4/repo/oss/.

  1. As a superuser, invoke yast.
    $ /usr/bin/sudo yast
  2. Select Software, Software Repositories and then Run.
  3. Select Add and then select OK.
  4. Select Specify URL and then select Next.
  5. Enter a meaningful name for the repository and enter the openSUSE Leap 15.4 repository URL (given earlier at the beginning of the procedure) and select Next.
  6. Wait for repository metadata to download.
  7. Read and choose whether to accept the license agreement. If you choose to accept, navigate to the I Agree to License Terms option and press the Space bar to select it, and then select Next.
  8. Then select OK and then Quit.

    Install QEMU user-mode (static) package from openSUSE Leap

    Run the following command to install QEMU user-mode package from openSUSE Leap:

    $ /usr/bin/sudo zypper install qemu-linux-user

    Now, skip ahead to the Enable automatic execution of foreign architecture programs section.

    Build and install QEMU user-mode from source

    1. Download QEMU source. Visit https://qemu.org/download. There you can find instructions for downloading the latest release, or pulling the latest development version. For this example, we proceed with release 7.0.0.

    $ curl -O https://download.qemu.org/qemu-7.0.0.tar.xz2.

    1. Extract the contents of the downloaded package.

    $ tar -xf qemu-7.0.0.tar.xz

    1. Make a new directory in which to build, and make it the current working directory.

    $ mkdir qemu-build

    $ cd qemu-build

    1. Install the prerequisites.

    $ /usr/bin/sudo zypper install glibc-devel-static pcre-devel-static zlib-devel-static glib2-devel-static

    1. Build a static QEMU user-mode program.

    $ ../qemu-7.0.0/configure --target-list=ppc64le-linux-user --static

    $ make -j

    The resulting program is qemu-ppc64le in the current directory.

    1. Install the resulting program in /usr/local.

    $ /usr/bin/sudo make install

    Enable automatic execution of foreign architecture programs

    If QEMU was installed from a package (not built from source), the qemu-binfmt-conf.sh script is in /usr/sbin. If QEMU was built from source, the script is in $HOME/qemu-7.0.0/scripts.

    Use the appropriate path in the following command:

    $ /usr/bin/sudo /usr/sbin/qemu-binfmt-conf.sh --persistent yes

    Note that this may generate a lot of errors for other architectures, all of which can be ignored.

    Ubuntu

    Install the qemu-user-static package:

    $ /usr/bin/sudo apt install qemu-user-static

    Install podman

    Install the podman package, which contains the primary tool, podman, for interacting with Linux containers.

    Red Hat

    $ /usr/bin/sudo dnf install podman

    SUSE

    $ /usr/bin/sudo zypper install podman

    Ubuntu

    $ /usr/bin/sudo apt install podman

    Download the container image

    Here, we download the container images for the respective Linux distributions. Note that it is certainly possible to run a container image for a Linux distribution which is different than the host Linux operating system. For example, you can run a Red Hat container on an Ubuntu operating system. Thus, the following commands can be run on any Linux-based operating system.

    Red Hat

    $ podman pull --arch=ppc64le registry.access.redhat.com/ubi9/ubi-init

    SUSE

    $ podman pull --arch=ppc64le registry.suse.com/bci/bci-base

    Ubuntu

    $ podman pull --arch=ppc64le ubuntu

    The long container image ID is printed last. The shorter ID can be found and used instead (example):

    $ podman images

    REPOSITORY                                TAG        IMAGE ID      CREATED       SIZE

    registry.access.redhat.com/ubi9/ubi-init latest     2b7a13eabe02   8 days ago   287 MB

    registry.suse.com/bci/bci-base           latest     651ff1cda340   2 weeks ago  160 MB

    docker.io/library/ubuntu                 latest     b4cdd8bc1823   4 weeks ago  121 MB

    Create and run the container

    Run the following command to create and run the container:

    $ podman run --interactive --tty <container image ID> /bin/bash

    [root@a1ad7b564e6f /]#

    You can easily select the model of CPU here by using the --env parameter to pass in an appropriate value for the QEMU_CPU environment variable, for example:

    $ podman run --interactive --tty --env QEMU_CPU=power10 <container image ID> /bin/bash

    [root@a1ad7b564e6f /]#

    A handy suggestion is to set the name and the prompt to match the emulated CPU model:

    $ podman run --interactive --tty --env QEMU=power10 --env PS1='power10# ' --name power10 <container image ID> /bin/bash
power10# <ctrl>p<ctrl>q
$ podman container list --all
CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES
2c6648a4cd84 .../ubi9/ubi-init /bin/bash 1 minute ago Up 1 minute ago power10
$ podman attach power10
power10#

    Validate the container

    Run the following command to validate the emulated architecture of the running container.

    # uname -a

    Note that ppc64le should be included in the output.

    [root@a1ad7b564e6f /]# uname -a
Linux a1ad7b564e6f 5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr 14 12:42:38 EDT 2022 ppc64le ppc64le ppc64le GNU/Linux
[root@a1ad7b564e6f /]#

    Further, you can determine the model of the CPU being emulated by examining the auxiliary vector using the following command:

    [root@a1ad7b564e6f /]# LD_SHOW_AUXV=1 /bin/true
AT_DCACHEBSIZE:       0x80
AT_ICACHEBSIZE:       0x80
AT_UCACHEBSIZE:       0x0
AT_PHDR:              0x4000000040
AT_PHENT:             56
AT_PHNUM:             11
AT_PAGESZ:            4096
AT_BASE:              0x4002842000
AT_FLAGS:             0x0
AT_ENTRY:             0x400001084c
AT_UID:               0
AT_EUID:              0
AT_GID:               0
AT_EGID:              0
AT_HWCAP:             vsx arch_2_06 dfp fpu altivec ppc64
AT_CLKTCK:            100
AT_RANDOM:            0x4002841f00
AT_SECURE:            0
AT_EXECFN:            /bin/true
AT_HWCAP2:            mma arch_3_1 darn ieee128 arch_3_00 vcrypto tar isel arch_2_07
[root@a1ad7b564e6f /]#

    Look at just the two HWCAP lines, and specifically the arch fields. In the above example, the largest value of an arch field is arch_3_1, which corresponds to Power ISA 3.1 or Power10.

    Fly! Be free!

    At this point, the container is ready for development and it is recommended to perform the following tasks:

    1. Configure repositories for IBM value-add software.
    2. Install the IBM Advance Toolchain.
    3. Install the IBM XL C/C++ compiler.
    4. Install other useful tools, like the IBM Power Performance Simulator and pipestat.

    For your projects:

    1. Download all prerequisites and the source code for the software to port or test.
    2. Build your project’s program(s).
    3. Run it!

    Detach from the container

    Use the Escape key sequence Ctrl+P, Ctrl+Q.

    Attach to the container

    Run the following command to attach to a container:

    $ podman container attach <container ID>

    Exit from the container

    Use the exit command to exit a container.

    # exit

    This puts the container in the exited state. To reattach, the container must first be restarted. See below.

    Restart a container

    Run the following command to restart a container:

    $ podman container restart <container ID>

    Delete a container

    Be warned that this will permanently erase all of the contents of the container. The container image contents will continue to exist, which are basically the original contents of the container before any modifications.

    $ podman container rm <container ID>

    Troubleshooting

    Refer to the following tips on troubleshooting container-related issues:

    • Container won’t start: The binfmt_misc infrastructure is not set up correctly.
      • The qemu-ppc64le program must be built statically.
      • /proc/sys/fs/binfmt_misc/qemu-ppc64le must exist and its contents (“interpreter”) must point to the statically built qemu-ppc64le program. The contents are defined when running the qemu-binfmt-conf.sh script. See the "Enable automatic execution of foreign architecture programs" section.
    • Attaching to a container fails: The container is in the wrong state (exited) for attachment.
    • Container doesn’t exist: Note the difference between a container image ID and a container ID. The container image ID is used when initially creating a container. Other interactions with a built container use the container ID.

    Summary

    This tutorial covered some basic enablement and usage steps for running emulated Power architecture containers on non-Power environments, like Linux or Windows (through WSL) on x86. These containers can be used as basic development environments to facilitate large-scale porting or small-scale performance analysis and tuning of Linux applications for the Power architecture.

    Take the next step

    Join the Power Developer eXchange Community (PDeX). PDeX is a place for anyone interested in developing open source apps on IBM Power. Whether you're new to Power or a seasoned expert, we invite you to join and begin exchanging ideas, sharing experiences, and collaborating with other members today!

    Permalink

    Comments

    Saif Ali Sabri
    7 days ago

    This is invaluable insight you shared.
    Your clear explanation of the process, coupled with practical examples, provides an excellent roadmap for anyone looking to leverage containerized environments for cross-architectural emulation. The way you broke down the complexities of creating a seamless and portable solution was not only impressive but also incredibly helpful for developers navigating this space.
    This post is a testament to your expertise in both containerization and architectural emulation. It is content like this that strengthens the community and empowers developers to overcome challenges in innovative ways.
     
    The blog is to craft such a detailed and accessible guide. I am certain it will serve as a valuable resource for many professionals and enthusiasts alike.