Programming Languages on Power

 View Only

Cross and source build .NET 8 on Ubuntu for IBM Power

By Ashwini Kadam posted Fri July 12, 2024 08:48 AM

  

Co-authored by Swapnali Pawar

Have you ever attempted to generate a .NET 8 tarball directly from the source code? Wondering why you cannot compile it on IBM Power to create the SDK? The reason lies in the compiler code itself, written in C#. To overcome this, you must first cross-build all required repositories on an x86 machine to generate the .NET SDK tarball for Power architecture. This tarball can then be installed on a Power virtual machine (VM), enabling you to leverage .NET for development. .NET 8 boasts numerous enhancements in performance, stability, security, platforms, and tools, ultimately leading to increased developer productivity and innovation velocity.

Let's delve into a step-by-step guide on cross-building .NET 8 on x86 Ubuntu for IBM Power, followed by source building on the Power VM.

Prerequisites

  • x86 Ubuntu VM: This VM will be used to create the cross build tarball.
  • Docker installed on the x86 Ubuntu VM: The scripts require Docker to run.

Refer to the Install Docker Engine on Ubuntu documentation for the Docker installation steps.

Steps

1. Cross-build .NET repositories to generate tarball for IBM Power

  1. Clone the dotnet-s390x repository: This repository contains scripts for building the .NET SDK inside a Docker container. Clone the repository using the following command. 
    git clone --recurse-submodules  https://github.com/IBM/dotnet-s390x.git
git checkout c6b6d47ab2b3ed4c165b7299526fb0229b47f624

    Note: This repository was originally created for building s390x and then extended to support building ppc64le (Power architecture).

  2. Navigate to the dotnet-s390x folder: Navigate to the -s390x directory using the following command. 
    cd dotnet-s390x
  3. Export the architecture for Power: Create a target SDKs for Power on the cross x86 architecture using the following command. 
    export ARCH=ppc64le
  4. Clean up stale files: Remove any stale and outdated files using the following command. 
    ./docker/run ./dotnet-cleanup
  5. Checkout source and create local tag: Check out the corresponding source code in the repository and create a local tag for the version you want to cross-build using the following command. 
    ./docker/run ./dotnet-bump "<installer_version>"

    Example:

    ./docker/run ./dotnet-bump "8.0.100"

    Note: Refer to Microsoft .Net page for valid <installer_version> values.

  6. Prepare repositories and patches: Clone the required .NET repositories and apply a few non-release patches using the following command. 
    ./docker/run ./dotnet-prepare

    Note: Ensure all repositories and patches are in place before proceeding.

  7. Build the tarball: Finally, build the tarball using the following command. 
    ./docker/run ./dotnet-build

    Upon successful build completion, you can find a file named dotnet-sdk-8.0.100-linux-ppc64le.tar.gz along with runtime-specific packages in the output folder, as follows:

    dotnet-runtime-8.0.0-linux-ppc64le.tar.gz                   
runtime.linux-ppc64le.Microsoft.NETCore.DotNetHost.8.0.0.nupkg
dotnet-runtime-symbols-linux-ppc64le-8.0.0.tar.gz           
runtime.linux-ppc64le.Microsoft.NETCore.DotNetHostPolicy.8.0.0.nupkg
dotnet-sdk-8.0.100-linux-ppc64le.tar.gz                     
runtime.linux-ppc64le.Microsoft.NETCore.DotNetHostResolver.8.0.0.nupkg
Microsoft.AspNetCore.App.Runtime.linux-ppc64le.8.0.0.nupkg  
runtime.linux-ppc64le.Microsoft.NETCore.ILAsm.8.0.0.nupkg
Microsoft.NETCore.App.Host.linux-ppc64le.8.0.0.nupkg        
runtime.linux-ppc64le.Microsoft.NETCore.ILDAsm.8.0.0.nupkg
Microsoft.NETCore.App.Runtime.linux-ppc64le.8.0.0.nupkg
  8. Copy the output folder: Copy the entire output folder to the Power VM where you intend to install the .NET SDK.

2. Install tarball on Power VM with Ubuntu 18.04

  1. Navigate to the output directory: On the Power VM, navigate to the directory where you copied the tarball generated on the x86 machine.
  2. Create a sub folder and extract tarball: Create a sub-folder and extract tarball using the following command. 
    DOTNET_FILE=dotnet-sdk-8.0.100-linux-ppc64le.tar.gz
mkdir -p .dotnet 
export DOTNET_ROOT=$(pwd)/.dotnet
tar zxf "$DOTNET_FILE" -C "$DOTNET_ROOT"
  3. Add .NET root folder: Add .NET root folder in path using the following command. 
    export PATH=$PATH:$DOTNET_ROOT
  4. Verify installation: Verify the successful installation of the .NET SDK using the following command. 
    dotnet --info

    This command should display information about the installed .NET SDK version, including details about the .NET runtime and workloads.

    Upon successful completion, you should see the output displayed as follows:

    # dotnet --info
.NET SDK:
 Version:           8.0.100
 Commit:            57efcf1350
 Workload version:  8.0.100-manifests.e15fffd0

Runtime Environment:
 OS Name:     rhel
 OS Version:  9
 OS Platform: Linux
 RID:         linux-ppc64le
 Base Path:   /root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/.dotnet/sdk/8.0.100/

.NET workloads installed:
 Workload version: 8.0.100-manifests.e15fffd0
There are no installed workloads to display.

Host:
  Version:      8.0.0
  Architecture: ppc64le
  Commit:       b3daf0fa09

.NET SDKs installed:
  8.0.100 [/root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/.dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.0 [/root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/.dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.0 [/root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/.dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_ROOT       [/root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/.dotnet]

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

3. Run Hello World program on Power VM with Ubuntu version 18.04

  1. Generate the console application: Create a new directory named HelloWorld and navigate to the HelloWorld directory. Generate the console application using the following command. 
    mkdir HelloWorld
cd HelloWorld
dotnet new console

    Upon successful completion, you should see the output displayed as follows:

    $ mkdir HelloWorld
$ cd HelloWorld
$/HelloWorld$ dotnet new console

Welcome to .NET 8.0!
---------------------
SDK Version: 8.0.100

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate, view the instructions: https://aka.ms/dotnet-https-linux

----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.

Processing post-creation actions...
Restoring /root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/HelloWorld/HelloWorld.csproj:
  Determining projects to restore...
  Restored /root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/HelloWorld/HelloWorld.csproj (in 1.2 sec).
  2. Build and run the application: Build the application using dotnet build command and then run the application using dotnet run command. 
    dotnet build
dotnet run

    Upon successful completion, you should see the output displayed as follows:

    ~/HelloWorld$ dotnet build
MSBuild version 17.8.3+195e7f5a3 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  HelloWorld -> /root/dotnet-sdk-upstream-8.0.100-ga.ppc64le/HelloWorld/bin/Debug/net8.0/HelloWorld.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:08.93

~/HelloWorld$ dotnet run
Hello, World!

    Note: These scripts have been tested on Ubuntu version 18.04, but they should also function correctly on Ubuntu version 20.04.

    The next step focuses on building the source code using the cross-build tarball generated earlier.

4. Source build .NET repositories to generate tarball for Power on Ubuntu

  1. Create the first level source build: Copy the following script to the Power VM and set the path of the cross-build SDK to the OUTPUT_DIR variable, as shown in line 15 of the script. 
    #!/bin/bash


# Performs a build and runs tests of https://github.com/dotnet/dotnet
# to detect regressions early

set -euxo pipefail


REPO=https://github.com/dotnet/dotnet
REF=release/8.0.1xx
CONFIGURATION="Release"
HOST_RID_OVERRIDE=""
SDK_VERSION=""
OUTPUT_DIR=/root/dotnet-sdk-upstream-8.0.105-ga.ppc64le

while [ $# -ne 0 ]
do
    name="$1"
    case "$name" in
        # --configuration: build configuration (Release/Debug)
        --configuration)
            shift
            CONFIGURATION="$1"
            ;;
        --ref)
            shift
            REF="$1"
            ;;
        --use-mono-runtime)
            build_arguments+=(--use-mono-runtime)
            ;;
        --sdk_version)
            shift
            SDK_VERSION="$1"
            ;;
        *)
            echo "Unknown argument \`$name\`"
            exit 1
            ;;
    esac
    shift
done

architecture=$(uname -m)

# install dependencies
sudo apt-get install clang cmake findutils git libc-dev hostname krb5-multidev libicu-dev liblttng-ust-dev llvm make libssl-dev python3 zlib1g-dev
sudo update-locale LANG=en_US.UTF-8

# function to rename nupkg
cp_nupkg_as_version()
{
    local file="$1"
    local targetversion="$2"

    local tmpdir=$(mktemp -d)
    unzip "$file" -d "$tmpdir"

    local nuspecfile=$(ls $tmpdir/*.nuspec)

    local id_line=$(grep -m 1 "<id>" "$nuspecfile")
    id_regex="<id>(.*)</id>"
    if [[ "$id_line" =~ $id_regex ]]; then
        local id="${BASH_REMATCH[1]}"
    else
        echo "failed to parse id from: '$id_line'"
        exit 1
    fi

    local version_line=$(grep -m 1 "<version>" "$nuspecfile")
    version_regex="<version>(.*)</version>"
    if [[ "$version_line" =~ $version_regex ]]; then
        local version="${BASH_REMATCH[1]}"
    else
        echo "failed to parse version from: '$version_line'"
        exit 1
    fi

    sed -i "s|$version_line|<version>$targetversion</version>|g" "$nuspecfile"

    local targetfile="$(dirname "$file")/$id.$targetversion.nupkg"
    targetfile=$(echo "$targetfile" | tr '[:upper:]' '[:lower:]')
    pushd "$tmpdir"
    zip -r "$targetfile" .
    popd
}

git clone "$REPO"
cd "$(basename "$REPO" .git)"
git checkout "$REF"
COMMIT=$(git rev-parse HEAD)
echo "$REPO is at $COMMIT"

BUILD_DIR="$(pwd)"
EXIT_CODE=256
BUILD_EXIT_CODE=256

if [[ "$architecture" = s390x ]] || [[ "$architecture" = "ppc64le" ]]; then
    # The dotnet SDK artifact uploaded will be in this form: ${NAME}-${VERSION}-${RELEASE}.${ARCH}.rpm.
    # Construct a ".rpm" like string after parsing the version from "global.json" file.
    mkdir dotnet-sdk-$architecture
    cp -r ${OUTPUT_DIR}/. dotnet-sdk-$architecture/.
    nuget_dir=$(readlink -f "dotnet-sdk-$architecture")
    pushd dotnet-sdk-$architecture

    if [ ! -d .dotnet ]; then
            mkdir .dotnet
            tar xf "dotnet-sdk-"*"${architecture}.tar.gz" -C .dotnet
    fi

    sdk_versions=( .dotnet/sdk/* )
    sdk_version=$(basename "${sdk_versions[0]}")

    popd

    cp -a "dotnet-sdk-${architecture}"/.dotnet .dotnet
    DOTNET_ROOT=$(pwd)/.dotnet
    export DOTNET_ROOT

    # replace version but only when it appears on the line after the "sdk" key
    # See https://stackoverflow.com/questions/18620153
    sed -i -E '/"sdk": \{/!b;n;s/"version": "[^"]+"/"version": "'"$sdk_version"'"/' global.json
    # replace dotnet but only when it appears on the line after the "tools" key
    sed -i -E '/"tools": \{/!b;n;s/"dotnet": "[^"]+"/"dotnet": "'"$sdk_version"'"/' global.json

    # Remove installation of runtimes, these will install invalid binaries, or,
    # worse, overwrite our architecture-specific binaries with x86_64 binaries
    sed -i -E '/"runtimes"/,+4d' global.json
    sed -i -zE 's/,\n *\}/\n\}/' global.json

    # Make the pre-built s390x/ppc64le nuget packages available
    find . -iname 'nuget.config' -exec sed -i -zE 's|(<packageSources>.*<clear ?/>)|\1\n<add key="'"$architecture"'" value="'"$nuget_dir"'" />|' {} \;
fi

if [[ "$architecture" = ppc64le ]]; then
    # Disable apphost for ppc64le as do not support .net6
    sed -i -E 's|</OutputType>|</OutputType><UseAppHost>false</UseAppHost>|' src/roslyn-analyzers/src/PerformanceTests/Tests/PerformanceTests.csproj
    sed -i -E 's|</OutputType>|</OutputType><UseAppHost>false</UseAppHost>|' src/vstest/test/Intent/Intent.csproj
fi


#prep
./prep.sh
# allow fetching additional prebuilds while building non-release branches.
if [[ "$REF" != release/* ]]; then
    build_arguments+=(--online)
fi

if [[ "$architecture" == "s390x" ]] || [[ "$architecture" == "ppc64le" ]]; then
    # Update ilasm nupkg version as per PackageVersions.prop
    mkdir -p packages-customized-local
    pushd packages-customized-local
    tar xf ../prereqs/packages/archive/Private.SourceBuilt.Artifacts.*.tar.gz

    ilasm_new_version=$(sed -n '/<MicrosoftNETCoreILAsmVersion>/,/<\/MicrosoftNETCoreILAsmVersion>/ s/.*<MicrosoftNETCoreILAsmVersion>\(.*\)<\/MicrosoftNETCoreILAsmVersion>.*/\1/p' PackageVersions.props | sed -n '1p')
    popd

    ilasm_version=$(ls $nuget_dir | grep -i ilasm | tr 'A-Z' 'a-z' | sed -E 's|runtime.linux-'"$architecture"'.microsoft.netcore.ilasm.||' | sed -E 's|.nupkg$||')
    echo $ilasm_version

    cp_nupkg_as_version "$nuget_dir/runtime.linux-$architecture.Microsoft.NETCore.ILAsm.$ilasm_version.nupkg" "$ilasm_new_version"
    cp_nupkg_as_version "$nuget_dir/runtime.linux-$architecture.Microsoft.NETCore.ILDAsm.$ilasm_version.nupkg" "$ilasm_new_version"

    cp -r "$nuget_dir"/*.nupkg prereqs/packages/prebuilt/

    build_arguments+=(--use-mono-runtime)
fi

BUILD_EXIT_CODE=0
./build.sh ${common_arguments[0]+"${common_arguments[@]}"} ${build_arguments[0]+"${build_arguments[@]}"} || BUILD_EXIT_CODE=$?
EXIT_CODE=$BUILD_EXIT_CODE

exit $EXIT_CODE

    This script is designed for the dotnet/dotnet repository, a virtual monolithic repository which includes all the source code and infrastructure required to build the .NET SDK (dotnet/dotnet). The script manages essential dependencies required for compiling .NET from its source code. Additionally, it prepares the source code for building and then proceeds to build the .NET SDK.

  2. Run the script: Run the script using the following command. 
    ./dotnet-dotnet.sh --ref release/8.0.1xx

    Here –ref parameter is to specify the check out

    For example, ./dotnet-dotnet.sh --ref release/8.0.1xx

    Refer to the version specified in the global.json file to determine the tarball version to be used for the source build.

    Note: If you're building from a specific tag, utilize the cross-built tarball from the previous tag. On the other hand, if you're working with a branch, use the most recent version available.

  3. Locate the built tarball: Locate the resulting SDK at the following path. 
    artifacts/ppc64le/Release/dotnet-sdk-8.0.100-your-RID.tar.gz.
  4. Specify the path: Provide the path of OUTPUT_DIR in the script where you have set up your cross-build untar folder.

Conclusion

That brings us to the end of this blog! We hope you feel more confident now that you have learned how to cross-build .NET 8 on x86 for the IBM Power architecture and perform a source build using the cross-built tarball. With numerous enhancements and improvements, .NET 8 offers a richer experience for developers. By installing the .NET 8 tarball on a Power VM, you can leverage .NET for your development needs. If you follow these steps and successfully build .NET on your Power VM, do tell us about your experience!

Permalink