Cloud Pak for Data

 View Only

Build a custom image for Watson Studio in Cloud Pak for Data 4

By Harris Yang posted Sun June 11, 2023 05:15 AM

  

This blog describes the detail steps to create a customer image python runtime template for Watson Studio in Cloud Pak for Data v4

Step 1: Preparation

Pull out a terminal on bastion node


export cpd_url=<url of CPD console, eg. https://cpd-cpd-instance1.apps.bj-prod-16.luban.cdl.ibm.com>
export cpd_py_runtime_file=jupyter-py39-server.json
export cust_runtime_file=pyecharts-server.json

export PRIVATE_REGISTRY_LOCATION=<your private container registry, eg. bastion.bj-prod-18.luban.cdl.ibm.com:15000>
export PRIVATE_REGISTRY_PULL_USER=<user_name>
export PRIVATE_REGISTRY_PULL_PASSWORD=<password_of_user>

Limitation: the cust_runtime_file has to be ended as -server.json

Step 2: Download default runtime configuration

Replace the <user_name:password> with your CPD login user name and password to get token

myToken=`curl -k ${cpd_url}/v1/preauth/validateAuth -u <user_name:password> | sed -n -e 's/^.*accessToken":"//p' | cut -d'"' -f1`

curl -X GET "${cpd_url}/zen-data/v1/volumes/files/%2F_global_%2Fconfig%2F.runtime-definitions%2Fibm%2F${cpd_py_runtime_file}" --header "Authorization: Bearer ${myToken}" -k > ${cpd_py_runtime_file}

Step 3: Pull default runtime image


cat ${cpd_py_runtime_file} | grep image

Example of the output:


    "image": "cp.icr.io/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572",
            "volume": "remoteimages",
            "mountPath": "/cc-home/_global_/.remote-images",
            "subPath": "_global_/.remote-images",

Copy out the image address and replace it with your private contianer registry address, eg. bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572


podman login -u ${PRIVATE_REGISTRY_PULL_USER} -p ${PRIVATE_REGISTRY_PULL_PASSWORD} ${PRIVATE_REGISTRY_LOCATION}

podman pull bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572

Step 4: build a custom image

Create a Dockerfile


vi Dockerfile

Content of the Dockerfile, in this example, pyecharts==1.9.1 is the package added, you can add more packages necessarily.


ARG base_image_tag
FROM ${base_image_tag}

# Operating system packages must be installed as root. 
USER root:root

# If you don't need to install additional operating system packages from the 
# default repositories, remove the "microdnf install" command and package list, 
# but keep the rest of this RUN statement. This will install security updates. 
# The chown and chmod commands are needed when system certificates get updated. 

RUN microdnf update \
 && microdnf install --nodocs \
 poppler-utils \
 && microdnf clean all \
 && rm -rf /var/cache/yum \
 && chown -cR :wsbuild \
 /etc/pki/ca-trust/source/anchors/ \
 /etc/pki/ca-trust/extracted/ \
 && chmod -cR g+w \
 /etc/pki/ca-trust/source/anchors/ \
 /etc/pki/ca-trust/extracted/
 
# ========================================= 
# Change display name of the Jupyter kernel 
# ========================================= 
# The kernel spec is in /opt/ibm/run/kernels/<name>/ 
# It must be modified as root. 

RUN sed -i -e '/display_name/{s/",/ with modifications",/}' \
 /opt/ibm/run/kernels/*/kernel.json

# ================================================== 
# Modify the conda environment of the Jupyter kernel 
# ================================================== 
# The name of the kernel conda environment is given by $DSX_KERNEL_CONDENV. 
# The examples assume that you are modifying the conda environment of the 
# base image. If you prefer to create and use a different conda environment 
# from scratch, set DSX_KERNEL_CONDENV to the name of that environment. 
# 
# The conda environment should be changed with user and group "wsbuild". 
# In each layer, call fix-conda-permissions to ensure group write permission, 
# because some files may get installed without respect for the umask. 

USER wsbuild:wsbuild

# If you need to add files to the image, use 
# COPY --chown=wsbuild:wsbuild ...

# ==================================== 
# Install packages with conda or mamba 
# ==================================== 
# If you don't need to install packages with conda, remove this RUN statement. 
# Instead of "conda install", you can also use "mamba install" for a potential speed-up 

RUN umask 002 \
 && conda install -n $DSX_KERNEL_CONDENV \
 py4j \
 && /bin/bash -c 'source /opt/ibm/build/bin/installutils.sh ; fix-conda-permissions'

# ========================= 
# Install packages with pip 
# ========================= 
# If you don't need to install packages with pip, remove this RUN statement. 

RUN umask 002 \
 && conda run -n $DSX_KERNEL_CONDENV pip install \
 astrotools==1.4.2 \
 scikit-multilearn \
 scikit-plot \
 seawater \
 fuzzy \
 pyecharts==1.9.1 \
 && /bin/bash -c 'source /opt/ibm/build/bin/installutils.sh ; fix-conda-permissions'

# ====================================== 
# change from build user to runtime user 
# ====================================== 
USER $DSX_USERNAME:wsbuild

Build the custom image, in this example, custom_image:image_tage is wslocal-cust-runtime-pyecharts:1.9.1


podman build -t wslocal-cust-runtime-pyecharts:1.9.1 --build-arg base_image_tag=bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572 -f ./Dockerfile

Step 5: Push custom image


podman tag wslocal-cust-runtime-pyecharts:1.9.1 bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-cust-runtime-pyecharts:1.9.1

podman push bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-cust-runtime-pyecharts:1.9.1

Step 6: Upload custom runtime configuration


cp ${cpd_py_runtime_file} ${cust_runtime_file}

vi ${cust_runtime_file}

Find image in cust_runtime_file and replace the image address with the custome image address, eg. bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-cust-runtime-pyecharts:1.9.1

Then upload the custom runtime configration


curl -k -X PUT \
"${cpd_url}/zen-data/v1/volumes/files/%2F_global_%2Fconfig%2F.runtime-definitions%2Fibm" \
--header "Authorization: Bearer ${myToken}" \
--header "content-type: multipart/form-data" \
-F upFile=@./${cust_runtime_file}

Step 7: Verify in Watson Studio

Go to a Watson Studio project, Tab Manage, Click Environments, Tab Templates, Click New template, you should be able to see custom image from Software version, and to create a template from it.

Create a test notebook from the custom image runtime template

Promote the custom image runtime template and test notebook to deployment space, and create a notebook job against custom image runtime template to run

You may refer CPD KC on this topic: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=environments-building-custom-images


#Spotlight

0 comments
748 views

Permalink