This blog describes the detail steps to create a customer image python runtime template for Watson Studio in Cloud Pak for Data v4
Step 1: Preparation
Pull out a terminal on bastion node
export cpd_url=<url of CPD console, eg. https://cpd-cpd-instance1.apps.bj-prod-16.luban.cdl.ibm.com>
export cpd_py_runtime_file=jupyter-py39-server.json
export cust_runtime_file=pyecharts-server.json
export PRIVATE_REGISTRY_LOCATION=<your private container registry, eg. bastion.bj-prod-18.luban.cdl.ibm.com:15000>
export PRIVATE_REGISTRY_PULL_USER=<user_name>
export PRIVATE_REGISTRY_PULL_PASSWORD=<password_of_user>
Limitation: the cust_runtime_file has to be ended as -server.json
Step 2: Download default runtime configuration
Replace the <user_name:password>
with your CPD login user name and password to get token
myToken=`curl -k ${cpd_url}/v1/preauth/validateAuth -u <user_name:password> | sed -n -e 's/^.*accessToken":"//p' | cut -d'"' -f1`
curl -X GET "${cpd_url}/zen-data/v1/volumes/files/%2F_global_%2Fconfig%2F.runtime-definitions%2Fibm%2F${cpd_py_runtime_file}" --header "Authorization: Bearer ${myToken}" -k > ${cpd_py_runtime_file}
Step 3: Pull default runtime image
cat ${cpd_py_runtime_file} | grep image
Example of the output:
"image": "cp.icr.io/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572",
"volume": "remoteimages",
"mountPath": "/cc-home/_global_/.remote-images",
"subPath": "_global_/.remote-images",
Copy out the image address and replace it with your private contianer registry address, eg. bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572
podman login -u ${PRIVATE_REGISTRY_PULL_USER} -p ${PRIVATE_REGISTRY_PULL_PASSWORD} ${PRIVATE_REGISTRY_LOCATION}
podman pull bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572
Step 4: build a custom image
Create a Dockerfile
Content of the Dockerfile, in this example, pyecharts==1.9.1
is the package added, you can add more packages necessarily.
ARG base_image_tag
FROM ${base_image_tag}
# Operating system packages must be installed as root.
USER root:root
# If you don't need to install additional operating system packages from the
# default repositories, remove the "microdnf install" command and package list,
# but keep the rest of this RUN statement. This will install security updates.
# The chown and chmod commands are needed when system certificates get updated.
RUN microdnf update \
&& microdnf install --nodocs \
poppler-utils \
&& microdnf clean all \
&& rm -rf /var/cache/yum \
&& chown -cR :wsbuild \
/etc/pki/ca-trust/source/anchors/ \
/etc/pki/ca-trust/extracted/ \
&& chmod -cR g+w \
/etc/pki/ca-trust/source/anchors/ \
/etc/pki/ca-trust/extracted/
# =========================================
# Change display name of the Jupyter kernel
# =========================================
# The kernel spec is in /opt/ibm/run/kernels/<name>/
# It must be modified as root.
RUN sed -i -e '/display_name/{s/",/ with modifications",/}' \
/opt/ibm/run/kernels/*/kernel.json
# ==================================================
# Modify the conda environment of the Jupyter kernel
# ==================================================
# The name of the kernel conda environment is given by $DSX_KERNEL_CONDENV.
# The examples assume that you are modifying the conda environment of the
# base image. If you prefer to create and use a different conda environment
# from scratch, set DSX_KERNEL_CONDENV to the name of that environment.
#
# The conda environment should be changed with user and group "wsbuild".
# In each layer, call fix-conda-permissions to ensure group write permission,
# because some files may get installed without respect for the umask.
USER wsbuild:wsbuild
# If you need to add files to the image, use
# COPY --chown=wsbuild:wsbuild ...
# ====================================
# Install packages with conda or mamba
# ====================================
# If you don't need to install packages with conda, remove this RUN statement.
# Instead of "conda install", you can also use "mamba install" for a potential speed-up
RUN umask 002 \
&& conda install -n $DSX_KERNEL_CONDENV \
py4j \
&& /bin/bash -c 'source /opt/ibm/build/bin/installutils.sh ; fix-conda-permissions'
# =========================
# Install packages with pip
# =========================
# If you don't need to install packages with pip, remove this RUN statement.
RUN umask 002 \
&& conda run -n $DSX_KERNEL_CONDENV pip install \
astrotools==1.4.2 \
scikit-multilearn \
scikit-plot \
seawater \
fuzzy \
pyecharts==1.9.1 \
&& /bin/bash -c 'source /opt/ibm/build/bin/installutils.sh ; fix-conda-permissions'
# ======================================
# change from build user to runtime user
# ======================================
USER $DSX_USERNAME:wsbuild
Build the custom image, in this example, custom_image:image_tage is wslocal-cust-runtime-pyecharts:1.9.1
podman build -t wslocal-cust-runtime-pyecharts:1.9.1 --build-arg base_image_tag=bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-runtime-py39main@sha256:a7756b946812f2454b01b9101ae984c37c3de0fd3fe1c9205d27e2bf05086572 -f ./Dockerfile
Step 5: Push custom image
podman tag wslocal-cust-runtime-pyecharts:1.9.1 bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-cust-runtime-pyecharts:1.9.1
podman push bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-cust-runtime-pyecharts:1.9.1
Step 6: Upload custom runtime configuration
cp ${cpd_py_runtime_file} ${cust_runtime_file}
vi ${cust_runtime_file}
Find image
in cust_runtime_file and replace the image address with the custome image address, eg. bastion.bj-prod-18.luban.cdl.ibm.com:15000/cp/cpd/wslocal-cust-runtime-pyecharts:1.9.1
Then upload the custom runtime configration
curl -k -X PUT \
"${cpd_url}/zen-data/v1/volumes/files/%2F_global_%2Fconfig%2F.runtime-definitions%2Fibm" \
--header "Authorization: Bearer ${myToken}" \
--header "content-type: multipart/form-data" \
-F upFile=@./${cust_runtime_file}
Step 7: Verify in Watson Studio
Go to a Watson Studio project, Tab Manage, Click Environments, Tab Templates, Click New template, you should be able to see custom image from Software version, and to create a template from it.
Create a test notebook from the custom image runtime template
Promote the custom image runtime template and test notebook to deployment space, and create a notebook job against custom image runtime template to run
You may refer CPD KC on this topic: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=environments-building-custom-images
#Spotlight