Cloud Pak for Data

 View Only

 Job Jupyter notebook on CP4D faced 502 bad gateway error, can't init Jupyter kernel

Vu Tuyen's profile image
Vu Tuyen posted Wed June 18, 2025 11:20 PM

We use Jupyter Notebook to run a Spark job. It runs successfully sometimes, but fails others (the Failure rate is 30%, and we usually run it again until it succeeds). Below is our environment and log. Does anyone have ideas on how to fix this issue?

We used CP4D version 4.6.6  and Jupyter notebook environment: 

- abseil-cpp=20210324.2
- absl-py=1.0.0
- aiohttp=3.8.1
- aiosignal=1.2.0
- anyio=3.5.0
- appdirs=1.4.4
- argon2-cffi=21.3.0
- argon2-cffi-bindings=21.2.0
- arrow-cpp=8.0.0
- asttokens=2.0.5
- astunparse=1.6.3
- async-timeout=4.0.1
- atomicwrites=1.4.1
- attrs=21.4.0
- autoai-libs=1.14.6
- autoai-ts-libs=1.2.0
- autovizwidget=0.20.0
- av=9.2.0
- azure-core=1.25.0
- azure-cosmos=4.3.0
- babel=2.9.1
- backcall=0.2.0
- beautifulsoup4=4.11.1
- black=22.6.0
- blas=1.0
- bleach=4.1.0
- blinker=1.4
- blosc=1.21.0
- bokeh=2.4.3
- boto3=1.24.28
- botocore=1.27.28
- bottleneck=1.3.5
- branca=0.6.0
- brotli=1.0.9
- brotli-bin=1.0.9
- brotlipy=0.7.0
- brunel=2.3
- brunsli=0.1
- bzip2=1.0.8
- c-ares=1.18.1
- ca-certificates=2022.07.19
- cached-property=1.5.2
- cachetools=4.2.2
- cairo=1.16.0
- cassandra-driver=3.25.0
- certifi=2022.9.24
- cffi=1.15.1
- cfitsio=3.470
- charls=2.2.0
- charset-normalizer=2.0.4
- click=8.0.4
- cloudpickle=2.0.0
- colorama=0.4.4
- conda=4.10.3
- conda-content-trust=0.1.1
- conda-package-handling=1.9.0
- cplex=22.1.0.0
- crypto=1.4.1
- cryptography=37.0.1
- cx-oracle=8.3.0
- cx_oracle=8.3.0
- cycler=0.11.0
- cytoolz=0.11.0
- dask=2022.7.0
- dask-core=2022.7.0
- dataclasses=0.8
- dataengine-spark=local
- dbus=1.13.18
- debugpy=1.5.1
- decorator=5.1.1
- defusedxml=0.7.1
- dill=0.3.5.1
- dm-tree=0.1.5
- docplex=2.22.213
- dsx-spark-job-monitor=0.8.0
- entrypoints=0.4
- et-xmlfile=1.1.0
- et_xmlfile=1.1.0
- etils=0.9.0
- executing=0.8.3
- expat=2.4.9
- fastjsonschema=2.16.2
- ffmpeg=4.2.2
- flatbuffers=1.12
- folium=0.13.0
- fontconfig=2.13.1
- fonttools=4.25.0
- freeglut=3.0.0
- freetds=1.3.3
- freetype=2.11.0
- frozenlist=1.2.0
- fsspec=2021.10.1
- future=0.18.2
- gast=0.4.0
- gensim=4.1.2
- geomet=0.2.1.post1
- gflags=2.2.2
- giflib=5.2.1
- glib=2.69.1
- glog=0.5.0
- gmp=6.2.1
- gnutls=3.6.15
- google-auth=1.33.0
- google-auth-oauthlib=0.4.4
- google-pasta=0.2.0
- googleapis-common-protos=1.56.4
- graphite2=1.3.14
- greenery=3.3.3
- greenlet=1.1.1
- grpc-cpp=1.41.0
- grpcio=1.42.0
- gst-plugins-base=1.14.0
- gstreamer=1.14.0
- h5py=3.6.0
- harfbuzz=1.8.8
- hdf5=1.12.1
- hdijupyterutils=0.20.0
- hyperopt=0.2.5
- ibm-cloud-sdk-core=3.10.1
- ibm-cos-sdk=2.12.0
- ibm-cos-sdk-core=2.12.0
- ibm-cos-sdk-s3transfer=2.12.0
- ibm-db=3.1.3
- ibm-db-sa=0.3.8
- ibm-watson-machine-learning=1.0.253
- ibm-watson-openscale=3.0.23
- ibm-watson-studio-lib=3.0.12
- ibmos2spark=1.0.1
- icu=58.2
- idna=3.3
- imagecodecs=2021.8.26
- imageio=2.19.3
- importlib-metadata=4.11.3
- importlib_resources=5.2.0
- iniconfig=1.1.1
- ipykernel=6.9.1
- ipython=8.4.0
- ipython_genutils=0.2.0
- ipywidgets=7.6.5
- itc-utils=0.1.15
- jasper=2.0.14
- jax=0.3.14
- jaxlib=0.1.75
- jaydebeapi=1.2.3
- jedi=0.18.1
- jinja2=3.0.3
- jmespath=0.10.0
- joblib=1.1.0
- jpeg=9e
- jpype1=1.4.0
- jpype1-py3=0.5.5.4
- json5=0.9.6
- jsonref=0.2
- jsonschema=4.4.0
- jsonsubschema=0.0.6
- jupyter=1.0.0
- jupyter-client=7.3.5
- jupyter-core=4.10.0
- jupyter-pip=0.3.1
- jupyter-server=1.18.1
- jupyter_client=7.3.5
- jupyter_console=6.4.3
- jupyter_core=4.10.0
- jupyter_server=1.18.1
- jupyterlab=3.4.4
- jupyterlab-server=2.15.2
- jupyterlab_pygments=0.1.2
- jupyterlab_server=2.15.2
- jupyterlab_widgets=1.0.0
- jxrlib=1.1
- keras=2.9.0
- keras-preprocessing=1.1.2
- kiwisolver=1.4.2
- krb5=1.19.2
- lale=0.6.19
- lame=3.100
- lcms2=2.12
- ld_impl_linux-64=2.38
- lerc=3.0
- leveldb=1.20
- libaec=1.0.4
- libblas=3.9.0
- libbrotlicommon=1.0.9
- libbrotlidec=1.0.9
- libbrotlienc=1.0.9
- libcblas=3.9.0
- libclang=14.0.6
- libclang13=14.0.6
- libcurl=7.84.0
- libdeflate=1.8
- libedit=3.1.20210910
- libev=4.33
- libevent=2.1.12
- libffi=3.3
- libgcc-ng=11.2.0
- libgfortran-ng=11.2.0
- libgfortran5=11.2.0
- libglu=9.0.0
- libgomp=11.2.0
- libidn2=2.3.2
- liblapack=3.9.0
- liblightgbm=3.3.2
- libllvm11=11.1.0
- libllvm14=14.0.6
- libnghttp2=1.46.0
- libopenblas=0.3.20
- libopencv=4.6.0
- libopus=1.3.1
- libpng=1.6.37
- libpq=12.9
- libprotobuf=3.19.1
- libsodium=1.0.18
- libssh2=1.10.0
- libstdcxx-ng=11.2.0
- libtasn1=4.16.0
- libthrift=0.15.0
- libtiff=4.2.0
- libunistring=0.9.10
- libutf8proc=2.6.1
- libuuid=1.0.3
- libvpx=1.7.0
- libwebp=1.2.2
- libwebp-base=1.2.2
- libxcb=1.15
- libxgboost=1.6.2
- libxml2=2.9.14
- libxslt=1.1.35
- libzopfli=1.0.3
- lightgbm=3.3.2
- llvm-openmp=14.0.6
- llvmlite=0.39.0
- lmdb=0.9.29
- locket=1.0.0
- lomond=0.3.3
- lxml=4.9.1
- lz4-c=1.9.3
- markdown=3.3.4
- markupsafe=2.1.1
- matplotlib=3.5.2
- matplotlib-base=3.5.2
- matplotlib-inline=0.1.6
- mistune=0.8.4
- mlxtend=0.20.0
- mock=4.0.3
- more-itertools=9.0.0
- mpi=1.0
- multidict=5.2.0
- munkres=1.1.4
- mypy-extensions=0.4.3
- mypy_extensions=0.4.3
- mysql-connector-python=8.0.30
- naked=0.1.32
- nbclassic=0.4.8
- nbclient=0.5.13
- nbconvert=6.5.4
- nbformat=5.3.0
- ncurses=6.3
- nest-asyncio=1.5.5
- nettle=3.7.3
- networkx=2.8.4
- nose=1.3.7
- notebook=6.4.10
- notebook-shim=0.2.2
- numactl=2.0.12
- numba=0.56.1
- numexpr=2.8.3
- numpy=1.23.1
- numpy-base=1.23.1
- oauthlib=3.2.0
- onnx=1.12.0
- onnxconverter-common=1.9.0
- opencv=4.6.0
- openh264=2.1.1
- openjpeg=2.4.0
- openmpi=4.1.1
- openpyxl=3.0.10
- openssl=1.1.1q
- opt_einsum=3.3.0
- orc=1.7.4
- packaging=21.3
- pandas=1.4.3
- pandocfilters=1.5.0
- parso=0.8.3
- partd=1.2.0
- pathspec=0.9.0
- patsy=0.5.2
- pcre=8.45
- pexpect=4.8.0
- pickleshare=0.7.5
- pillow=9.2.0
- pip=22.1.2
- pixman=0.40.0
- platformdirs=2.5.2
- plotly=5.9.0
- pluggy=1.0.0
- portion=2.3.0
- portpicker=1.3.1
- project-lib=2.0.7
- prometheus-client=0.14.1
- prometheus_client=0.14.1
- prompt-toolkit=3.0.20
- prompt_toolkit=3.0.20
- protobuf=3.19.1
- psutil=5.9.0
- psycopg2=2.8.6
- ptyprocess=0.7.0
- pure_eval=0.2.2
- py=1.11.0
- py-opencv=4.6.0
- py4j=0.10.9.5
- pyarrow=8.0.0
- pyasn1=0.4.8
- pyasn1-modules=0.2.8
- pycosat=0.6.4
- pycparser=2.21
- pycryptodome=3.10.1
- pycryptodomex=3.15.0
- pydeprecate=0.3.2
- pygments=2.11.2
- pyjwt=2.4.0
- pymssql=2.1.5
- pyodbc=4.0.34
- pyopenssl=22.0.0
- pyparsing=3.0.9
- pyqt=5.9.2
- pyrsistent=0.18.0
- pysocks=1.7.1
- pyspark=3.3.1
- pyspnego=0.3.1
- pyst=1.3.0
- pytest=3.10.1
- python=3.10.4
- python-clang=14.0.6
- python-dateutil=2.8.2
- python-fastjsonschema=2.16.2
- python-graphviz=0.20.1
- python_abi=3.10
- pytorch-base=1.12.1
- pytorch-cpu=1.12.1
- pytorch-lightning=1.6.5
- pytz=2022.1
- pywavelets=1.3.0
- pyyaml=6.0
- pyzmq=23.2.0
- qt=5.9.7
- qtconsole=5.3.2
- qtpy=2.2.0
- re2=2022.04.01
- readline=8.1.2
- requests=2.28.1
- requests-kerberos=0.14.0
- requests-oauthlib=1.3.0
- rsa=4.7.2
- ruamel-yaml=0.16.12
- ruamel-yaml-clib=0.2.6
- ruamel-yaml-conda=0.15.100
- ruamel.yaml=0.16.12
- ruamel.yaml.clib=0.2.6
- ruamel_yaml=0.15.100
- s3transfer=0.6.0
- scikit-image=0.19.2
- scikit-learn=1.1.1
- scipy=1.8.1
- seaborn=0.11.2
- send2trash=1.8.0
- sentencepiece=0.1.96
- setuptools=63.4.1
- shellescape=3.8.1
- simplejson=3.17.6
- sip=4.19.13
- six=1.16.0
- skl2onnx=1.12
- sklearn=0.0.post1
- smart-open=6.1.0
- snapml=1.8.10
- snappy=1.1.9
- sniffio=1.2.0
- sortedcontainers=2.4.0
- soupsieve=2.3.1
- sparkmagic=0.20.0
- sparktspy-nojars=2.3.0.0
- sqlalchemy=1.4.39
- sqlite=3.39.2
- stack_data=0.2.0
- statsmodels=0.13.2
- tabulate=0.8.10
- tbb=2021.5.0
- tenacity=8.0.1
- tensorboard=2.9.1
- tensorboard-data-server=0.6.1
- tensorboard-plugin-wit=1.6.0
- tensorflow=2.9.1
- tensorflow-addons=0.17.0
- tensorflow-base=2.9.2
- tensorflow-cpu=2.9.2
- tensorflow-estimator=2.9.0
- tensorflow-hub=0.12.0
- tensorflow-io-gcs-filesystem=0.26.0
- tensorflow-metadata=1.8.0
- tensorflow-model-optimization=0.7.3
- tensorflow-probability=0.17.0
- tensorflow-text=2.9.0
- termcolor=1.1.0
- terminado=0.13.1
- tf-model-optimization-nightly=0.7.3.dev0
- tf2onnx=1.12.0
- threadpoolctl=2.2.0
- tifffile=2021.7.2
- tinycss2=1.2.1
- tk=8.6.12
- tomli=2.0.1
- toolz=0.11.2
- torch=1.12.1
- torchmetrics=0.8.1
- torchtext=0.13.1a0+35066f2
- torchtext-base=0.13.1
- torchvision=0.13.1
- torchvision-base=0.13.1
- tornado=6.2
- tqdm=4.64.0
- traitlets=5.1.1
- tspy-nojars=2.3.0.0
- typeguard=2.12.0
- typing-extensions=4.3.0
- typing_extensions=4.3.0
- tzdata=2022c
- unixodbc=2.3.11
- urllib3=1.26.11
- wcwidth=0.2.5
- webencodings=0.5.1
- websocket-client=0.58.0
- werkzeug=2.0.3
- wheel=0.37.1
- widgetsnbextension=3.5.2
- wrapt=1.14.1
- x264=1!157.20191217
- xgboost=1.3.3
- xlrd=2.0.1
- xz=5.2.5
- yaml=0.2.5
- yarl=1.8.1
- zeromq=4.3.4
- zfp=0.5.5
- zipp=3.8.0
- zlib=1.2.12
- zstd=1.4.9
This is error log:
uid=1000650000(1000650000) gid=103000(wscommon) groups=103000(wscommon),100(users),3000(wsbuild),1000650000
logfile /var/ibm/logs/jupyter-20250616_073019.log
mkdir: created directory '/home/wsuser/work'
mkdir: created directory '/tmp/1000650000'
mkdir: created directory '/tmp/1000650000/.cache'
---------------------------- User Environment -------------------------
APP_ENV_APSX_API=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_AX_PROJECTS_API=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_CDSX_NOTEBOOKS_API=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_DEPLOYMENT_TARGET=icp4data
APP_ENV_IAM_KEYS_ENDPOINT=https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/auth/jwtpublic
APP_ENV_JOB_ASSET_TYPE=notebook
APP_ENV_JOB_ASSET_UPDATE_URL=/v2/asset_files/notebook/attachment_for_notebook_1fa591c2_d96f_4815_aae2_ace2aef7710d_b902pghoe16t31t8zrjwd6zwf.ipynb?project_id=d79a0e03-38f4-4201-9c31-761e903c4cf5&signature=WkIQFY8zx5%2BPf71aVwO0Qg%3D%3D%3AJlraOHHVfpOIKEUwW4Dv73O3O9ZUyZLeiuTvj8np4sDQKMkK5woblhbGZyOodjcTcrNPozBzAN5GMji%2BdiPApsB%2FdLfJgRqHXu5XSr1ZSgOVSWdzE3luE9HsoNfCpvtXFG4L4F0fmqdXLTWjhv%2BLCTYawS7YeLz9M74GIovDgRzGqZQIE1BWzoetw2E0dZgCMSN5oXOUwqg6Km%2FB%2BVcukaW29vyOrF2KdfEPzscKMSXo8AqPwTnrwLMnASG4nbOF5ET2YY4aEw9B9i34B1bToa7QLr0vk3iPBgbFmWOgD9%2FVQA%3D%3D
APP_ENV_JOB_JOBS_MANAGER_GUID=012414cd-1679-415e-85f9-f1fcf5f62042
APP_ENV_JOB_NOTEBOOK_GUID=1fa591c2-d96f-4815-aae2-ace2aef7710d
APP_ENV_JOB_RUN_ID=ae7a01d4-48c2-4c05-b2e7-945b5d9bf6cc
APP_ENV_JOB_RUN_RESULTS=log_and_new_notebook
APP_ENV_JOBS_LOG=/var/ibm/logs/jupyter.log
APP_ENV_JOB_VERSION_GUID=209275d2-dccc-4ba8-9aae-b8a5c25979ca
APP_ENV_KG_CONNECT_TIMOUT=600
APP_ENV_KG_REQUEST_TIMOUT=600
APP_ENV_KG_STRATEGY=DefaultSpark.Strategy
APP_ENV_NB_CLOUDANT_DB_HOST=wdp-couchdb-svc.cp4dmedia.svc.cluster.local:6984
APP_ENV_NB_CLOUDANT_DB_NAME=prod-ax-jupyter-notebook
APP_ENV_NOTEBOOKS_API_V2=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_NOTEBOOKS_JOB_MANAGER=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_NOTEBOOKS_USER=1000331007
APP_ENV_REAL_TIME_LOG_ENABLED=true
APP_ENV_RUNTIMES_HOST=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_TENANT_MANAGER_TIMEOUT=10
APP_ENV_VOLUMES_CONNECTIONS=[{"id":"1684908850876867","mountPath":"/mnts/huongdgnewcode"},{"id":"1692327854331575","mountPath":"/mnts/Fastmoney-submit"}]
CONDA_HOME=/opt/conda
DSX_CLIENT_DSI_MOUNT_PATH=/etc/cp4d/credentials/client-dsi/
DSX_COMPONENT=jupyter-222n-py
DSX_CONFIG=wslocal-nb2kg
DSX_CPU_LIMIT=1000
DSX_CPU_REQUEST=1000
DSX_CUSTOMIZATION_LOG=/var/ibm/logs/customization-20250616_073019.log
DSX_CUSTOM_LABELS=icpdsupport/jobRunId,icpdsupport/environmentType,icpdsupport/addOnId,icpdsupport/ignore-on-nd-backup,app.kubernetes.io/version,app.kubernetes.io/instance,icpdsupport/app,velero.io/exclude-from-backup,app.kubernetes.io/name
DSX_EVENT_FILTERS=[{"action":"ignore","reasons":["DNSConfigForming","ErrorAddingLogicalPort"]}]
DSX_GPU_LIMIT=-1
DSX_GPU_REQUEST=-1
DSX_INFLUXDB_PORT_8086_TCP_ADDR=172.30.254.97
DSX_INFLUXDB_PORT_8086_TCP_PORT=8086
DSX_INFLUXDB_PORT_8086_TCP_PROTO=tcp
DSX_INFLUXDB_PORT_8086_TCP=tcp://172.30.254.97:8086
DSX_INFLUXDB_PORT=tcp://172.30.254.97:8086
DSX_INFLUXDB_SERVICE_HOST=172.30.254.97
DSX_INFLUXDB_SERVICE_PORT=8086
DSX_INFLUXDB_SERVICE_PORT_API=8086
DSX_JOB_ID=90da918327cac5fd
DSX_JOB_NAME=90da918327cac5fd
DSX_JUPYTER_CONDENV=IBM-WS-System
DSX_JUPYTERLAB_VERSION=3.5
DSX_KERNEL_CONDENV=Python-3.10-Premium
DSX_LOG_DIR=/var/ibm/logs
DSX_MAIN_CONTAINER=90da918327cac5fd-1000331007-1750059007-c
DSX_MAIN_SCRIPT=bash /opt/ibm/ws/bin/run_job.sh
DSX_MEM_LIMIT=1073741824
DSX_MEM_REQUEST=1073741824
DSX_NAMESPACE=cp4dmedia
DSX_NGINX_SVC_URL=https://internal-nginx-svc.cp4dmedia.svc:12443
DSX_NOTEBOOK_VERSION=6.5
DSX_PROJECT_ID=d79a0e03-38f4-4201-9c31-761e903c4cf5
DSX_PROJECT_NAME=FastMoney Scoring
DSX_PROVIDES_PYTHON=3.10
DSX_PROVIDES_WATSON_NLP=3.6
DSX_RTA_SPEC_HASH=18299014916916332898
DSX_RUN_NAME=jupyter-222n-py-headless
DSX_RUNTIME_ASSEMBLY_ID=9f6a159f-5a4f-4b87-b9f2-255a4047ec05
DSX_RUNTIME_ENV_ID=NOT_SPECIFIED
DSX_RUNTIME_TYPE=jupyter-222n-py
DSX_SCOPE_ID=d79a0e03-38f4-4201-9c31-761e903c4cf5
DSX_SCOPE_NAME=FastMoney Scoring
DSX_SCOPE_STORAGE_TYPE=assetfiles
DSX_SCOPE_TYPE=project
DSX_SERVICE_NAME=
DSX_SETTING=Local
DSX_SHARED_MEM_ENABLED=0
DSX_SHARED_MEM_LIMIT=-1
DSX_SYSTEM_CONDENV=IBM-WS-System
DSX_TLS_KEY=/var/ibm/tls/jupyter-key.pem
DSX_USER_ID=1000331007
DSX_USERNAME=wsuser
DSX_VAR_DIR=/var/ibm
DSX_VARIANT=wslocal-x86
ENVIRONMENT_GUID=6818b2a0-4aa1-4d4e-a9bd-f8f814eb8fbb
HOME=/home/wsuser
HOSTNAME=90da918327cac5fd-1000331007-1750059007-8gpc8
IBM_DB_HOME=/opt/ibm/dsdriver
JAVA_HOME=/etc/alternatives/java_sdk_openjdk
JOBLIB_TEMP_FOLDER=/tmp/1000650000
KERNEL_CONDENV=Python-3.10-Premium
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
LD_LIBRARY_PATH=/opt/ibm/dsdriver/lib:/opt/oracle/lib
LOAD_PATH=/opt/ibm/nlpmodels
OLDPWD=/home/wsuser
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/ibm/dsdriver/bin
PROJECT_ID=d79a0e03-38f4-4201-9c31-761e903c4cf5
PROJECT_NAME=FastMoney Scoring
PWD=/home/wsuser/work
PYTHONPATH=/var/ibm/.ws/ax-ext/lib
REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
RUNTIME_ENV_ACCESS_TOKEN_FILE=/etc/cp4d/credentials/client-dsi//DSX_TOKEN
RUNTIME_ENV_APSX_URL=https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
RUNTIME_FLIGHT_SERVICE_URL=grpc+tls://wdp-connect-flight.cp4dmedia.svc.cluster.local:443
RUNTIME_HARDWARE_SPEC={"num_cpu": "1", "mem": "1"}
SETUPTOOLS_USE_DISTUTILS=stdlib
SHELL=/usr/bin/bash
SHLVL=2
SSL_CERT_FILE=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
TERM=xterm
TF_NUM_INTEROP_THREADS=1
TF_NUM_INTRAOP_THREADS=1
TMPDIR=/tmp/1000650000
TZ=UTC
USER_ACCESS_TOKEN=...
USER_ID=1000331007
_=/usr/bin/printenv
-----------------------------------------------------------------------
mkdir: created directory '/home/wsuser/jars'
JUPYTER_CONFIG_DIR is '/var/ibm/.ws/ax-ext/config/wslocal-nb2kg'
mkdir: created directory '/tmp/1000650000/jupyter-data-45HNRWk1/kernels'
'/opt/ibm/run/kernels/python3' -> '/tmp/1000650000/jupyter-data-45HNRWk1/kernels/python3'
'/opt/ibm/run/kernels/python3/kernel.json' -> '/tmp/1000650000/jupyter-data-45HNRWk1/kernels/python3/kernel.json'
mkdir: created directory '/home/wsuser/.local'
mkdir: created directory '/home/wsuser/.local/lib'
mkdir: created directory '/home/wsuser/.local/lib/python3.10'
mkdir: created directory '/home/wsuser/.local/lib/python3.10/site-packages'
-----------------------------------------------------------------------
PYTHONPATH=/var/ibm/.ws/ax-ext/lib
JUPYTER_PATH=
Jupyter paths: {"runtime": ["/tmp/1000650000/jupyter-data-45HNRWk1/runtime"], "config": ["/var/ibm/.ws/ax-ext/config/wslocal-nb2kg", "/home/wsuser/.local/etc/jupyter", "/opt/conda/envs/IBM-WS-System/etc/jupyter", "/usr/local/etc/jupyter", "/etc/jupyter"], "data": ["/tmp/1000650000/jupyter-data-45HNRWk1", "/home/wsuser/.local/share/jupyter", "/opt/conda/envs/IBM-WS-System/share/jupyter", "/usr/local/share/jupyter", "/usr/share/jupyter"]}
Jupyter command: jupyter notebook --KernelSpecManager.ensure_native_kernel=False --NotebookApp.port=8888 --NotebookApp.keyfile=/var/ibm/tls/jupyter-key.pem --NotebookApp.certfile=/var/ibm/tls/jupyter-cert.pem
-----------------------------------------------------------------------
APP_ENV_APSX_API=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_AX_PROJECTS_API=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_CDSX_NOTEBOOKS_API=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_DEPLOYMENT_TARGET=icp4data
APP_ENV_IAM_KEYS_ENDPOINT=https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/auth/jwtpublic
APP_ENV_JOB_ASSET_TYPE=notebook
APP_ENV_JOB_ASSET_UPDATE_URL=/v2/asset_files/notebook/attachment_for_notebook_1fa591c2_d96f_4815_aae2_ace2aef7710d_b902pghoe16t31t8zrjwd6zwf.ipynb?project_id=d79a0e03-38f4-4201-9c31-761e903c4cf5&signature=WkIQFY8zx5%2BPf71aVwO0Qg%3D%3D%3AJlraOHHVfpOIKEUwW4Dv73O3O9ZUyZLeiuTvj8np4sDQKMkK5woblhbGZyOodjcTcrNPozBzAN5GMji%2BdiPApsB%2FdLfJgRqHXu5XSr1ZSgOVSWdzE3luE9HsoNfCpvtXFG4L4F0fmqdXLTWjhv%2BLCTYawS7YeLz9M74GIovDgRzGqZQIE1BWzoetw2E0dZgCMSN5oXOUwqg6Km%2FB%2BVcukaW29vyOrF2KdfEPzscKMSXo8AqPwTnrwLMnASG4nbOF5ET2YY4aEw9B9i34B1bToa7QLr0vk3iPBgbFmWOgD9%2FVQA%3D%3D
APP_ENV_JOB_JOBS_MANAGER_GUID=012414cd-1679-415e-85f9-f1fcf5f62042
APP_ENV_JOB_NOTEBOOK_GUID=1fa591c2-d96f-4815-aae2-ace2aef7710d
APP_ENV_JOB_RUN_ID=ae7a01d4-48c2-4c05-b2e7-945b5d9bf6cc
APP_ENV_JOB_RUN_RESULTS=log_and_new_notebook
APP_ENV_JOBS_LOG=/var/ibm/logs/jupyter.log
APP_ENV_JOB_VERSION_GUID=209275d2-dccc-4ba8-9aae-b8a5c25979ca
APP_ENV_KG_CONNECT_TIMOUT=600
APP_ENV_KG_REQUEST_TIMOUT=600
APP_ENV_KG_STRATEGY=DefaultSpark.Strategy
APP_ENV_NB_CLOUDANT_DB_HOST=wdp-couchdb-svc.cp4dmedia.svc.cluster.local:6984
APP_ENV_NB_CLOUDANT_DB_NAME=prod-ax-jupyter-notebook
APP_ENV_NOTEBOOKS_API_V2=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_NOTEBOOKS_JOB_MANAGER=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_NOTEBOOKS_USER=1000331007
APP_ENV_REAL_TIME_LOG_ENABLED=true
APP_ENV_RUNTIMES_HOST=internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
APP_ENV_TENANT_MANAGER_TIMEOUT=10
APP_ENV_VOLUMES_CONNECTIONS=[{"id":"1684908850876867","mountPath":"/mnts/huongdgnewcode"},{"id":"1692327854331575","mountPath":"/mnts/Fastmoney-submit"}]
_CE_CONDA=
_CE_M=
CLASSPATH=/home/wsuser/jars/*:/user-home/_global_/dbdrivers/*:/user-home/_global_/dbdrivers/jdbc/*:/opt/jdbc/*:/user-home/_global_/dbdrivers/jdbc/default/*
CONDA_DEFAULT_ENV=IBM-WS-System
CONDA_EXE=/opt/conda/bin/conda
CONDA_HOME=/opt/conda
CONDA_PREFIX=/opt/conda/envs/IBM-WS-System
CONDA_PROMPT_MODIFIER=(IBM-WS-System) 
CONDA_PYTHON_EXE=/opt/conda/bin/python
CONDA_SHLVL=1
DSX_CLIENT_DSI_MOUNT_PATH=/etc/cp4d/credentials/client-dsi/
DSX_COMPONENT=jupyter-222n-py
DSX_CONFIG=wslocal-nb2kg
DSX_CPU_LIMIT=1000
DSX_CPU_REQUEST=1000
DSX_CUSTOMIZATION_LOG=/var/ibm/logs/customization-20250616_073019.log
DSX_CUSTOM_LABELS=icpdsupport/jobRunId,icpdsupport/environmentType,icpdsupport/addOnId,icpdsupport/ignore-on-nd-backup,app.kubernetes.io/version,app.kubernetes.io/instance,icpdsupport/app,velero.io/exclude-from-backup,app.kubernetes.io/name
DSX_EVENT_FILTERS=[{"action":"ignore","reasons":["DNSConfigForming","ErrorAddingLogicalPort"]}]
DSX_GPU_LIMIT=-1
DSX_GPU_REQUEST=-1
DSX_INFLUXDB_PORT_8086_TCP_ADDR=172.30.254.97
DSX_INFLUXDB_PORT_8086_TCP_PORT=8086
DSX_INFLUXDB_PORT_8086_TCP_PROTO=tcp
DSX_INFLUXDB_PORT_8086_TCP=tcp://172.30.254.97:8086
DSX_INFLUXDB_PORT=tcp://172.30.254.97:8086
DSX_INFLUXDB_SERVICE_HOST=172.30.254.97
DSX_INFLUXDB_SERVICE_PORT=8086
DSX_INFLUXDB_SERVICE_PORT_API=8086
DSX_JOB_ID=90da918327cac5fd
DSX_JOB_NAME=90da918327cac5fd
DSX_JUPYTER_CONDENV=IBM-WS-System
DSX_JUPYTERLAB_VERSION=3.5
DSX_KERNEL_CONDENV=Python-3.10-Premium
DSX_LOG_DIR=/var/ibm/logs
DSX_MAIN_CONTAINER=90da918327cac5fd-1000331007-1750059007-c
DSX_MAIN_SCRIPT=bash /opt/ibm/ws/bin/run_job.sh
DSX_MEM_LIMIT=1073741824
DSX_MEM_REQUEST=1073741824
DSX_NAMESPACE=cp4dmedia
DSX_NGINX_SVC_URL=https://internal-nginx-svc.cp4dmedia.svc:12443
DSX_NOTEBOOK_VERSION=6.5
DSX_PROJECT_ID=d79a0e03-38f4-4201-9c31-761e903c4cf5
DSX_PROJECT_NAME=FastMoney Scoring
DSX_PROVIDES_PYTHON=3.10
DSX_PROVIDES_WATSON_NLP=3.6
DSX_RTA_SPEC_HASH=18299014916916332898
DSX_RUN_NAME=jupyter-222n-py-headless
DSX_RUNTIME_ASSEMBLY_ID=9f6a159f-5a4f-4b87-b9f2-255a4047ec05
DSX_RUNTIME_ENV_ID=NOT_SPECIFIED
DSX_RUNTIME_TYPE=jupyter-222n-py
DSX_SCOPE_ID=d79a0e03-38f4-4201-9c31-761e903c4cf5
DSX_SCOPE_NAME=FastMoney Scoring
DSX_SCOPE_STORAGE_TYPE=assetfiles
DSX_SCOPE_TYPE=project
DSX_SERVICE_NAME=
DSX_SETTING=Local
DSX_SHARED_MEM_ENABLED=0
DSX_SHARED_MEM_LIMIT=-1
DSX_SYSTEM_CONDENV=IBM-WS-System
DSX_TLS_KEY=/var/ibm/tls/jupyter-key.pem
DSX_USER_ID=1000331007
DSX_USERNAME=wsuser
DSX_VAR_DIR=/var/ibm
DSX_VARIANT=wslocal-x86
ENVIRONMENT_GUID=6818b2a0-4aa1-4d4e-a9bd-f8f814eb8fbb
HOME=/home/wsuser
HOSTNAME=90da918327cac5fd-1000331007-1750059007-8gpc8
IBM_DB_HOME=/opt/ibm/dsdriver
JAVA_HOME=/etc/alternatives/java_sdk_openjdk
JOBLIB_TEMP_FOLDER=/tmp/1000650000
JUPYTER_CONFIG_DIR=/var/ibm/.ws/ax-ext/config/wslocal-nb2kg
JUPYTER_DATA_DIR=/tmp/1000650000/jupyter-data-45HNRWk1
KERNEL_CONDENV=Python-3.10-Premium
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
LD_LIBRARY_PATH=/opt/ibm/dsdriver/lib:/opt/oracle/lib
LOAD_PATH=/opt/ibm/nlpmodels
OLDPWD=/home/wsuser
PATH=/opt/conda/envs/IBM-WS-System/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/ibm/dsdriver/bin
PROJECT_ID=d79a0e03-38f4-4201-9c31-761e903c4cf5
PROJECT_NAME=FastMoney Scoring
PWD=/home/wsuser/work
PYTHONPATH=/var/ibm/.ws/ax-ext/lib
REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
RUNTIME_ENV_ACCESS_TOKEN_FILE=/etc/cp4d/credentials/client-dsi//DSX_TOKEN
RUNTIME_ENV_APSX_URL=https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443
RUNTIME_FLIGHT_SERVICE_URL=grpc+tls://wdp-connect-flight.cp4dmedia.svc.cluster.local:443
RUNTIME_HARDWARE_SPEC={"num_cpu": "1", "mem": "1"}
SETUPTOOLS_USE_DISTUTILS=stdlib
SHELL=/usr/bin/bash
SHLVL=2
SSL_CERT_FILE=/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
TERM=xterm
TF_NUM_INTEROP_THREADS=1
TF_NUM_INTRAOP_THREADS=1
TMPDIR=/tmp/1000650000
TZ=UTC
USER_ACCESS_TOKEN=...
USER_ID=1000331007
_=/usr/bin/printenv
-----------------------------------------------------------------------
[I 07:30:20.974 NotebookApp] Requested environment for parameter cache_expired: None
[I 07:30:20.990 NotebookApp] Requested environment for parameter dsx_customization_log: None
[I 07:30:20.990 NotebookApp] Requested environment for parameter jobs_manager.job_working_dir: None
[I 07:30:21.077 NotebookApp] iopub data rate limit parameter will be set to 10000000000.0
[I 07:30:21.118 NotebookApp] Requested environment for parameter iam_keys_path: None
[I 07:30:21.119 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter COS.softlayer.host: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter COS.bluemix.host: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter notebooks_api.request_timeout: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter notebooks_api.connect_timeout: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter notebooks_api.retries: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter spark.tenant_id: None
[I 07:30:21.134 NotebookApp] Requested environment for parameter deployment: None
[I 07:30:21.437 NotebookApp] Writing notebook server cookie secret to /tmp/1000650000/jupyter-data-45HNRWk1/runtime/notebook_cookie_secret
[I 07:30:25.891 NotebookApp] Requested environment for parameter jobs_manager.url: None
[I 07:30:25.891 NotebookApp] Requested environment for parameter jobs_manager.notification_path: None
[I 07:30:25.891 NotebookApp] Requested environment for parameter jobs_manager.url: None
[I 07:30:25.891 NotebookApp] Requested environment for parameter jobs_manager.log_status_path: None
[I 07:30:25.891 NotebookApp] Job execution state notifications sending will be done using the following URI: https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v1/notebook_executions/notifications
[I 07:30:25.898 NotebookApp] Requested environment for parameter iam.keys_endpoint: b
[I 07:30:25.898 NotebookApp] Starting request to the URL: https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/auth/jwtpublic
[I 07:30:25.898 NotebookApp] Requested environment for parameter http_client.verify_ssl: None
[I 07:30:25.922 NotebookApp] Request finished to the URL: https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/auth/jwtpublic
[I 07:30:25.926 NotebookApp] Installing AutoShutdownHandler: poll every 80s, linger for 215s
[I 07:30:25.926 NotebookApp] Loaded nb2kg extension
[I 07:30:25.926 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/kernelspecs/(?P<kernel_name>[\\w\\.\\-%]+)$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.KernelSpecHandler'>, kwargs=None, name=None)
[I 07:30:25.926 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/kernelspecs$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.MainKernelSpecHandler'>, kwargs=None, name=None)
[I 07:30:25.926 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/kernels/(?P<kernel_id>\\w+-\\w+-\\w+-\\w+-\\w+)/channels$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.WebSocketChannelsHandler'>, kwargs=None, name=None)
[I 07:30:25.926 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/kernels/(?P<kernel_id>\\w+-\\w+-\\w+-\\w+-\\w+)/(?P<action>restart|interrupt)$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.KernelActionHandler'>, kwargs=None, name=None)
[I 07:30:25.926 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/kernels/(?P<kernel_id>\\w+-\\w+-\\w+-\\w+-\\w+)$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.KernelHandler'>, kwargs=None, name=None)
[I 07:30:25.926 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/ax/kernels$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.KernelShutdownHandler'>, kwargs=None, name=None)
[I 07:30:25.927 NotebookApp] Overriding handler URLSpec('/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/api/kernels$', <class 'cdsax_jupyter_extensions.nb2kg.handlers.MainKernelHandler'>, kwargs=None, name=None)
[I 07:30:25.927 NotebookApp] Prestarting kernel
[I 07:30:25.927 NotebookApp] One of the manadatory parameters for kernel initialization is not defined: notebook id, project id or space id, token
[I 2025-06-16 07:30:26.087 ElyraApp] Config {'KernelSpecManager': {'ensure_native_kernel': False}, 'NotebookApp': {'port': 8888, 'keyfile': '/var/ibm/tls/jupyter-key.pem', 'certfile': '/var/ibm/tls/jupyter-cert.pem', 'log_level': 'INFO', 'base_url': '/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5', 'port_retries': 512, 'ip': '*', 'open_browser': False, 'allow_remote_access': True, 'ssl_options': {'ssl_version': <_SSLMethod.PROTOCOL_TLSv1_2: 5>, 'ciphers': 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305'}, 'allow_origin': '*', 'trust_xheaders': True, 'tornado_settings': {'log_function': <function ax_log_request at 0x7f63dfee58b0>, 'headers': {'Content-Security-Policy': "frame-ancestors 'self' http://localhost:* https://localhost:* http://ibm-nginx-svc.ibm-private-cloud:* https://ibm-nginx-svc.ibm-private-cloud:* https://*.dataplatform.test.cloud.ibm.com:*", 'Cache-Control': 'no-cache, no-store, must-revalidate', 'Strict-Transport-Security': 'max-age=31536000', 'X-Content-Type-Options': 'nosniff', 'Pragma': 'no-cache', 'X-XSS-Protection': '1; mode=block'}, 'xsrf_cookies': False}, 'extra_template_paths': ['/var/ibm/.ws/ax-ext/config/wslocal-nb2kg/custom/templates'], 'jinja_template_vars': {'spark_monitor': 'enabled'}, 'iopub_data_rate_limit': 10000000000.0, 'terminals_enabled': False, 'contents_manager_class': 'cdsax_jupyter_extensions.CDSAXContentsManager.CDSAXContentsManager', 'login_handler_class': 'cdsax_jupyter_extensions.nb2kg.NB2KGLoginHandler', 'kernel_manager_class': 'cdsax_jupyter_extensions.nb2kg.managers.RemoteKernelManager', 'session_manager_class': 'cdsax_jupyter_extensions.nb2kg.managers.SessionManager', 'kernel_spec_manager_class': 'cdsax_jupyter_extensions.nb2kg.managers.DefaultSparkKernelSpecManager', 'nbserver_extensions': {'cdsax_jupyter_extensions.CDSAXExecutionHandler': True, 'cdsax_jupyter_extensions.AXZMQChannelsHandler': False, 'cdsax_jupyter_extensions.CDSAXNotebooksAPIHandler': True, 'cdsax_jupyter_extensions.CDSAXNotebookSessionsHandler': True, 'cdsax_jupyter_extensions.MonitoringHandler': True, 'cdsax_jupyter_extensions.nb2kg': True, 'cdsax_jupyter_extensions.ActivitySummaryHandler': True, 'cdsax_jupyter_extensions.icp4data.AutoShutdownHandlerRTM': True, 'cdsax_jupyter_extensions.RunningNotebooksHandler': True, 'elyra': True, 'jupyter_lsp': True, 'jupyter_resource_usage': True, 'jupyterlab': True, 'jupyterlab_git': True, 'nbdime': True}}, 'HTMLExporter': {'require_js_url': '/data/notebooks/converter/static/ajax/libs/require.js/2.1.10/require.min.js', 'mathjax_url': '/data/notebooks/converter/static/MathJax-2.7.7/MathJax.js?config=TeX-AMS_HTML'}, 'NotebookExecutionClient': {'real_time_log_enabled': True}}
[I 2025-06-16 07:30:26.158 ElyraApp] Registering airflow processor 'elyra.pipeline.airflow.processor_airflow.AirflowPipelineProcessor'...
[I 2025-06-16 07:30:26.576 ElyraApp] Registering kfp processor 'elyra.pipeline.kfp.processor_kfp.KfpPipelineProcessor'...
[I 2025-06-16 07:30:26.654 ElyraApp] Registering local processor 'elyra.pipeline.local.processor_local.LocalPipelineProcessor'...
[I 07:30:26.864 NotebookApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[W 2025-06-16 07:30:27.156 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.156 LabApp] 'keyfile' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.156 LabApp] 'certfile' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.156 LabApp] 'certfile' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.156 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.156 LabApp] 'port_retries' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.156 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'allow_remote_access' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'ssl_options' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'allow_origin' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'trust_xheaders' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'tornado_settings' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2025-06-16 07:30:27.157 LabApp] 'jinja_template_vars' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2025-06-16 07:30:27.157 LabApp] 'iopub_data_rate_limit' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'terminals_enabled' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'contents_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'login_handler_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'kernel_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'session_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'kernel_spec_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2025-06-16 07:30:27.157 LabApp] 'kernel_spec_manager_class' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2025-06-16 07:30:27.159 LabApp] JupyterLab extension loaded from /opt/conda/envs/IBM-WS-System/lib/python3.9/site-packages/jupyterlab
[I 2025-06-16 07:30:27.159 LabApp] JupyterLab application directory is /opt/conda/envs/IBM-WS-System/share/jupyter/lab
[I 07:30:27.225 NotebookApp] Serving notebooks from local directory: /home/wsuser/work
[I 07:30:27.225 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 07:30:27.225 NotebookApp] https://90da918327cac5fd-1000331007-1750059007-8gpc8:8888/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/?token=c6a4b091d78eb89a4829a215f35bdb375ccecdfc9b8ad760
[I 07:30:27.225 NotebookApp]  or https://127.0.0.1:8888/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/?token=c6a4b091d78eb89a4829a215f35bdb375ccecdfc9b8ad760
[I 07:30:27.225 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 07:30:27.229 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///tmp/1000650000/jupyter-data-45HNRWk1/runtime/nbserver-10-open.html
    Or copy and paste one of these URLs:
        https://90da918327cac5fd-1000331007-1750059007-8gpc8:8888/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/?token=c6a4b091d78eb89a4829a215f35bdb375ccecdfc9b8ad760
     or https://127.0.0.1:8888/dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/?token=c6a4b091d78eb89a4829a215f35bdb375ccecdfc9b8ad760
[I 07:30:30.050 NotebookApp] Requested environment for parameter tenants_api.request_timeout: b
[I 07:30:30.050 NotebookApp] Check access with session key: POST, localhost:8888, /dsx-jupyter/ibmdsxuser-1000331007/d79a0e03-38f4-4201-9c31-761e903c4cf5/v2/jobs
[I 07:30:30.050 NotebookApp] Requested environment for parameter session_job_key: None
[I 07:30:30.050 NotebookApp] Access with job_session_key is denied
[I 07:30:30.050 NotebookApp] Attempt 0. Decoding user profile
[I 07:30:30.050 NotebookApp] Starting verification
[I 07:30:30.051 NotebookApp] Verification finished
[I 07:30:30.051 NotebookApp] User Access Granted
[I 07:30:30.051 NotebookApp] Authorization will be done with bearer token
[I 07:30:30.051 NotebookApp] user context is set
[I 07:30:30.052 NotebookApp] Requested environment for parameter apsx_api.url: b
[I 07:30:30.052 NotebookApp] KERNEL_ACTIVITY_LOG is not set.
[I 07:30:30.052 NotebookApp] START SUBMITTING JOB
[I 07:30:30.052 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:30:30.052 NotebookApp] Asset path: , asset_type: notebook
[I 07:30:30.052 NotebookApp] Requested environment for parameter environment_guid: None
[I 07:30:30.053 NotebookApp] Requested environment for parameter apsx_api.assets_path: b
[I 07:30:30.053 NotebookApp] Requested environment for parameter apsx_api.url: b
[I 07:30:30.053 NotebookApp] Retrieving asset metadata from https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/assets/1fa591c2-d96f-4815-aae2-ace2aef7710d?project_id=d79a0e03-38f4-4201-9c31-761e903c4cf5
[I 07:30:30.053 NotebookApp] Requested environment for parameter http_client.verify_ssl: None
[I 07:30:30.092 NotebookApp] Retrieving asset metadata retrieved from https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/assets/1fa591c2-d96f-4815-aae2-ace2aef7710d?project_id=d79a0e03-38f4-4201-9c31-761e903c4cf5
[I 07:30:30.093 NotebookApp] Get Default kernel
[I 07:30:30.093 NotebookApp] Hummingbird listng kernels
[I 07:30:30.093 NotebookApp] Request list kernel specs at: /api/kernelspecs
[I 07:30:30.093 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:30:30.093 NotebookApp] Compute cache is expired.
[I 07:30:30.093 NotebookApp] Requested environment for parameter environments_api.url: b
[I 07:30:30.093 NotebookApp] Requests access info https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/environments/6818b2a0-4aa1-4d4e-a9bd-f8f814eb8fbb/access_info?project_id=d79a0e03-38f4-4201-9c31-761e903c4cf5
[I 07:30:30.093 NotebookApp] Requested environment for parameter http_client.verify_ssl: None
[I 07:30:30.187 NotebookApp] Environments Access Info retrieved successfully for project: d79a0e03-38f4-4201-9c31-761e903c4cf5
[I 07:30:30.187 NotebookApp] Compute cache was set successfully
[I 07:30:30.187 NotebookApp] User is allowed to access spark service.
[I 07:30:30.187 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:30:30.188 NotebookApp] Start GET request to url https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/spark/ae/df79c7a8c4214bbcaff120e61b9f989d/jkg/api/kernelspecs
[I 07:30:30.267 NotebookApp] GET request to url https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/spark/ae/df79c7a8c4214bbcaff120e61b9f989d/jkg/api/kernelspecs finished: status: 200
[I 07:30:30.267 NotebookApp] KERNEL LIST: ['r42', 'r', 'scala', 'python39', 'python310']
[I 07:30:30.267 NotebookApp] Hummingbird kernels: {'kernelspecs': {'python310': {'name': 'python310', 'resources': {}, 'spec': {'interrupt_mode': 'signal', 'display_name': 'Python 3.10 with Spark', 'env': {'PATH': '/opt/ibm/conda/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin/:sbin:/bin', 'PYSPARK_PYTHON': '/opt/ibm/conda/miniconda/bin/python', 'PYTHONPATH': '/opt/ibm/spark/python:/opt/ibm/spark/python/lib/py4j-0.10.7-src.zip', 'INSTALL_DIR': '/opt/ibm', 'SPARK_HOME': '/opt/ibm/spark', 'PYTHONSTARTUP': '/opt/ibm/kernelScript/python/shell.py'}, 'language': 'python3', 'process_proxy': {'class_name': 'enterprise_gateway.services.processproxies.processproxy.LocalProcessProxy', 'config': {}}, 'argv': ['/opt/ibm/kernelScript/python/python-kernel-wrapper.sh', 'python3.10', '{connection_file}'], 'metadata': {}}}}, 'default': 'scala'}
[I 07:30:30.268 NotebookApp] Default kernel: python310
[I 07:30:30.268 NotebookApp] Default kernel: python310
[I 07:30:30.268 NotebookApp] Request start kernel: kernel_id=None, path="bg-1fa591c2-d96f-4815-aae2-ace2aef7710d"
[I 07:30:30.268 NotebookApp] Request new kernel at: /api/kernels
[I 07:30:30.268 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:30:30.268 NotebookApp] Requested environment for parameter environments_api.cluster_id: None
[I 07:30:30.268 NotebookApp] The following variables will be set: dict_keys(['PROJECT_ID', 'USER_ACCESS_TOKEN', 'RUNTIME_ENV_APSX_URL', 'USER_ID', 'PROJECT_NAME'])
[I 07:30:30.269 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:30:30.269 NotebookApp] Start POST request to url https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/spark/ae/df79c7a8c4214bbcaff120e61b9f989d/jkg/api/kernels
[I 07:31:09.300 NotebookApp] POST request to url https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/spark/ae/df79c7a8c4214bbcaff120e61b9f989d/jkg/api/kernels finished: status: 201
[I 07:31:09.300 NotebookApp] Kernel started: 8e636e0f-2204-48d1-8b38-72a2875aeaed
[I 07:31:09.300 NotebookApp] Requested environment for parameter nb2kg.cloudant.host: None
[I 07:31:09.300 NotebookApp] Requested environment for parameter nb2kg.cloudant.account: None
[I 07:31:09.300 NotebookApp] Requested environment for parameter nb2kg.cloudant.api_key: None
[I 07:31:09.300 NotebookApp] Requested environment for parameter nb2kg.cloudant.secret: None
[I 07:31:09.300 NotebookApp] Requested environment for parameter nb2kg.cloudant.database: None
[I 07:31:09.300 NotebookApp] [AX Cloudant Connector] Starting request to Cloudant, method: POST, path: , URL: https://wdp-couchdb-svc.cp4dmedia.svc.cluster.local:6984/prod-ax-jupyter-notebook
[I 07:31:09.346 NotebookApp] [AX Cloudant Connector] Request successfully proceed code: 201, method: POST, path: 
[I 07:31:09.347 NotebookApp] Cloudant insert result: {'ok': True, 'id': '4b7feef9bca132c1b3a6b8a4d4b98576', 'rev': '1-7a4d3d1aba864f6f9afdb7bcb217065c'}
[I 07:31:09.347 NotebookApp] Request kernel at: /api/kernels/8e636e0f-2204-48d1-8b38-72a2875aeaed
[I 07:31:09.347 NotebookApp] Requested environment for parameter nb2kg.strategy: None
[I 07:31:09.347 NotebookApp] Start GET request to url https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/spark/ae/df79c7a8c4214bbcaff120e61b9f989d/jkg/api/kernels/8e636e0f-2204-48d1-8b38-72a2875aeaed
[E 07:31:09.443 NotebookApp] HTTP Error 502 requesting https://internal-nginx-svc.cp4dmedia.svc.cluster.local:12443/v2/spark/ae/df79c7a8c4214bbcaff120e61b9f989d/jkg/api/kernels/8e636e0f-2204-48d1-8b38-72a2875aeaed. Response: b'<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.20.1</center>\r\n</body>\r\n</html>\r\n'
[I 07:31:09.443 NotebookApp] Unable to get kernel using url: /api/kernels/8e636e0f-2204-48d1-8b38-72a2875aeaed, method: GET, status code: 502
[I 07:31:09.443 NotebookApp] Requested environment for parameter notebook_jobs_log_path: None

Related Content