Cloud Pak for Data

 View Only

  • 1.  Failed notebook in WSL on ICPD 3.0.1

    Posted Thu July 16, 2020 10:44 AM
    I just installed ICPD 3.0.1 and WSL service on OCP 4.4 with 3 master and 3 worker nodes ruinning CoreOS. When I try to create a notebook using  Default Python 3.6 environment it fails with "Failed to load notebook" message. For some reason the jupyter-py36 deployment cannot be created therefore the environment runtime is unable to start.
    I changed the spawner log level  to TRACE.
    This is the log of spawner pod:

    Log level set to LOG_LEVEL_TRACE
    2020/07/15 11:31:51 utils : INFO : RT Defs: /user-home/_global_/config/.runtime-definitions/ibm
    2020/07/15 11:31:51 utils : INFO : RT Defs: /user-home/_global_/config/.runtime-definitions/custom
    2020/07/15 11:31:52 Serving spawner at https://[::]:8443
    2020/07/15 11:32:23 utils : TRACE : Token extracted from authorization header
    2020/07/15 11:32:23 utils : TRACE : SPAWNER_PUBLIC_KEY path: /user-home/_global_/config/jwt/public.pem
    2020/07/15 11:32:23 utils : DEBUG : Token for user 1000330999 validated.
    2020/07/15 11:32:23 utils : TRACE : spawner_runtimes_all_get: ->::<-
    2020/07/15 11:32:52 utils : TRACE : Token extracted from authorization header
    2020/07/15 11:32:52 utils : DEBUG : Token for user 1000330999 validated.
    2020/07/15 11:32:52 utils : TRACE : spawner_list_services_get: ->::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Token extracted from authorization header
    2020/07/15 11:32:52 utils : DEBUG : Token for user 1000330999 validated.
    2020/07/15 11:32:52 utils : DEBUG : Projects RESTApiImpl: getMemberRole()
    2020/07/15 11:32:52 utils : DEBUG : Projects RESTApiImpl: getProjectId()
    2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7' ...
    2020/07/15 11:32:52 utils : DEBUG : Remote GET: https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7
    2020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
    2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7/members' ...
    2020/07/15 11:32:52 utils : DEBUG : Remote GET: https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7/members
    2020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/v2/runtime_definitions?include=launch_configuration&project_id=f72658c8-af7f-4df1-ae11-23cb6dfdfab7' ...
    2020/07/15 11:32:52 utils : DEBUG : Remote GET: https://internal-nginx-svc:12443/v2/runtime_definitions?include=launch_configuration&project_id=f72658c8-af7f-4df1-ae11-23cb6dfdfab7
    2020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
    2020/07/15 11:32:52 utils : TRACE : spawner_start_post: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getServerSpec()
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : DEBUG : Loading environment defined by the request: ->{"definition":"jup","env":["APP_ENV_DEPLOYMENT_TARGET=icp4data","APP_ENV_CDSX_NOTEBOOKS_API=internal-nginx-svc:12443","APP_ENV_NOTEBOOKS_JOB_MANAGER=internal-nginx-svc:12443","APP_ENV_AX_PROJECTS_API=internal-nginx-svc:12443","APP_ENV_APSX_API=internal-nginx-svc:12443","APP_ENV_IAM_KEYS_ENDPOINT=https://internal-nginx-svc:12443/auth/jwtpublic","APP_ENV_NOTEBOOKS_API_V2=internal-nginx-svc:12443","RUNTIME_ENV_APSX_URL=https://internal-nginx-svc:12443","APP_ENV_TENANT_MANAGER_TIMEOUT=10","ENVIRONMENT_GUID=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7","DSX_JUPYTER_BASE_ROUTE=dsx-jupyter-py36","APP_ENV_IDLE_LINGER_SECS=64800","DSX_CONFIG=wslocal"],"resources":{"cpu":{"minimum":1000,"request":1000},"gpu":{"minimum":-1,"request":-1},"memory":{"minimum":2048,"request":2048}}}<-
    2020/07/15 11:32:52 k8s : DEBUG : Cannot find existing runtime pod using selector [type=jupyter-py36,dsxProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7,dsxUserId=1000330999,runtimeEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7], creating new...
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/api/v1/usermgmt/v1/usermgmt/getTimedToken' ...
    2020/07/15 11:32:52 utils : DEBUG : Remote POST: https://internal-nginx-svc:12443/api/v1/usermgmt/v1/usermgmt/getTimedToken
    2020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: checkDefExists()
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
    2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
    2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
    2020/07/15 11:32:52 k8s : DEBUG : Claims as volume: scripts
    2020/07/15 11:32:52 k8s : DEBUG : Claims as volume: dbdrivers
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Volume subPath: 'projects/$project_id/assets/data_asset' => 'projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7/assets/data_asset'.
    2020/07/15 11:32:52 k8s : DEBUG : Claims as volume: project_dir
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
    2020/07/15 11:32:52 k8s : DEBUG : Creating jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d-svc service in cp4d namespace
    2020/07/15 11:32:52 k8s : DEBUG : Creating jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d deployment in cp4d namespace
    2020/07/15 11:32:52 k8s : DEBUG : EVs as Secrets: DSX_TOKEN,
    2020/07/15 11:32:52 k8s : TRACE : Secret does not exist: secrets "jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d-sct" not found
    2020/07/15 11:32:52 k8s : DEBUG : Creating jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d-sct secret in cp4d namespace
    2020/07/15 11:32:52 k8s : DEBUG : Encountered an error while creating a deployment
    2020/07/15 11:32:52 spawner_start : ERROR : the server could not find the requested resource

    And nginx pod's log:
    2020/07/15 11:32:49 [error] 256#256: *715 open() "/usr/local/openresty/nginx/html/v2/lineage_events" failed (2: No such file or directory), client: 10.254.5.246, server: internal-nginx-svc, request: "POST /v2/lineage_events HTTP/1.1", host: "internal-nginx-svc:12443"
    2020/07/15 11:32:49 [error] 256#256: *715 open() "/usr/local/openresty/nginx/html/v2/lineage_events" failed (2: No such file or directory), client: 10.254.5.246, server: internal-nginx-svc, request: "POST /v2/lineage_events HTTP/1.1", host: "internal-nginx-svc:12443"

    Any suggestions?

    Thanks in advance!

    ------------------------------
    Andrey Kirilov
    ------------------------------

    #CloudPakforDataGroup


  • 2.  RE: Failed notebook in WSL on ICPD 3.0.1

    Posted Thu July 16, 2020 10:59 AM
    Hi.

    OCP 4.4 uses kubernetes v1.17+, which has changed some APIs (from betav1->v1) - this is causing issues in our product, and at this time we can not support it.

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 3.  RE: Failed notebook in WSL on ICPD 3.0.1

    Posted Thu September 10, 2020 08:01 AM
    I installed ICPD on OCP 4.5.8 but still receiving the same error.
    https://www.ibm.com/support/knowledgecenter/en/SSQNUZ_3.0.1/cpd/plan/rhos-reqs.html

    ------------------------------
    Andrey Kirilov
    ------------------------------

    Original Message


  • 4.  RE: Failed notebook in WSL on ICPD 3.0.1

    Posted Thu September 10, 2020 08:04 AM
    Hi,

    You need to apply patches as per:
    https://www.ibm.com/support/knowledgecenter/SSQNUZ_3.0.1/cpd/overview/whats-new.html?view=embed#whats-new__sept-2020

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 5.  RE: Failed notebook in WSL on ICPD 3.0.1

    Posted Thu September 10, 2020 10:04 AM
    It works now!
    I installed cpd-3.0.1-ccs-patch-3 and cpd-3.0.1-wsl-patch-2 patches by specifying their names in ./cpd-linux patch command.
    Previously, I used ./cpd-linux status --repo repo.yaml --namespace cpd --patches --available-updates --assembly wsl, which did not find any updates available...
    Thank you Tomasz!

    ------------------------------
    Andrey Kirilov
    ------------------------------

    Original Message