I just installed ICPD 3.0.1 and WSL service on OCP 4.4 with 3 master and 3 worker nodes ruinning CoreOS. When I try to create a notebook using
Default Python 3.6 environment it fails with "Failed to load notebook" message. For some reason the jupyter-py36 deployment cannot be created therefore the environment runtime is unable to start.
I changed the spawner log level to TRACE.
This is the log of spawner pod:
Log level set to LOG_LEVEL_TRACE
2020/07/15 11:31:51 utils : INFO : RT Defs: /user-home/_global_/config/.runtime-definitions/ibm
2020/07/15 11:31:51 utils : INFO : RT Defs: /user-home/_global_/config/.runtime-definitions/custom
2020/07/15 11:31:52 Serving spawner at
https://[::]:84432020/07/15 11:32:23 utils : TRACE : Token extracted from authorization header
2020/07/15 11:32:23 utils : TRACE : SPAWNER_PUBLIC_KEY path: /user-home/_global_/config/jwt/public.pem
2020/07/15 11:32:23 utils : DEBUG : Token for user 1000330999 validated.
2020/07/15 11:32:23 utils : TRACE : spawner_runtimes_all_get: ->::<-
2020/07/15 11:32:52 utils : TRACE : Token extracted from authorization header
2020/07/15 11:32:52 utils : DEBUG : Token for user 1000330999 validated.
2020/07/15 11:32:52 utils : TRACE : spawner_list_services_get: ->::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Token extracted from authorization header
2020/07/15 11:32:52 utils : DEBUG : Token for user 1000330999 validated.
2020/07/15 11:32:52 utils : DEBUG : Projects RESTApiImpl: getMemberRole()
2020/07/15 11:32:52 utils : DEBUG : Projects RESTApiImpl: getProjectId()
2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7' ...
2020/07/15 11:32:52 utils : DEBUG : Remote GET:
https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab72020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7/members' ...
2020/07/15 11:32:52 utils : DEBUG : Remote GET:
https://internal-nginx-svc:12443/v2/projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7/members2020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/v2/runtime_definitions?include=launch_configuration&project_id=f72658c8-af7f-4df1-ae11-23cb6dfdfab7' ...
2020/07/15 11:32:52 utils : DEBUG : Remote GET:
https://internal-nginx-svc:12443/v2/runtime_definitions?include=launch_configuration&project_id=f72658c8-af7f-4df1-ae11-23cb6dfdfab72020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
2020/07/15 11:32:52 utils : TRACE : spawner_start_post: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getServerSpec()
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : DEBUG : Loading environment defined by the request: ->{"definition":"jup","env":["APP_ENV_DEPLOYMENT_TARGET=icp4data","APP_ENV_CDSX_NOTEBOOKS_API=internal-nginx-svc:12443","APP_ENV_NOTEBOOKS_JOB_MANAGER=internal-nginx-svc:12443","APP_ENV_AX_PROJECTS_API=internal-nginx-svc:12443","APP_ENV_APSX_API=internal-nginx-svc:12443","APP_ENV_IAM_KEYS_ENDPOINT=https://internal-nginx-svc:12443/auth/jwtpublic","APP_ENV_NOTEBOOKS_API_V2=internal-nginx-svc:12443","RUNTIME_ENV_APSX_URL=https://internal-nginx-svc:12443","APP_ENV_TENANT_MANAGER_TIMEOUT=10","ENVIRONMENT_GUID=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7","DSX_JUPYTER_BASE_ROUTE=dsx-jupyter-py36","APP_ENV_IDLE_LINGER_SECS=64800","DSX_CONFIG=wslocal"],"resources":{"cpu":{"minimum":1000,"request":1000},"gpu":{"minimum":-1,"request":-1},"memory":{"minimum":2048,"request":2048}}}<-
2020/07/15 11:32:52 k8s : DEBUG : Cannot find existing runtime pod using selector [type=jupyter-py36,dsxProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7,dsxUserId=1000330999,runtimeEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7], creating new...
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : TRACE : Using internal cert @ '/etc/certificate' for URL 'https://internal-nginx-svc:12443/api/v1/usermgmt/v1/usermgmt/getTimedToken' ...
2020/07/15 11:32:52 utils : DEBUG : Remote POST:
https://internal-nginx-svc:12443/api/v1/usermgmt/v1/usermgmt/getTimedToken2020/07/15 11:32:52 utils : DEBUG : Remote response code: 200
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: checkDefExists()
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 utils : DEBUG : Runtime defs RESTApiImpl: getDef() -- jupyter-py36
2020/07/15 11:32:52 utils : TRACE : Listing defs for CRI: ->::RTDefName=jupyter-py36::RTEnvId=jupconda36-f72658c8-af7f-4df1-ae11-23cb6dfdfab7::ProjectName=1234::ProjectId=f72658c8-af7f-4df1-ae11-23cb6dfdfab7::<-
2020/07/15 11:32:52 utils : TRACE : Loading single rt def (jupyter-py36) from partial cache ...
2020/07/15 11:32:52 k8s : DEBUG : Claims as volume: scripts
2020/07/15 11:32:52 k8s : DEBUG : Claims as volume: dbdrivers
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Volume subPath: 'projects/$project_id/assets/data_asset' => 'projects/f72658c8-af7f-4df1-ae11-23cb6dfdfab7/assets/data_asset'.
2020/07/15 11:32:52 k8s : DEBUG : Claims as volume: project_dir
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: user-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Repeat claim: cc-home-pvc
2020/07/15 11:32:52 k8s : DEBUG : Creating jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d-svc service in cp4d namespace
2020/07/15 11:32:52 k8s : DEBUG : Creating jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d deployment in cp4d namespace
2020/07/15 11:32:52 k8s : DEBUG : EVs as Secrets: DSX_TOKEN,
2020/07/15 11:32:52 k8s : TRACE : Secret does not exist: secrets "jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d-sct" not found
2020/07/15 11:32:52 k8s : DEBUG : Creating jupyter-py36-81627a37-76a3-455f-a0ea-addfe25b9c4d-sct secret in cp4d namespace
2020/07/15 11:32:52 k8s : DEBUG : Encountered an error while creating a deployment
2020/07/15 11:32:52 spawner_start : ERROR : the server could not find the requested resource
And nginx pod's log:
2020/07/15 11:32:49 [error] 256#256: *715 open() "/usr/local/openresty/nginx/html/v2/lineage_events" failed (2: No such file or directory), client: 10.254.5.246, server: internal-nginx-svc, request: "POST /v2/lineage_events HTTP/1.1", host: "internal-nginx-svc:12443"
2020/07/15 11:32:49 [error] 256#256: *715 open() "/usr/local/openresty/nginx/html/v2/lineage_events" failed (2: No such file or directory), client: 10.254.5.246, server: internal-nginx-svc, request: "POST /v2/lineage_events HTTP/1.1", host: "internal-nginx-svc:12443"
Any suggestions?
Thanks in advance!
------------------------------
Andrey Kirilov
------------------------------
#CloudPakforDataGroup