We are running CA 11.2.4 on a RHEL 7.9 host and Jupiter Server 11.2.4 on a RHEL 8.8 host running podman 4.4.1.

Opening a new notebook in CA gives 'An unexpected error occurred. Check the Jupyter server logs.'

On the notebook server side, the newly created 'ca_jupyter_server_instance' is terminated immediately, does not contain a single log entry while the logs in conainer 'ca_jupyter_hub' say:

[I 2023-07-18 15:30:22.448 JupyterHub base:1242] Spawning bi_origin is http://cognos.test.xyz for request /bi/v1/jupyter/hub/user/xOjpBbm9ueW1vdXM_.fd305be0b0afd6e36d2653e685e93d13/caload/notebooks/new?kernel_name=python3&trust=jupyter
[I 2023-07-18 15:30:22.549 JupyterHub dockerspawner:452] Container 'ca_jupyter-xOjpBbm9ueW1vdXM_5F_2Efd305be0b0afd6e36d2653e685e93d13' is gone
[I 2023-07-18 15:30:22.620 JupyterHub dockerspawner:540] Created container 'ca_jupyter-xOjpBbm9ueW1vdXM_5F_2Efd305be0b0afd6e36d2653e685e93d13' (id: a529aa2) from image ca_jupyter_server_instance:latest
[I 2023-07-18 15:30:22.620 JupyterHub dockerspawner:557] Starting container 'ca_jupyter-xOjpBbm9ueW1vdXM_5F_2Efd305be0b0afd6e36d2653e685e93d13' (id: a529aa2)
[E 2023-07-18 15:30:22.694 JupyterHub user:482] Unhandled error starting xOjpBbm9ueW1vdXM_.fd305be0b0afd6e36d2653e685e93d13's server: 500 Server Error for http+docker://localhost/v1.41/containers/a529aa270e36ac80688bca10c5bee96f3e21086219c00f168bd04406650624de/start: Internal Server Error ("failed to mount runtime directory for rootless netns: no such file or directory")
[E 2023-07-18 15:30:22.714 JupyterHub web:1808] Uncaught exception GET /bi/v1/jupyter/hub/user/xOjpBbm9ueW1vdXM_.fd305be0b0afd6e36d2653e685e93d13/caload/notebooks/new?kernel_name=python3&trust=jupyter (::ffff:10.89.0.3)
  HTTPServerRequest(protocol='http', host='10.x.y.z:8000', method='GET', uri='/bi/v1/jupyter/hub/user/xOjpBbm9ueW1vdXM_.fd305be0b0afd6e36d2653e685e93d13/caload/notebooks/new?kernel_name=python3&trust=jupyter', version='HTTP/1.1', remote_ip='::ffff:10.89.0.3')

Any idea what could be wrong ?

In the mean time managed to start a terminated ca_jupyter_server_instance container. This is the logs:

podman logs ca_jupyter-xOjpBbm9ueW1vdXM_5F_2Efd305be0b0afd6e36d2653e685e93d13
Executing the command: jupyterhub-singleuser --ip=0.0.0.0
[I 2023-07-19 07:23:04.319 SingleUserNotebookApp notebookapp:1594] Authentication of /metrics is OFF, since other authentication is disabled.
[I 2023-07-19 07:23:05.029 SingleUserNotebookApp notebookapp:2076] Will shut down after 3600 seconds with no kernels or terminals.
[I 2023-07-19 07:23:05.029 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.4
[I 2023-07-19 07:23:05.039 SingleUserNotebookApp notebookapp:2303] Serving contents
[I 2023-07-19 07:23:05.040 SingleUserNotebookApp notebookapp:2303] Jupyter Notebook 6.4.10 is running at:
[I 2023-07-19 07:23:05.040 SingleUserNotebookApp notebookapp:2303] http://a529aa270e36:8888/bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.fd305be0b0afd6e36d2653e685e93d13/
[I 2023-07-19 07:23:05.040 SingleUserNotebookApp notebookapp:2304] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Thomas,

It's been a while since I've looked at the Jupyter Notebook installation but the most common issue that I encountered when I did was that the container cannot resolve host names. It seems containers don't inherit hosts files so make sure you can resolve host names like the Cognos servers without needing to update hosts files to do that.

Robert,

I've changed URLs to hostnames in Browser, CA and Jupyter configs, and verified name resolution.  Also, i added the rewrite/forward rule in CA apache for the jupyter folder

• That fixed the CORS error

Yet,  the rootless netns error persists. Along with that, the spawned notebook container usually crashes immediately into a semi-determined state. This in turn causes the podman-ca-jupyter container to refuse to use it.

I managed to open a new notebook in CA by the following procedure:

• stop and start the spawned notebook container (so its in the 'started' state)
• stop and start the ca_jupyter_hub container (so it does not remember the fault it ran into with the spawned notebook container last time round)

With this the startup log looks something like this:

[I 2023-07-25 10:58:47.662 JupyterHub dockerspawner:545] Found existing container 'ca_jupyter-xOjpBbm9ueW1vdXM_5F_2E7cee7c7773001cbda93bdd5373423248' (id: ac56b30)
[I 2023-07-25 10:58:47.662 JupyterHub dockerspawner:557] Starting container 'ca_jupyter-xOjpBbm9ueW1vdXM_5F_2E7cee7c7773001cbda93bdd5373423248' (id: ac56b30)
[W 2023-07-25 10:58:47.678 JupyterHub user:457] xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248's server specified its own API token that's not in the database
[I 2023-07-25 10:58:47.707 JupyterHub base:763] User xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248 took 0.119 seconds to start
[I 2023-07-25 10:58:47.707 JupyterHub proxy:242] Adding user xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248 to proxy /bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248/ => http://10.89.0.11:8888
10:58:47.708 - info: [ConfigProxy] Adding route /bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248 -> http://10.89.0.11:8888
10:58:47.708 - info: [ConfigProxy] 201 POST /api/routes/bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248 

The spawned container starts like this:

[I 2023-07-25 10:44:32.366 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.4
[I 2023-07-25 10:44:32.370 SingleUserNotebookApp notebookapp:2303] Serving contents
[I 2023-07-25 10:44:32.370 SingleUserNotebookApp notebookapp:2303] Jupyter Notebook 6.4.10 is running at:
[I 2023-07-25 10:44:32.370 SingleUserNotebookApp notebookapp:2303] http://ac56b30d4e50:8888/bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248/
[I 2023-07-25 10:44:32.370 SingleUserNotebookApp notebookapp:2304] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2023-07-25 10:58:47.706 SingleUserNotebookApp log:158] 302 GET /bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248/ -> /bi/v1/jupyter/user/xOjpBbm9ueW1vdXM_.7cee7c7773001cbda93bdd5373423248/caNotebookInfo? (@10.89.0.12) 1.02ms

and then serves request as should, including change of kernel from python to R and opening a different notebook in CA

The environment in the notebook includes e.g.

JUPYTERHUB_API_URL=http://ca_jupyter_hub:8080/bi/v1/jupyter/hub/api
NB_WSHOST=ws://ams4apw01ls.test.xyz:8000

So normal operation still not possible. Any other suggestios to follow ?