The release of IBM Spectrum Conductor 2.5.0 includes many enhancements to the existing Jupyter service package. In addition, we created a separate updated Jupyter service package that takes advantage of new component-specific features.
Among the enhancements that both service packages receive are those enhancements that are controlled by the following new environment variables:
CONDA_ENV_NAME
If you use notebooks, you can specify the individual notebook configuration to use a different conda environment other than the default value.
JEG_STARTUP_TIMEOUT
If you use notebooks, you can specify this environment variable to indicate the amount of time (in seconds) for the Jupyter start script to wait after initiating the Jupyter Enterprise Gateway (JEG) process to determine whether it succeeded or failed to start, as the start-up time might be slower on some hosts. If this value is not specified, the default time of 5 seconds is set.
SITE_PACKAGE_PATH
If you use notebooks, you can speed up the process of the Jupyter start script by specifying the directory path to the site-packages of the conda environment instead of having the start script run a find command to locate the path. This environment variable is useful on file systems where the find command would take a long time to process, such as for large conda environments or slow processing file systems. You can save time when the path to the site-packages directory is known by setting this SITE_PACKAGE_PATH environment variable.
SITE_PACKAGE_FIND_DEPTH
If you use notebooks, you can specify this environment variable (instead of the previous environment variable) to speed up the process of the Jupyter start script by indicating the maximum depth for the find command when you search for the site-packages directory of the conda environment. This environment variable is useful on file systems where the find command would otherwise take a long time to process, such as for large conda environments or slow processing file systems. You can save time when the path to the site-packages directory is not exactly known by setting this SITE_PACKAGE_FIND_DEPTH environment variable.
Additionally, these parameters are now referenced in the log output of the Jupyter service logs to give you ideas of performance enhancements when you notice large gaps in log entry timestamps.
JUPYTER_ENV_ALLOWLIST
If you use notebooks, you can specify a list of environment variables that you want to pass from the kernel launching process into the kernel itself. The syntax is a comma-delimited list of environment variable names that are wrapped in single quotation marks, for example: ‘VAR1’,’VAR2’. The variables can be either specified by you, or the environment variables are automatically set when the user logs in to the operating system (OS). When specified, the user has access to these environment variables in Jupyter GUI cells, otherwise available only in the Jupyter terminals.
The new Jupyter 6.0.0 service package
IBM Spectrum Conductor 2.5.0 includes a new Jupyter service package with a version-level set at 6.0.0. Similar to the 5.4.0 package, this version is the version of the service package itself, and is independent of the conda Jupyter notebook package that is used to run the Jupyter notebook. The new Jupyter service package supports the notebook package version 6.0.0 and later, and supports jupyter_enterprise_gateway version 2.1.1 and later.
This new notebook service package does not include any conda or pip packages, so you have no risk of interference with your conda environments. The notebook service package deployment notifies you when you do not have the required packages in your conda environment.
The new service package introduces some new enhancements such as adding the kernel copy logic into the Jupyter start script. These kernels refer to the IBM Spectrum Conductor kernels that were introduced to be used inside the Jupyter notebooks. Previously, the kernels were only copied at deployment time. This enhancement means that even if notebooks are started in conda environments on hosts that have not previously received the deployment, for example when you use the CONDA_ENV_NAME environment variable to use a non-default environment, the IBM Spectrum Conductor kernels are available to your notebook in that conda environment. Additionally, the new service package detects if JupyterLab is installed in the conda environment, and automatically generates the Jupyter notebook GUI link to the JupyterLab URL instead of the default Jupyter URL. The user notebook services are also changed so that the services no longer require the Application Service Controller daemon (ascd), REST, or notebook-master services to be STARTED and stay STARTED. If the ascd, REST, or notebook master service is restarted or migrated to a new host, the notebooks for the user can stay running.
The Jupyter 6.0.0 service package also uses the new features that are enabled by the introduction of the concept of components in IBM Spectrum Conductor 2.5.0 as can be seen from the /components/Jupyter/6.0.0 REST API. The Jupyter 6.0.0 component contains information about its conda and pip package dependencies for each system architecture. In addition, the Jupyter 6.0.0 component includes the list of sample conda environment YAML files that are included in IBM Spectrum Conductor 2.5.0.
In conclusion, use these new features to deploy notebooks in your instance groups with improved user and administrator experiences. See a better time-to-value by using the new sample conda environment files to get your instance groups up and running quickly from a fresh installation.
Give it a try and tell us what you think by downloading IBM Spectrum Conductor 2.5.0 on Passport Advantage or the evaluation version! To get the most out of Jupyter, apply the sc-2.5-build600258 interim fix, which contains several enhancements on top of the base integration. We hope you are as excited as we are about this new release!
Log in to this community page to comment on this blog post. We look forward to hearing from you on the new features, and what you would like to see in future releases.
#SpectrumComputingGroup