Spectrum Conductor

New notebook features in IBM Spectrum Conductor 2.4.0

By Archive User posted Wed November 27, 2019 10:39 AM


Originally posted by: Jenna Lau-Caruso

image Version 2.4.0 brings several notebooks enhancements -- aimed to increase the flexibility of notebooks for all use cases. Read all about them in this blog!


Notebook impersonation
Starting in IBM Spectrum Conductor 2.4.0, notebooks now support impersonation. When enabled, all processes associated with the notebook, including both the notebook service and Spark workload, run as the notebook owner's operating system user.


Notebook impersonation is controlled at the instance group level: to enable impersonation for an instance group, select the Enable impersonation to have Spark applications run as the submission user option in the instance group security settings within the cluster management console:image

This setting enables impersonation for all Spark workload and notebook services which support user impersonation. Jupyter 5.4.0 notebooks, included in IBM Spectrum Conductor 2.4.0, supports impersonation.


For details about this new notebook security setting, see IBM Knowledge Center.


Notebook user environment variables
When creating a notebook, the instance group administrator can set environment variables, which apply to all notebooks in the instance group. In some cases, you may need to overwrite these values or provide additional environment variables which are only applicable to their your personal notebook. To add or modify an environment variable to a notebook that you own, select the new Configure button on the notebooks tab of the instance group within the cluster management console:


Upon clicking the button, the notebook configuration dialog opens:



For the Jupyter 5.4.0 notebook included with IBM Spectrum Conductor 2.4.0, you can use the JUPYTER_USER_SPARK_OPTS environment variable to set one or more Spark configuration parameters by specifying the --conf key=value notation. For example, the screenshot shows using this environment variable to set the spark.yarn.principal and spark.yarn.keytab parameters to enable Kerberos authentication for your notebook.


Check out IBM Knowledge Center for specific details on setting up notebook environment variables.


GPU management for notebooks

You can now specify the number of exclusive GPUs required by a notebook, thereby, preventing resource contention between notebooks. When your IBM Spectrum Conductor 2.4.0 cluster is configured to use GPUs, you can see the new Number of GPUs field in the notebook configuration dialog within the cluster management console when you enable a notebook for an instance group:


Once you provide a value for this field, all notebooks of this type will be allocated the requested number of GPUs in exclusive mode when the notebook starts. 


Read more about using this new GPU field when you enable your notebook for your instance group, in IBM Knowledge Center.


Notebook state control

When you start a notebook, it consumes resources on the cluster. To help better utilize cluster resources, we have introduced new ways to control notebooks:

  • Users with the Consumer User or Data Scientist role are now able to stop and start their own notebooks from the notebooks tab of an instance group. This enables users to stop their notebook when it is not in use. Check out IBM Knowledge Center for how to start or stop your notebook in an instance group.
  • By default, notebooks start when an instance group starts. To prevent notebooks from starting automatically with an instance group, cluster administrators can now configure the the new CONDUCTOR_SPARK_AUTOSTART_NOTEBOOKS_WITH_INSTANCE_GROUP parameter to OFF in the in ascd.conf configuration file. For clusters with many notebooks, this setting prevents all notebooks from claiming resources before they are really needed. For details about this new ascd.conf parameter, see IBM Knowledge Center.

Let us know what you think!

We hope that these new features help you to get the most out of IBM Spectrum Conductor 2.4.0 notebooks! Give them a try and let us know what you think in our Slack channel.