High Performance Computing

Originally posted by: Jenna Lau-Caruso

For custom applications to run, you often need to copy application JAR files, or other files, to each host in an instance group.

Now, in IBM Spectrum Conductor 2.4.0, we've made this process easier than ever: no more writing service packages!

That's right, we've eliminated the overhead of writing installation scripts and creating a service package to deploy files to an instance group. Instead, simply copy a single file (or single archive of a bunch of files) to an instance group:

1. From the Packages tab for an instance group within the cluster management console, click the new Create Single-File Packages button to directly upload the files you want to copy.
2. Specify where you want to copy the file to, and what the permissions to use.
3. The system handles handle the rest!

The result is it creates the service package and deploys it to all hosts in the instance group for you. For archive files, you can also choose to let the deployment extract the archive at its destination.

And there's more: need those JAR files in your class path? No problem: just select the Append the destination path checkbox to automatically add the destination path of your files to the spark.driver.extraClassPath and spark.executor.extraClassPath parameters in your Spark configuration. Done. It's that easy!

For more complex packages, which require a real installation, you can continue to write service packages with custom installation scripts. With this version of IBM Spectrum Conductor, you can now use the $DEPLOY_HOME and $SPARK_HOME environment variables in your service package scripts to easily install your package relative to each instance group deployment.

Check out this streamlined package deployment process in Adding dependent packages for an instance group in IBM Knowledge Center, then give it a try and let us know what you think in our Slack channel!