Originally posted by: LiorAronovich

To deploy an IBM Spectrum Conductor with Spark cluster as a long running parallel job within an IBM Spectrum LSF cluster, you can use the integration implementation and documentation provided on our IBM Cloud conductor-lsf sample page.

With this implementation, you can start, resize and terminate an IBM Spectrum Conductor with Spark cluster within an IBM Spectrum LSF cluster, by using LSF’s bsub, bresize, and bkill commands correspondingly.

You can use IBM Spectrum LSF’s bsub command to submit an IBM Spectrum Conductor with Spark cluster controller job. The IBM Spectrum Conductor with Spark cluster controller starts, controls and terminates the IBM Spectrum Conductor with Spark cluster, by using IBM Spectrum Conductor with Spark commands.

Specifically, the IBM Spectrum Conductor with Spark cluster controller obtains information from IBM Spectrum LSF, including the list of hosts allocated by IBM Spectrum LSF for the job, and controls the startup and termination of the IBM Spectrum Conductor with Spark cluster on the allocated hosts.

An IBM Spectrum Conductor with Spark cluster can be dynamically resized, namely extended or reduced with compute hosts dynamically, by using IBM Spectrum LSF’s bresize command. This activates an IBM Spectrum Conductor with Spark cluster resize procedure that uses IBM Spectrum Conductor with Spark commands to resize the cluster.

Specifically, the IBM Spectrum Conductor with Spark cluster resize procedure obtains information from IBM Spectrum LSF, including the list of hosts to add to or remove from the IBM Spectrum Conductor with Spark cluster. The resize procedure controls the addition or removal of the relevant hosts to or from the cluster, and communicates the updated cluster formation to the IBM Spectrum Conductor with Spark cluster controller.

An IBM Spectrum Conductor with Spark cluster can be terminated, by using IBM Spectrum LSF’s bkill command. This activates the termination procedure in the IBM Spectrum Conductor with Spark cluster controller, which terminates the cluster.

Hosts are acquired from IBM Spectrum LSF in an exclusive mode to form an IBM Spectrum Conductor with Spark cluster.

IBM Spectrum Conductor with Spark manages the resources within its cluster internally.

After deploying an IBM Spectrum Conductor with Spark cluster as a long running parallel job, the IBM Spectrum Conductor with Spark cluster can be accessed by using the cluster management console, CLI commands, and RESTful APIs.

In this architecture an IBM Spectrum LSF cluster can be used to host and share resources between batch workloads and IBM Spectrum Conductor with Spark workloads, including Apache Spark workloads and Deep Learning workloads.

This architecture leverages IBM Spectrum LSF's advanced capabilities of resource scheduling and allocation, while IBM Spectrum Conductor with Spark provides users with unique capabilities such as management of multiple Spark instances, fine-grained resource sharing, monitoring, and more. Furthermore, the IBM Spectrum Conductor with Spark cluster within an IBM Spectrum LSF cluster is elastic and can grow and shrink dynamically.

If you have not yet tried IBM Spectrum Conductor with Spark version 2.2.0, you can download an evaluation version here. If you have any questions about our product, let us know using our Slack channel!