High Performance Computing

High Performance Computing Group

Connect with HPC subject matter experts and discuss how hybrid cloud HPC Solutions from IBM meet today's business needs.

 View Only

Shared file system installation

By Archive User posted Mon August 22, 2016 10:48 AM

  

Originally posted by: Jenna Lau-Caruso


imageWith IBM Spectrum Conductor with Spark v2.1.0.1, you can now install your cluster on a shared file system such as IBM Spectrum Scale. By installing your cluster on a shared file system, you can reduce installation and deployment time and contain storage costs and resources used by system services -- all while taking full advantage of the scalability, security, high availability, recovery, and management capabilities provided by the file system.

With IBM Spectrum Conductor with Spark installed on a shared file system, you gain the following efficiencies:

  • Save installation time and reduce installation footprint by installing IBM Spectrum Conductor with Spark once for the entire cluster.
  • Save deployment time when creating a Spark instance group by deploying Spark and notebook packages just once for the entire Spark instance group.
  • Increase application performance by disabling the Spark shuffle service. By default, the Spark shuffle service is disabled on shared file system installations, as data written to a shared file system does not need to be shuffled between hosts. You can optionally enable the shuffle service if configuring your Spark instance group to use spark.local.dir on local disk.
  • Reduce the number of SparkCleanup service instances (for periodic cleanup of Spark instance groups) to 1. The instance runs on a single management host, instead of every host in the cluster. 

To install IBM Spectrum Conductor with Spark v2.1.0.1 on a shared file system, simply set the SHARED_FS_INSTALL=Y environment variable before running the installation. You only need to run the installer once and all hosts in the cluster will share the same installation. Note that shared file system installation applies only to standalone software packages.

Take the next step: Download v2.1.0.1 from IBM Fix Central and get started with your installation. For complete details, see Installing to a shared file system in the IBM Knowledge Center.

Comment or question? Talk to us in our forum.


#SpectrumComputingGroup
6 comments
1 view

Permalink

https://community.ibm.com/community/user/blogs/archive-user/2020/05/28/shared-file-system-installation

Comments

Archive User
Wed September 14, 2016 03:38 PM

Originally posted by: Jenna Lau-Caruso


The shared file system installation enhancement is currently for Conductor with Spark only. If you require Conductor running in the same cluster, you will need to follow the regular installation process for local file system.

Archive User
Wed September 14, 2016 04:33 AM

Originally posted by: hideno


Hi Jenna, Thank you! Very nice enhancement, especially for large cluster. Let me ask one more question; Background; I'd like to install both Spectrum Conductor and Spectrum Conductor with Spark using shared file system. Then I should install Conductor with Spark first, and overlay Conductor follows, according to the manual. http://www.ibm.com/support/knowledgecenter/SS4H63_2.1.0/install/add_product.html How can it be done for Spectrum Conductor? Just the same to Conductor with Spark?

Archive User
Fri September 09, 2016 08:51 AM

Originally posted by: Jenna Lau-Caruso


Hi Hidenori, There is no extra installation binaries required for compute hosts. You are correct that you will need to join compute hosts to the cluster using the egoconfig​ command. Please make sure that you have completed step 9 from the installation instructions. The profile with .comp suffix is what you will want to source on compute host environments. You will also need to create a unique directory for each host linked to the Elasticsearch harvest location. This is step 10 from the knowledge center instructions. Please let us know if you have any further questions!

Archive User
Fri September 09, 2016 12:56 AM

Originally posted by: hideno


Hi, Thanks for the link. What I meant to say in more detail is; 1. Install to management host. This is exactly in the link. 2. Install to compute hosts. I didn't figure out how to. If the install was done in step 1 into shared file system directory (i.e. --prefix shared_file_system_dir), no further installation is required for compute hosts and only additional work is joining to the cluster (egoconfig join compute_host_name)? Or install runtime binary to shared file system is not supported?

Archive User
Fri September 02, 2016 05:00 PM

Originally posted by: Sheryl_M


Hello Hidenori, thank you for the feedback. The link to the instructions is right at the end of the blog. Here's the link in any case: https://www.ibm.com/support/knowledgecenter/SSZU2E_2.1.0/installing/install_shared_fs.html Let us know how it goes!

Archive User
Thu September 01, 2016 09:04 PM

Originally posted by: hideno


Hi, this is a great function and one of them which we're looking for! >You only need to run the installer once and all hosts in the cluster will share the same installation. Are there any instructions to install to compute hosts?