File and Object Storage

IBM Spectrum Scale Sharing Nothing Cluster performance tuning guide has been posted and please refer to link before you doing the below change.

All configuration options listed in this section are applicable only to Hadoop-like applications such as Hadoop and Spark:


Note: If the cluster is not dedicated for Hadoop workloads, take the default value for the above configurations.

For Hadoop-like workloads, one JVM process can open a lot of files. Therefore, tune the ulimit values:
vim /etc/security/limits.conf
# add the following lines at the end of /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536


kernel.pid_max
The default value is usually 32K. If you see the error “allocate memory”or “unable to create new native thread”, you could try to increase kernel.pid_max by adding “kernel.pid_max=99999” at the end of /etc/sysctl.conf and then “sysctl -p”.





#Softwaredefinedinfrastructure
#hadooptuning
#FPO
#Data-centricdesign
#Customerexperienceandengagement