Skip main navigation (Press Enter).
Log in
Toggle navigation
Community
Topic Groups
Champions
Meet the Champions
Program overview
Rising Champions
IBM Champions group
User Groups
Find your User Group
Program overview
Events
TechXchange Conference
Community events
All IBM events
Participate
IBM Community Hub
Welcome Corner
Blogging Guidelines
Member directory
Community leaders
Resources
Badge Program
IBM Support Customer Day
Log in
File and Object Storage
×
File and Object Storage
Software-defined storage for building a global AI, HPC and analytics data platform
Group Home
Threads
188
Blogs
574
Upcoming Events
0
Library
21
Members
3.1K
View Only
Share
Share on LinkedIn
Share on X
Share on Facebook
Back to Blog List
How to configure and performance tuning Spark workloads on IBM Spectrum Scale Sharing Nothing Cluster
By
Archive User
posted
11/27/17 02:17 AM
Like
IBM Spectrum Scale Sharing Nothing Cluster performance tuning guide has been posted and please refer to
link
before you doing the below change.
Here is the tuning steps.
Step1: Configure spark.shuffle.file.buffer
By default, this must be configured on
$SPARK_HOME/conf/spark-defaults.conf
.
To optimize Spark workloads on an IBM Spectrum Scale filesystem, the key tuning value to set is the 'spark.shuffle.file.buffer' configuration option used by Spark (defined in a spark config file) which must be set to match the block size of the IBM Spectrum Scale filesystem being used.
The user can query the size of the blocksize for an IBM Spectrum Scale filesystem by running: 'mmlsfs
#cognitivecomputing
#Real-timeanalytics
#Softwaredefinedstorage
#Customerexperienceandengagement
#sparkworkloadtuning
#Data-centricdesign
#Workloadandresourceoptimization
#FPO
0 comments
0 views
Permalink
Copy
https://community.ibm.com/community/user/blogs/archive-user/2017/11/27/how-to-configure-and-performance-tuning-spark-workloads-on-ibm-spectrum-scale-sharing-nothing-cluster
Copyright � 2026 IBM Community. All rights reserved.
Powered by Higher Logic