z/OSMF REST data set and file services is one of the most popular z/OSMF REST services. You can drive it to operate data sets or USS files basically from any program languages which support HTTP. There are many customers now depend on z/OSMF REST data set and file services for critical z/OS management tasks. Therefore, we now often heard about questions related with scalability of z/OSMF REST data set and file services. This question is sometimes not easy because REST data set and file services depends on a few other z/OS components like CEA, MessageQueue, etc. This blog summarizes the typical aspects that could tune the scalability of z/OSMF REST data set and file service. According to our lab result (your result might vary), below tuning aspects could allow a single z/OSMF server running on a z/OS image which is running on top of z/VM with 2 virtual CPUs to successfully handle 500 requests per 1 minute. Like many other REST services, you can setup second server to implement workload balancing to further increase scalability of z/OSMF REST data set and file service. But that’s not covered in this blog.
Every aspect below includes “Error symptom” and “Suggestion”. You can utilize below information in two ways:
1. Leverage below information only when you hit some scalability issue.
Depends on your environment setup and capacity, the number of concurrent requests z/OSMF REST data set and file service can handle varies. You might not hit any of below errors for quite a long time. Therefore, you could use below information as a trouble shooting material. When you hit scalability issue, you could search keywords in this document to see if it is already covered by this tuning guide.
2. Review all aspects below and pickup whatever suggestion you feel proper to follow.
Typically, if you are still exploring the REST services, this approach could help you to start with a fine tuned z/OSMF REST data set and file services with good scalability.
Scalability depends on a lot of things in general. Therefore, values suggested or mentioned below are just for your reference. You might need customized value based on your environment.
1. Setup a proper limitation for Message Queue which is used undercover by REST data set and file servicesError symptom：A specific error will be returned in the response body to indicate that the “IPCMSGQYBETS is less than xxxx”
Suggestion:It is recommended that you specify an IPCMSGQBYTES value of at least 20971520 (20 M) in BPXPRMxx. To set this value dynamically, you can enter the following operator command: SETOMVS IPCMSGQBYTES=20971520
Here is more information about the above command:Specifies the maximum number of bytes in a single message queue. Value range: nnnnn is a decimal value from 0 to 2147483647. The high end of this range is not obtainable due to storage constraints. The actual maximum range varies due to storage allocation and system usage.Default: 2147483647 (2 G).Use the SETOMVS or SET OMVS command to change the value of IPCMSGQBYTES dynamically.
2. Setup a proper value for max TSO address spaces CEA can create for z/OSMF
Error symptom:Error messages will be generated in the z/OSMF log IZUG0.log with the reason code 1000 and 1005 in hexadecimalYou can get the detail info for the 1000 and 1005 in Knowledge Center as below:Reason code 1000: CEATSOMSGQSERVICEFAILED which means "Message queue processing was unsuccessful"Reason code 1005: CEATSOStartFAILED which means "CeaTsoReq failed to create an address space"
Suggestion:
z/OSMF calls z/OS CEA component to create TSO address space for handling data set operations. Typically, this error is caused by the max number of TSO address spaces CEA can create is reached. By default, it’s 50. You can update it as below:
1. Check the number of TSO address space that has already been created by below command
/F CEA,DIAG,SESSTABLE
2: Check the current setup of CEA with below command
/F cea,d,parms
3: Create a parmlib with tailored option “MAXSESSIONS” and issue below command to enable the new parmlib (For instance, the new parmlib is CEAPRMHL)
/F cea,cea=(01,HL)
E.g. To support 500 concurrent requests, you might need to setup “MAXSESSIONS” to a value greater than 500.
You can get more info for the max session setup from knowledge center:
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.ieac100/cea-reason-codes-section.htm
3. Add auxiliary storage for paging
Error symptom:
You might see message “*IRA200E AUXILIARY STORAGE SHORTAGE” in the syslog
Suggestion:
Use the command $d spool to check current utilization of the spool files. Add a page dataset if needed.
4. Update MAXUSER value of IEASYSxx
Error symptom:
An error message “exceeding the xxx users” will be generated in the syslog
Suggestion:
Update the MAXUSER=nnnnnn in the IEASYSxx
To support 500 concurrent requests, you might need update the MAXUSER from xxx to a proper value which we suggest to be greater than 500.
You can get more info for the MAXUSER in IEASYxx setup from knowledge center:
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.ieae200/ieae200312.htm
5. Increase max threads can be created by OMVS
Error symptom:
There will be some error in syslog indicates that the max threads has been reached.
Suggestion:
Increase the max threads can be generated by OMVS. You can use below commands to checkout if there is OMVS thread shortage risk:
1: D OMVS,A=ALL to check ZOSMFSVR PID
2: D OMVS,L,PID=xxx to check value of the HIGHWATER USAGE
To support 500 concurrent requests, update the MAXTHREADS to a proper value at least greater than 500, such as 600 with below command:
/setomvs MAXTHREADS =600
MAXTHREADS 154 156 600
You can also set the MAXTHREADTASKS in BPXPRMxx. Here is more information in Knowledge Center:
https://www.ibm.com/support/knowledgecenter/en/SSBDYH_3.1.1/com.ibm.zexpl.config.hostconfigref.doc/topics/hostinfo80.html?cp=SS7EY3_3.1.0
6. Increase the memory limits of z/OSMF server
Error symptom:
You might see “CEE3250C The system or user abend SDC2 R=4A001620 was issued. From entry point ntv_createJoinWorkUnit at compile unit offset +000000003BC592B2 at entry offset +00000000000CEDE2 at adder” in the job log of z/OSMF server.
Suggestion:
Increase the memory limit of z/OSMF server to a proper value. By default, it’s 4G. E.g. if you would like to increase it to 8G, you could update z/OSMF server procedure (IZUSVR1 by default) like below:
//IZUMEM=8G /* Server memlimit */
7. Tune the performance goal of z/OSMF server in WLM
Error symptom:
You might see “"rc":16,"reason":1141,"category":2,"message":"login: _msg_rcv_timed() error on” in the response body of REST API.
Suggestion:
Use WLM to tune the performance goal of z/OSMF server. Here is the link where you can get detailed guide by searching “WLM” keyword:
https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102808
8. Add auxiliary storage space
Error symptom:
“*IRA206I ZFS ASID 000E FRAMES 0000009858 SLOTS 0000108436 % OF AUX “ will be generated in the syslog
Suggestion:
You can use system command “d asm,page=all” to check current utilization for auxiliary storage, if there is shortage risk, you can use system command like ”PA SYS1.PEVxxx.LOCAL.PAGE” to add auxiliary storage dynamically.
9. Increase the max number of Message Queue
Error symptom:
An error message with time out for the request in the response body will be returned
Suggestion:
You can check the current max number of message queue by issuing USS command “IPCS -qc”. If you need to increase the number to a proper value such as 600 in order to handle 500 concurrent requests, you could issue below system command to increase the max number of message queue:
/setomvs IPCMSGNIDS =600
10. Increase JVM heap size used by z/OSMF
Error symptom:
Message “JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2020/04/25 11:00:59 - please wait…” will be generated in the job log of z/OSMF server address space.
Suggestion:
Increase the heap size of JVM to 2G (by default, z/OSMF is using 1G) for z/OSMF. Please be noticed that this step should be done together with step 6. Below steps are reference about how to update JVM options for z/OSMF:
1. Create a file named local_override.cfg under <zOSMF data directory>/configuration/
2. Add below line in local_override.cfg:
JVM_OPTIONS='-Xms128M\n-Xmx2048M’
3. Restart z/OSMF server
References:
1. z/OSMF Marketplace
https://www.ibm.com/us-en/marketplace/zos-management-facility
2. z/OSMF Knowledge Center
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.izua300/IZUHPINFO_OverviewMain.htm
3. The optional enhanced look and feel of z/OSMF on YouTube
https://www.youtube.com/watch?v=9bI0G2ZKBds
4. z/OSMF One Stop Hub
https://www.ibm.com/developerworks/community/blogs/e0c474f8-3aad-4f01-8bca-f2c12b576ac9/entry/z_OSMF_One_Stop_Hub?lang=en