When you previously mentioned "6 concurrent users per core", I think you picked it up from my CASE 2 article in which I discussed about batch jobs. So, this is your misunderstanding. For a POWER8 core with SMT-8 capability, I would figure a simple rule-of-thumb for 30 "active" interactive jobs per core. (6 active "batch" jobs per core).
And with the new total users of 400 that you indicated for your core bank workload, I would assume 50% active user jobs for peak workload period. This would mean you may need at most 200/30 = 7 cores. (But you also need to check on the batch workload dimension as well. If it needs more CPUs or not.) With an existing machine carrying the current workload, you have an advantage of using performance data to verify this rule of thumb. You do this by looking at PDI's Wait Overview chart and focus on CPU Queuing wait component. If this wait time does not appears to be too much (as I described in my CASE 2 article) you have no need to go for 7 cores.
Make sure you pay attention to Wait Overview chart during the night batch run duration and wait time in generic jobs that run night batch process as well. More day-time banking transactions can mean more batch process workload as well. If substantial CPU Queuing wait time comes up in either day-time interactive or night-time batch period, you need to add more core(s) to eliminate CPU Queuing time or reduce it to just a small amount. How many more core to add depends on the "worst" proportion between CPU Queuing wait against Dispatch CPU Time that you see in the chart. Do not look at just one day chart. Look at 3-4 days' chart of high/peak workload before making a conclusion.
Original Message:
Sent: Wed March 22, 2023 01:51 AM
From: Joel J.
Subject: Webfacing memory allocation
Hi Satid,
Its a Power 8 with 4 cores 64Gb memory and IBM i 7.4.
Only the webfacing server QQFWFSVR is multi threaded based on WRKACTJOB F11.
Temporary
Subsystem/Job User Number Type CPU % Threads Storage
QQF055B3D1 SUPERIT 373408 INT .0 1 15
QQF055B3D2 SUPERIT 373303 INT .0 1 15
QQFVTSVR QUSER 351037 BCI .0 1 8
QQFVTSVR QUSER 351038 BCI .0 1 8
QQFWFSVR QUSER 351036 BCI .0 68 18
I will collect the PDI charts so i can follow your guidance above. Again many thanks!
All the best,
Joel
------------------------------
[Joel]
Original Message:
Sent: Tue March 21, 2023 06:55 AM
From: Satid Singkorapoom
Subject: Webfacing memory allocation
Dear Joel
First I forgot to ask what model of IBM i server it is? How many CPU and memory and disk type (HDD or SSD)? IBM i version?
I then ask that you check if the job QQFVTSVR is multithreaded one or not. You do this by WRKACTJOB and press F11 until you se the column named "Thread". If there are multiple count of thread s for each or one of the jobs, you can check its actual memory need by the Java Memory by Job (and by Subsystem if one exists) chart at the future peak workload period. For now, you should allocate at least additional 2GB to *BASE pool for these jobs.
>>>> Is my understanding correct that we should then base the sizing / resource requirements of the production IBM i server based on the current CPU utilization of QUSRWRK, i.e. if the current subsystem utilization is at 15% average for 200 green screen sessions, and we are adding 200 webfacing jobs to the same subsystem, so we expect the subsystem CPU utilization to grow to 30%? <<<<
If the additional 200 jobs via Webfacing do the same type of work as the existing 200 jobs and the existing 200 jobs' workload will not be reduced as the result of the additional ones, then yes is a general answer. It can also be less than a total of 30% if the new 200 jobs will not carry similar amount of work done. Let's say 30% is the worst case scenario. But there should also be additional CPU power needed by Webfacing server jobs that do GUI conversion as well. (Those 3 jobs in QUSRWRK - these jobs may increase in number as well). You should keep your eyes on this new part of workload as well.
>>>> Similarly for memory, we should check the current allocated vs used heap memory given the 200 green screen workload, and we could theoretically use this currently used heap memory and extrapolate what will be the likely new value of the used heap memory based on additional 200 webfacing users? <<<<
I see there is no "current" heap memory because you said the current 200 interactive jobs are from 5250 sessions without Webfacing. I can give you a better advise on this if I can see the current memory pool size for interactive jobs and PDI chart on "Wait by Subsystem" for current peak workload day. If available, you should use the PDI Chart I mentioned in my CASE 5 article to see if the current interactive memory pool still has unused memory or not. If so, it will be a good news. Without these information, I suggest you add 20-30% more memory to interactive pool's current size. If none is available to add, you should produce PDI chart on current "Wait by Subsystem" of peak workload day and keep it to compare with this same chart after 200 more are added and focus on CPU Queuing and Disk Page Faults Time. If the latter does NOT increase a lot, no need to add more memory to interactive pool.
>>>> So if the rule of thumb is 6 concurrent users per core, and i have 200 users concurrently using the production IBM i via webfacing, this translates to 33 cores? At 8 Gb /core, translates to 264Gb memory? What should be our definition of "concurrent users"? Am asking because the 200 webfacing users might all be logged into the core banking application and doing Tellering transactions for example, at the same time across 50 branches for example, is that considered concurrent users? <<<<
My personal definition of concurrent users is the number of user who press enter at the same time (assuming each transaction runs for 3-4 seconds on average). If you have 200 users session, I would say it is not very likely all of them press enter at the same time at all time.
In my experience, "6 concurrent users per core" is quite a "very heavy-weight" workload indeed (assuming POWER7 and later CPU type). Although I had just a few past experience with customers who ran your core bank app (MISYS?), I'm still sure this rule is an overkill. Since you already have existing 200-user workload, a look at PDI chart on overall CPU utilization on peak workload period gives the best answer if your rule is reasonable or not.
>>>> Our larger banking clients have > 15K Tellers across 1000+ branches so i am being very careful on how to calculate the recommended sizing for any production IBM i related hardware sizing recommendations. How do i measure theconcurrent users our of 15K Tellers for the rule of thumb of 6 concurrent users per core? <<<<
If I were you, I would produce PDI charts to get to know your application performance profile from a few existing large workload customers, it will serve as an information base for future sizing. I suggest you consider paying for a service from IBM Lab Services team (it has a new name now like Technology Expert Services or so.) through IBM Philippines to engage with your team for a study that produce answer to all your sizing questions you asked here. It will be the best approach and you and your colleagues will get a chance to learn from my ex-colleagues experts. I can also do this for you as an independent consultant if I have time and you allow remote work.
One last advise. It is advisable that you convince the customer to use IBM Lab Services team on overall performance health assessment BEFORE adding these additional 200 users. In my experience, I was involved in several performance problem cases in which the customer increased the workload substantially and encountered serious overall performance degradation to the dismay of all. Many customers already had certain performance problem before the workload increase BUT they were not aware because the overall performance was still acceptable BUT the problem was already near upper performance limit of the system. When additional workload was added, it became OVERWHELMING and all were in trouble of various manners. An overall performance assessment will establish the base profile which helps them address the issues detected before the new workload is added and trouble is prevented.
If you desire we can discuss this over a web meeting. Let me know at my e-mail <my first name>119 at <Google Mail domain name>
------------------------------
Education is not the learning of facts but the training of the mind to think. -- Albert Einstein.
------------------------------
Satid S.
Original Message:
Sent: Tue March 21, 2023 03:24 AM
From: Joel J.
Subject: Webfacing memory allocation
Thank you Satid for your response.
When i do STRTCPSVR *WEBFACING, 3 jobs are started in QSYSWRK subsystem:
Subsystem/Job User Type CPU % Function Status
QQFVTSVR QUSER BCI .0 PGM-QQFVTSVR DEQW
QQFVTSVR QUSER BCI .0 PGM-QQFVTSVR DEQW
QQFWFSVR QUSER BCI .0 PGM-QQFWFSVR TIMW
Then when i access the Webfacing menu options via a browser, a QQF* job in QINTER subsystem is created for each webfacing session that i open from the browser.
Current
Subsystem/Job User Type CPU % Function Status
QQF06544D0 JAWILIJ1 INT .2 PGM-QQFINVOKER DSPW
QQF06544D1 JAWILIJ1 INT .3 PGM-QQFINVOKER DSPW
So i expect 200 additional QQF* interactive jobs in QUSRWRK subsystem if all my users logged in concurrently during business hours.
The same users will still use some of the core banking system menu options from the green screen, so in effect we will now have 200 green screen interactive jobs + additional 200 QQF* webfacing jobs in QINTER subsystem.
Is my understanding correct that we should then base the sizing / resource requirements of the production IBM i server based on the current CPU utilization of QUSRWRK, i.e. if the current subsystem utilization is at 15% average for 200 green screen sessions, and we are adding 200 webfacing jobs to the same subsystem, so we expect the subsystem CPU utilization to grow to 30%?
Similarly for memory, we should check the current allocated vs used heap memory given the 200 green screen workload, and we could theoretically use this currently used heap memory and extrapolate what will be the likely new value of the used heap memory based on additional 200 webfacing users?
The "evidence of absence" in the PDI performance article implies that we gather the data at peak hours, so we need to have those 200 webfacing users log in and do their normal workload also during business hours, then we can do the allocated vs used heap memory analysis from the performance data gathered. Am thinking, this will already be after the fact i.e. if the evidence of absence shows used heap memory is > allocated memory, we will need to advise our client to increase their memory in production. But is there a way to project this beforehand, so we can advise our client if they need to prepare for an increase of memory and cpu in production IBM i partition before they implement Webfacing for the 200 users?
So if the rule of thumb is 6 concurrent users per core, and i have 200 users concurrently using the production IBM i via webfacing, this translates to 33 cores? At 8 Gb /core, translates to 264Gb memory? What should be our definition of "concurrent users"? Am asking because the 200 webfacing users might all be logged into the core banking application and doing Tellering transactions for example, at the same time across 50 branches for example, is that considered concurrent users?
Our larger banking clients have > 15K Tellers across 1000+ branches so i am being very careful on how to calculate the recommended sizing for any production IBM i related hardware sizing recommendations. How do i measure theconcurrent users our of 15K Tellers for the rule of thumb of 6 concurrent users per core?
My apologies in advance Satid, i have so many more questions than answers at this point and i am going through all the articles you mentioned above to gain more knowledge on Performance tuning as well as proper sizing for our core banking application. I appreciate your help on this and all the articles you have provided so far, its very helpful for me to better understand how to do a proper sizing of our application from an IBM business partner perspective.
------------------------------
[Joel]
Original Message:
Sent: Mon March 20, 2023 05:23 AM
From: Satid Singkorapoom
Subject: Webfacing memory allocation
One last point, for Java jobs with A LOT OF threads (as I suspect Webfacing server part is), you may need to monitor CPU Queuing Wait to make sure it does not exist too much. PDI charts in Wait category is useful here - Wait Overview, Wait by Subsystem, Wait by Generic Job or Task. This article explains my point : https://www.itjungle.com/2022/03/21/guru-ibm-i-experience-sharing-case-2-dealing-with-cpu-queuing-wait-time/
------------------------------
Education is not the learning of facts but the training of the mind to think. -- Albert Einstein.
------------------------------
Satid S.
Original Message:
Sent: Mon March 20, 2023 12:31 AM
From: Joel J.
Subject: Webfacing memory allocation
We have a banking application developed in RPGLE deployed on IBM i OS. A number of years back we have started using IBM Webfacing feature of IBM HATS 9.6 for accessing our banking application screens via browser.
Does IBM have document/article/redbook baseline sizing recommendations for Memory, CPU and Disk for an IBM i partition, if for example we have 200 users of 5250 to access IBM i, and those 200 users will be given access to the same set of banking application menu options via HATS with Webfacing?
What additional workload will those 200 Webfacing users impose on the Power hardware where we run IBM i + our banking application?
How much additional memory and CPU allocation should we prepare for our IBM i partition if we are to add the Webfacing workload of 200 users to the IBM i partition? Is the Peak temporary storage used attribute of the job run attributes of a QQFWF* IBM i webfacing job a good starting point for the baseline memory needed for each webfacing job? For the CPU utilization, is there a metric we can use as a baseline?
Many thanks!
Joel
------------------------------
[Joel]
------------------------------