AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.

 View Only

  • 1.  Newbie - performance problem with LPARs

    Posted Tue October 08, 2013 03:14 PM

    Originally posted by: margeson


    Running into performance issues with WAS application servers on two of LPAR's or like configuration having slow response time on webpages during high user activity.

    Please forgive me I'm new to AIX and most my expertise is in the Linux space.

    Thanks for your help!!

    Here's the run down: 

    The problem appears to be CPU related just don't know where... maybe entitlement??? There is 0 paging going on according to vmstat.

    The run queue is .5625 based from 9 running threads and 16 total threads according to vmstat.

    There is 0 waits and idle at its lowest is 26%. user processes 72% and 2% system.  

    Here is output from sar:

    System configuration: lcpu=16 ent=4.00 mode=Capped 
     
     cpu    %usr    %sys    %wio   %idle   physc   %entc
    0       88       5       0       7    0.33     8.3
              1       83       2       0      15    0.26     6.5
              2       63       1       0      35    0.14     3.6
              3       63       1       0      36    0.14     3.6
              4       92       3       0       5    0.37     9.2
              5       83       2       0      15    0.28     6.9
              6       59       2       0      39    0.14     3.4
              7       58       1       0      41    0.14     3.4
              8       91       3       0       6    0.35     8.7
              9       83       2       0      15    0.27     6.7
              10       64       2       0      35    0.14     3.6
              11       59       1       0      40    0.13     3.3
              12       91       3       0       6    0.32     8.0
              13       87       2       0      11    0.28     6.9
              14       58       1       0      41    0.13     3.2
              15       57       2       0      41    0.12     3.1
              U        -       -       0      12    0.47    11.8

    Here is the output from lpartstat -i

    Node Name                                  : node1
    Partition Name                             : node1
    Partition Number                           : 4
    Type                                       : Shared-SMT-4
    Mode                                       : Capped
    Entitled Capacity                          : 4.00
    Partition Group-ID                         : 32772
    Shared Pool ID                             : 0
    Online Virtual CPUs                        : 4
    Maximum Virtual CPUs                       : 8
    Minimum Virtual CPUs                       : 2
    Online Memory                              : 16384 MB
    Maximum Memory                             : 24576 MB
    Minimum Memory                             : 8192 MB
    Variable Capacity Weight                   : 0
    Minimum Capacity                           : 2.00
    Maximum Capacity                           : 8.00
    Capacity Increment                         : 0.01
    Maximum Physical CPUs in system            : 64
    Active Physical CPUs in system             : 18
    Active CPUs in Pool                        : 18
    Shared Physical CPUs in system             : 18
    Maximum Capacity of Pool                   : 1800
    Entitled Capacity of Pool                  : 1400
    Unallocated Capacity                       : 0.00
    Physical CPU Percentage                    : 100.00%
    Unallocated Weight                         : 0
    Memory Mode                                : Dedicated
    Total I/O Memory Entitlement               : -
    Variable Memory Capacity Weight            : -
    Memory Pool ID                             : -
    Physical Memory in the Pool                : -
    Hypervisor Page Size                       : -
    Unallocated Variable Memory Capacity Weight: -
    Unallocated I/O Memory entitlement         : -
    Memory Group ID of LPAR                    : -
    Desired Virtual CPUs                       : 4
    Desired Memory                             : 16384 MB
    Desired Variable Capacity Weight           : 0
    Desired Capacity                           : 4.00
    Target Memory Expansion Factor             : -
    Target Memory Expansion Size               : -
    Power Saving Mode                          : Disabled

    Here is the output from smtctl

    This system is SMT capable.

    This system supports up to 4 SMT threads per processor.
    SMT is currently enabled.
    SMT boot mode is not set.
    SMT threads are bound to the same virtual processor.
     
    proc0 has 4 SMT threads.
    Bind processor 0 is bound with proc0
    Bind processor 1 is bound with proc0
    Bind processor 2 is bound with proc0
    Bind processor 3 is bound with proc0
     
    proc4 has 4 SMT threads.
    Bind processor 4 is bound with proc4
    Bind processor 5 is bound with proc4
    Bind processor 6 is bound with proc4
    Bind processor 7 is bound with proc4
     
    proc8 has 4 SMT threads.
    Bind processor 8 is bound with proc8
    Bind processor 9 is bound with proc8
    Bind processor 10 is bound with proc8
    Bind processor 11 is bound with proc8
     
    proc12 has 4 SMT threads.
    Bind processor 12 is bound with proc12
    Bind processor 13 is bound with proc12
    Bind processor 14 is bound with proc12
    Bind processor 15 is bound with proc12
     

     

     



  • 2.  Re: Newbie - performance problem with LPARs

    Posted Tue October 08, 2013 03:51 PM

    Originally posted by: Wouter Liefting


    At a first glance I would indeed say that you don't have enough CPU resources. You might want to increase the entitlement (and you will need to increase the number of virtual processors at the same time) and see what happens. With your current config you can go as high as 8.0 entitlement / 8 virtual CPUs via a DLPAR operation (so without a reboot).

    BUT

    At the moment you are running a 4.0 entitlement, capped, and that means that your WAS licence will be based on 4 CPUs. As soon as you increase the entitlement, you may find your licence cost increasing as well.

    I'm also curious about the "9 running threads". I'm no WAS expert, but is this the number of WAS threads? Is that in any way configurable? Nine seems to be an odd number on a 4 core, 4-way SMT Power7 system. You would either use 16 (to maximize the use of all hardware threads), or 8 or 4 (to benefit the most from virtual processor folding - which increases the per-thread throughput).

    For further investigation, the PowerVM Virtualization Performance Advisor is probably a good start:

    https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/PowerVM%20Virtualization%20Performance%20Advisor

    You may also get some great info from the Java Performance Advisor:

    https://www.ibm.com/developerworks/community/wikis/home?lang=en#/wiki/Power%20Systems/page/Java%20Performance%20Advisor%20%28JPA%29



  • 3.  Re: Newbie - performance problem with LPARs

    Posted Tue October 08, 2013 05:38 PM

    Originally posted by: margeson


    Excellent feedback, thank you!

    Regarding the running threads of 9. Based on vmstat 9 of the 16 total threads were in use over a sampling period.

    I'm researching our current WAS entitlment.

    What I'm thinking is since 7 threads are free is the following: increase virtual CPU from 4 to 8, reduce SMT from 4 to 2. The reason I'm thinking of reducing SMT is I'm only seeing 9 total threads used. Rather than leaving it at 4 and bumping to 8 virtual CPUs as there would be 24 total threads; likely the logical CPU's would become wasted. I also read that reducing the SMT threads would increase the CPU horsepower.

    What are your thoughts?

    I'm also looking into configuring WAS to leverage additional threads, maybe this would give us the horsepower to handle the requests...???

    Thanks again,
    Rob Margeson
    HP

     



  • 4.  Re: Newbie - performance problem with LPARs

    Posted Tue October 08, 2013 06:08 PM

    Originally posted by: Wouter Liefting


    You've got capped processors. In that case it doesn't make sense to make the number of virtual processors larger than the entitlement, other than rounding the entitlement up to the nearest whole number (e.g. an entitlement of 2.5 requires 3 virtual processors). If you keep the entitlement at 4.0 but increase the VP to eight, you still get the same amount of CPU cycles per 10ms timeslice, but these cycles may now be spread over eight physical instead of four physical processors. Which leads to cache invalidation and related problems, and may actually hurt performance.

    You could make your processors uncapped, which will make it possible to use more than your entitlement, but the first consequence is that your licences are no longer based on entitlement but on the maximum number of processors that you can conceivably use. And performance tuning of uncapped LPARs (entitlement and virtual processors) is considerably harder - a lot depends on your workloads (how high are the peaks vs. the troughs) and how much spare CPU capacity is available in the pool. When I teach the PowerVM performance course we spend at least half a day on this topic.

    You don't have to reduce the SMT mode manually. AIX uses "virtual processor folding". If you don't have sufficient tasks to keep all 16 hardware threads busy, AIX will automatically fold to SMT-2 or even ST mode, which means that the *per thread* performance will increase. Whether that leads to an overall throughput increase or decrease depends heavily on the application. You can see this beautifully if you use nmon. Press "c" to see the CPU display. You'll see that the busy threads are deliberately spread out over the four physical processors: AIX will load up logical CPUs 0, 4, 8 and 12 (which is physical CPU 0, 1, 2, 3), before it starts loading logical CPU 1, 5, 9 and 13. And so forth.