AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.

 View Only
Back to discussions
Expand all | Collapse all

AIX performance slowness check

  • 1.  AIX performance slowness check

    Posted 22 days ago

    Hi Team,

    How can we do the performance/slowness check of AIX LPAR? If someone facing slowness issue what parameter and command we can look into it. Do we have any general documents/commands to check in terms of memory/CPU/disk/network/error? I know there is no definitive answer but wanted to know if some basic commands we can start from and from where we can move further. Thank you in advance. 



    ------------------------------
    Manoj Kumar
    ------------------------------


  • 2.  RE: AIX performance slowness check

    Posted 22 days ago
    like Rit De reacted to your message:



    Original Message


  • 3.  RE: AIX performance slowness check

    Posted 22 days ago
    Edited by Alexander Pettitt 22 days ago

    If it is not something obvious like paging or out of CPU I would open a perfPMR.



    ------------------------------
    Alexander Pettitt
    ------------------------------

    Original Message


  • 4.  RE: AIX performance slowness check

    Posted 22 days ago

    Thank you perfpmr is also a option and to analyze it we have to go back to IBM. But before going to IBM we wanted to make sure everything is good as per system performance perspective. Some time app team complain about slowness but actually system perform good. What command we can use to check for those system health? Sometime we have to join P1/P2 calls and there we have to do system health and at that time we do not have ample time to respond. there we have to quickly does the health check. How can we do that? any reading material about that? I need from basic commands to advanced so that we can be make sure from AIX side that we are good. They need to check within app/db.



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 5.  RE: AIX performance slowness check

    Posted 22 days ago

    nmon is an easy way to look at a LPAR.

    Putting a crontab entry will generate a file so you have historical data since the complaint is always about the past.
    #entry for nmon
    0 0 * * * /usr/bin/topas_nmon -s300 -c288 -f -Y -V -P -M -^ -A

    The nmon analyser can process those files and help you visualize the data. Don't share the graphs or people will want you to make them all the time. This is super helpful since you can see a trend of change. 

    You can grab the file from today and process it without waiting for those health checks. 

    Make sure you have some way to clean up/remove/save the old files or they will fill up the filesystem.

    There is also NMONVisualizer but I have never used that.

    Thanks to Nigel Griffiths who still lurks here.



    ------------------------------
    Alexander Pettitt
    ------------------------------

    Original Message


  • 6.  RE: AIX performance slowness check

    Posted 22 days ago 
    CPU Usage:
  vmstat 1 5                  # CPU, memory, and I/O stats
  sar -u 1 5                  # CPU usage over time
  mpstat                      # Per-processor CPU stats
  topas                       # Real-time performance monitor
  nmon                        # Interactive monitor (press 'c' for CPU)

Memory Stats:
  svmon -G                    # Global memory usage
  vmstat -v                   # Virtual memory stats
  lsps -a                     # Paging space usage
  vmo -a                      # Virtual memory tunables
  nmon                        # Press 'm' for memory stats

Disk I/O:
  iostat -D 1 5               # Disk I/O per device
  sar -d 1 5                  # Disk activity over time
  filemon -v -o /tmp/filemon.out -O all   # Detailed file system I/O
  lvmstat                     # Logical volume performance (if available)
  nmon                        # Press 'd' for disk stats

Network I/O:
  netstat -v                  # Interface statistics
  entstat -d <entX>           # Ethernet adapter stats (replace <entX>)
  ifconfig -a                 # Interface configuration
  netpmon                     # Network performance monitor
  nmon                        # Press 'n' for network stats

System Errors & Logs:
  errpt -a                    # Detailed error report
  errpt -d H                  # Hardware-related errors
  dmesg                       # Kernel messages
  alog -o -t boot             # Boot log
  lsattr -El sys0             # System attributes

Top Processes:
  ps -ef                      # List all processes
  ps aux | sort -nrk 3 | head # Top CPU-consuming processes
  ps aux | sort -nrk 4 | head # Top memory-consuming processes
  topas -P                    # Real-time process monitoring
  nmon                        # Press 't' for top processes

Optional Tools:
  nmon -f -s 10 -c 6          # Collect performance data for analysis
  perfpmr                     # IBM's performance analysis tool
  snap -a                     # Collect system config/logs for IBM support


    ------------------------------
    Rit De
    ------------------------------

    Original Message


  • 7.  RE: AIX performance slowness check

    Posted 10 days ago

    Alexander,

    Thanks for pointing out nmon.

    One point of clarification - the nmon command web page (at the top) states:

    Note: In recording mode, specify only one of the -f, -F, -z, -x, or -X flags as the first argument.

    So: /usr/bin/topas_nmon -s300 -c288 -f ...
    Should be: /usr/bin/topas_nmon -f -s300 -c288 ...

    Also, computers and disks have become faster and much larger in recent years.
    So, most nmon users collect in a day more than 288 snapshots with smaller sleep periods.

    Typically, nmon -f -s 60 -c 1440 --or-- nmon - s30 -c 2880
    -This is once a minute or once every 30 seconds.

    You are correct that system administrators need to take care that the output directory/filesystem does not fill up.

    One further point:

    The Excel-based nmon analyser is excellent but has drawbacks: 1) manpower intensive (if you have lots of servers), 2) Excel can crash or get stuck, 3) Can be slow and 4) the output files can be very large and thus hard to share.

    To avoid these problems, the nmonchart Ksh script can be used to create the graphs (and a few more helpful graphs) into a smaller web page file (.html).
    nmonchart can be found here: https://nmon.sourceforge.io/pmwiki.php?n=Site.Nmonchart
    or you can access the online service to run nmonchart for you here https://light-magazine.org/mr-nmon/pmwiki.php

    nmon is not really for the performance tuning novice, but there is some advice on my nmon YouTube Videos here:
    https://www.youtube.com/playlist?list=PLKQlFnmiWVyf3xWPDB-PxUYibVOcNkbml

    Cheers, Nigel @mrnmon Griffiths



    ------------------------------
    Nigel Griffiths - IBM retired
    London, UK
    @mr_nmon
    ------------------------------

    Original Message


  • 8.  RE: AIX performance slowness check

    Posted 19 days ago

    Manoj - I spent over 18 years in performance supporting customers. Because the issue may be almost anywhere (network, disk, CPU, memory, external resource like DNS) it take a skilled person to determine what is happening once you get past the simple obvious things. Also, please keep in mind that if the issue is a 20 'pause' in response and your tools are collecting data with a 30 second interval, the tools may very well miss it. 

    I recommend you open a case with AIX support or if you internal to IBM, you can ping me and I will try to assist you. 



    ------------------------------
    Grover Davidson
    ------------------------------

    Original Message


  • 9.  RE: AIX performance slowness check

    Posted 21 days ago

    Manoj;

    Another good option is Performance Navigator for AIX.  It uses the NMON data and condenses it so you can have 1 year's worth of data to see trends and history of any process.  It of course uses NMON data for capacity planning, trouble shooting, and reporting for AIX (Vios, Linux and IBM i).  The collection process is free and we do a service to help identify performance issues with a process, processor, memory, storage etc.https://www.fortra.com/products/capacity-planning-and-performance-analysis-software



    ------------------------------
    Tom Huntington
    EVP of Technical Solutions
    Fortra
    Eden Prairie MN
    9523349940
    ------------------------------

    Original Message


  • 10.  RE: AIX performance slowness check

    Posted 20 days ago

    Monitoring AIX, IBM i, Linux on Power



    ------------------------------
    Christopher Lang
    ------------------------------

    Original Message


  • 11.  RE: AIX performance slowness check

    Posted 20 days ago

    Another repository of good information is the Best Practices top-level page:
    https://www.ibm.com/support/pages/node/883882



    ------------------------------
    Glen Corneau
    ------------------------------

    Original Message


  • 12.  RE: AIX performance slowness check

    Posted 19 days ago

    Manoj,

    A good option for quick visibility and details into the performance status of PowerVM, AIX and Linux LPARs is                                                                                                        BVQ - Monitoring for IBM PowerVM, Storage and SAN IO   https://svasoftware.com/bvq-ibm/

    Some of the metrics you can see quickly are on PowerVM, OS level AIX or Linux, IBM Storage and SAN IO/ Example:

    Processor

    System Cores entitled/utilized

    Core Usage

    LPAR/VM Cores entitled/utilized

    Logical CPU

    vCPU utilization*

    IBM Flash IO and latency as well as SAN port IO, and other metrics



    ------------------------------
    Rosario Neuman
    ------------------------------

    Original Message


  • 13.  RE: AIX performance slowness check

    Posted 16 days ago

    Hi Manoj

    some very good and valid suggestions in the posts. I will add to the list another one we often use to get a quick glance before diving deeper into a call with IBM is LPAR2RRD and STOR2RRD. 

    Our Application guys use these graphs to check if the system is CPU or Memory starved.

    NMON on AIX we use to look what the system is doing realtime and most times we find its CPU entitlements are out and a few adjustments help but not always.

    As another post mentioned, performance tuning is tricky and we often go to IBM with perform data.



    ------------------------------
    Regards

    Anwar Williams
    ------------------------------

    Original Message


  • 14.  RE: AIX performance slowness check

    Posted 5 days ago

    Hi Anwar,

    nice to mention our tools LPAR2RRD/STOR2RRD.

    BTW you might already know that we have a brand new monitoring tool called XorMon Next Generation, which is the replacement for LPAR2RRD and STOR2RRD: https://xormon.com/documentation-ng.php.
    I recommend testing it; it's a much better tool. It has many new features compared to our older tools.



    ------------------------------
    pavel hampl
    ------------------------------

    Original Message


  • 15.  RE: AIX performance slowness check

    Posted 4 days ago

    Hi Pavel,

    Oh Yes offcourse we use it XorMon I just couldn't get to the name :-)

    It is our Primary dashboard, we simply love it and the Application guys love it even more since they can tell us when the systems having CPU spikes but then I reply we have Shared Processor Pools and point them to Nigel Griffiths legendary tutorials. LOL



    ------------------------------
    Regards

    Anwar Williams
    ------------------------------

    Original Message


Related Content