AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
 View Only
Expand all | Collapse all

Very busy paging, how to solve

  • 1.  Very busy paging, how to solve

    Posted Tue July 03, 2007 08:34 PM

    Originally posted by: SystemAdmin


    My AIX 5.3L p550 has 3792M. After I increased the maxuproc and maxservers for the sake of Oracle 10g R2, the paging went very busy constantly, even without Oracle using the system, the paging (hd6 and paging00) are still very busy once a while. The hd6 was default with 6272M (on hdisk0), then I added another paging00 (3200M on hdisk1) other date to ease paging. It did not work.

    Should I make it equal size (6272M) or should I decrease the hd6?
    Someone in this forum mentioned the paging should not mirror, however, the hd6 was setup by default on hdisk0, where ever lv is mirrored to hdisk1. So How can I remove the mirroring of hd6 on hdisk1?

    Besides, I think there must be some other activites causing the paging. Just could not figure where they came from? I used the filemon and topas to found high values of write on hd6 and paging00.

    This problem has bothered us since last Friday after we change this Box. Another AIX 5.3 p550 has the same configurations goes without problem (3920M memory). Need anyone´s urgent suggestions.
    #AIX-Forum


  • 2.  Re: Very busy paging, how to solve

    Posted Wed July 04, 2007 12:42 AM

    Originally posted by: SystemAdmin


    Since my hdisk2 is least hdisk on all four hds 99% of the time, I will replace my paging00 on hdisk1, which has the Oracle engine on, to a new paging space on the hdisk2 (at a different volume group). And see what happen.
    #AIX-Forum


  • 3.  Re: Very busy paging, how to solve

    Posted Wed July 04, 2007 08:20 AM

    Originally posted by: SystemAdmin


    There are some AIX parameters you might want to adjust to control this. Have a look at http://download.oracle.com/docs/html/A97297_01/appa_aix.htm paying particular attention to minperm and maxperm. This has worked wonders for me in the past.

    Regards
    Jim Lane
    #AIX-Forum


  • 4.  Re: Very busy paging, how to solve

    Posted Wed July 04, 2007 10:56 AM

    Originally posted by: SystemAdmin


    Thanks. Maxperm%, minperm% and maxclient% are the parameters I have not altered on my system. Are they associated with the lgpg usage on POWER 5 system, ie, do they work as a whole set?

    More important how can I pursude my manager to use those parameters on Prod system. (my other Test Box works fine without

    In addition to the paper Jaqui Lynch's paper http://users.ca.astound.net/baspence/AIXtip/AIX5_Initial_Tuning.htm
    she has a whole set of parameters. Comparing to my, I have missed a lot, especially numfsbufs, maxrandwrt and j2_maxRandomWrite. I think that paper provides a baseline for performance. However, my question is to implement them as whole or not?

    I move my paging00 on hdisk1 to hdisk2, the least active hd on my system. But the busy is still there, just now moves to hdisk2 instead of hdisk1.
    #AIX-Forum


  • 5.  Re: Very busy paging, how to solve

    Posted Wed July 04, 2007 10:57 AM

    Originally posted by: SystemAdmin


    Thanks. Maxperm%, minperm% and maxclient% are the parameters I have not altered on my system. Are they associated with the lgpg usage on POWER 5 system, ie, do they work as a whole set?

    More important how can I pursude my manager to use those parameters on Prod system. (my other Test Box works fine with those changes)

    In addition to the paper Jaqui Lynch's paper http://users.ca.astound.net/baspence/AIXtip/AIX5_Initial_Tuning.htm
    she has a whole set of parameters. Comparing to my, I have missed a lot, especially numfsbufs, maxrandwrt and j2_maxRandomWrite. I think that paper provides a baseline for performance. However, my question is to implement them as whole or not?

    I move my paging00 on hdisk1 to hdisk2, the least active hd on my system. But the busy is still there, just now moves to hdisk2 instead of hdisk1.
    #AIX-Forum


  • 6.  Re: Very busy paging, how to solve

    Posted Wed July 04, 2007 02:41 PM

    Originally posted by: SystemAdmin


    Check carefully with my vmstat -v, I think that I found the causes of the busy paging:
    903286 pending disk I/Os blocked with no pbuf
    3327074 paging space I/Os blocked with no psbuf
    2740 filesystem I/Os blocked with no fsbuf
    5179 client filesystem I/Os blocked with no fsbuf
    8121 external pager filesystem I/Os blocked with no fsbuf
    my pending disk is high, the paging space I/O is extremely high, of course the client filesytem also high according to some papers. Hence my questions are
    1. How to solve the "no psbuf" for paging space?
    one paper mentioned that it is the bug of OS (5.2) my is OS 5300-03-00
    2. Which parameter I can apply for correct "no psbuf"?
    3. for client system problem, I can increase the numfsbufs (my is 196 now). But the AIX doc says that I need to umount/mount the filesystem. which filesystem?
    my /dbbaan or my $ORACLE_HOME. Steps as
    a. umount /dbbaan (if this is the one causing problem)
    b. ioo -p -o numfsbufs=1024
    c. mount /dbbaan
    after I apply, do I still need to reboot
    4. for paging how can I umount/mount the pagaing space? by deactivate/activate
    or
    5. no paging for certain activities? How
    #AIX-Forum


  • 7.  Re: Very busy paging, how to solve

    Posted Thu July 05, 2007 01:23 PM

    Originally posted by: SystemAdmin


    So. I apllied the general solution for this type problem, that is to implement minperm%=5, maxperm%=90, maxclient%=90 and adjusted other ioo parameters. However, such solution simply soothes the bottleneck. I still wonder why system runs so many paging.

    Even with the fix, my "paging space I/Os blocked with no psbuf" still increases.
    #AIX-Forum


  • 8.  Re: Very busy paging, how to solve

    Posted Fri July 06, 2007 07:51 AM

    Originally posted by: SystemAdmin


    I don't think you've got it yet. If I remember correctly to reduce paging in an AIX/Oracle system you need to drastically reduce maxperm (and maxclient, I think). Try something like maxperm=30.

    Regards,
    Jim Lane
    #AIX-Forum


  • 9.  Re: Very busy paging, how to solve

    Posted Fri July 06, 2007 10:41 AM

    Originally posted by: SystemAdmin


    However, according to the latest IBM doc such as http://www-941.ibm.com/collaboration/wiki/download/attachments/436/VMM+Tuning+Tip+-+Proctecting+Comp+Memory.pdf?version=1, because of lru_file_repage = 0 for oracle, the maxperm% (maxclient% too) should set up at high value. Someone in the Oracle forum just mentioned a similar case last week (June 29), in which IBM engineer used the such value in stead of the old changes (maxperm%= 20~25, maxclient%=20~25).
    But you might be right. I will try and see what will happen.

    Also right now, when I checked with vmstat -a, occasionally at non busy hour, I saw the numperm% is less that minperm%. I remembered that some post mentioned that minperm% should at least less than the numperm%, a dynamic value of the running system, am I right?

    I tried the change of the minperm% to 4 (from 5), however the reboot took a much longer time. Any special concern for lower value of minperm%.
    #AIX-Forum


  • 10.  Re: Very busy paging, how to solve

    Posted Fri July 20, 2007 05:24 PM

    Originally posted by: vbaranov


    I/O buffer bottleneck.
    If "paging space I/Os blocked with no psbuf" constantly grows, that mean that "... bufstruct is unavailable and the VMM puts a thread on the VMM wait list". The number of bufstructs per file system, known as numfsbufs, affects JFS and paging space and is not dynamic, therefore, it's not in effect immediately after the change with "ioo" (mention parameter "M" in "ioo -L" output: changes are only effective for future mountings).
    Well, after the reboot it will be definitely in effect. It's enough to unmount/mount JFS filesystem, for paging space (just a guess) try deacivate/activate paging spaces to avoid a reboot. See this link for the details:
    http://publib.boulder.ibm.com/infocenter/pseries/v5r3/topic/com.ibm.aix.prftungd/doc/prftungd/fs_buff_tuning.htm
    #AIX-Forum


  • 11.  Re: Very busy paging, how to solve

    Posted Mon July 23, 2007 11:37 PM

    Originally posted by: SystemAdmin


    When I indicated that after fix of the new parameters for ioo, I meant that after the reboot. I used an application, OSWatcher from Oracle, to minitor the vmstat,ie the avm column. Apparently, the memroy usage is growing, not release, so that the "paging I/O.." kept increasing.
    I followed the ORacle suggestion to reduce the total SGA space, but it has very effect.
    I did set the lgpg_regions to the total SGA size because the lock_up = true of the SGA.
    Therefore I am looking for what caused the memory leak.
    #AIX-Forum