AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
 View Only
Expand all | Collapse all

Memory paging during Oracle RMAN backups

  • 1.  Memory paging during Oracle RMAN backups

    Posted Mon May 18, 2009 02:14 PM

    Originally posted by: higby


    Does anybody have any idea why AIX 6.1 pages out hundreds of pages during the Oracle RMAN backups, even though there are oodles and gobs of free pages available at all times? Most of my vmo settings are at the default except for minperm% which I've lowered to 1% to try to keep numperm above it. lru_file_repage is zero. There are 5 memory pools and three of them are idenical in size and the other two are very close. IBM hasn't got a clue, but I'm convinced their VMM algorithm doesn't always work as advertised in 6.1. In AIX 5.3, I was running Oracle with 9.5 GB of memory and it was not paging. In AIX 6.1, I've had to increase it to 14 GB and it's still paging out during RMAN backups. The rest of the time, the machine is fine and doesn't page a lick. Each night, it adds about 5% to the 4 GB paging space. All thoughts appreciated.
    #AIX-Forum


  • 2.  Re: Memory paging during Oracle RMAN backups

    Posted Mon May 18, 2009 02:28 PM

    Originally posted by: alethad


    Have you checked with Oracle to see if they have any fixes for AIX6.1? Or recommended AIX PTFS?

    You know we had this same issue in our AIX5.3 a few years ago but there was an AIX PTF and an Oracle patch that was required for us to go to a certain TL level in AIX5.3. We actually found out about it through the Oracle support. Could this be the same thing?
    #AIX-Forum


  • 3.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 06:44 AM

    Originally posted by: Holgervk


    I doubt that there are lots of free pages at the moment that aix pages out to paging-space.

    There may be millions of free pages and at the same time there may be pages in paging-space - but thats another thing.

    Run something like vmstat -Itw 600 when the backup runs and post the output here.
    #AIX-Forum


  • 4.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 08:23 AM

    Originally posted by: higby


    Not true. It really does have oodles and gobs of free space. Here's the vmstat -Iw taken at one second intervals from last night. See the 'fre' column and the 'po' column. Why does it page out (seeline 2) when it has 81,972 free pages available. The 'fi' column (file pages being read in) was only demanding 22792 free frames, yet it still paged out 98 to make room. Why? I don't think the VMM algorithm from IBM works as advertised in all cases.

    kthr memory page faults cpu

    ---------------------

    -----------------------
    r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
    0 6 0 2993061 85125 21998 8000 0 0 35583 45859 638 16486 10771 12 50 17 22 1.62 70.4
    13 6 0 2996725 81972 22792 8526 0 98 36781 59633 597 2982 17074 3 49 25 23 1.36 59.0
    3 7 2 2998306 81376 24570 8212 7 54 40485 83635 640 7308 15756 5 50 29 16 1.41 61.2
    0 11 2 2999411 83908 11571 12488 10 124 12747 73144 842 7778 5183 4 28 36 31 0.88 38.4
    #AIX-Forum


  • 5.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 08:36 AM

    Originally posted by: Holgervk


    strange...
    you use CIO but nevertheless have much fi/fo

    what is your minfree-/maxfree setting?

    did you install 6.1 "freshly" or did you migrate a system from 5.3?
    #AIX-Forum


  • 6.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 02:30 AM

    Originally posted by: niella


    I wouldn't be so quick to assume this, the VMM is really complex so there might be some reasonable explanation.

    How much memory on this system, what does 'svmon -G' and 'vmstat -v' show at the time of your vmstat paging?

    Niel
    #AIX-Forum


  • 7.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 08:45 AM

    Originally posted by: higby


    We don't use CIO. We tried it several years ago, but the performance really suffered for some reason, so they backed it out and are afraid to try again. We just lowered minfree and maxfree by one-third to 8192 and 8704. They had been 12288 and 12800.
    #AIX-Forum


  • 8.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 08:54 AM

    Originally posted by: Holgervk


    well, you have threads in the "p" column, meaning "Number of threads waiting on I/O to raw devices per second."
    so, if you dont use CIO (which - correctly configured - nearly always improves performance) you use raw logical volumes??
    you know that starting with version 10 oracle uses CIO on its own (whether the fs is cio-mounted or not) if you configure setall in fs_options?

    anyway... if you increased minfree from 960 (default) to 8192 vmm starts freeing pages at 40960 free pages (5 mempools * 8192).

    so,
    3 7 2 2998306 81376 24570 8212 7 54 40485 83635 640 7308 15756 5 50 29 16 1.41 61.2
    aix had to free 40485 pages leaving 40960 (additionally) freed, those are about 81000 pages total
    #AIX-Forum


  • 9.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 09:06 AM

    Originally posted by: Holgervk


    my answer is partly wrong as "fre" does not tell how much pages were free as a peak... but as an average
    thus, one cannot determine this very detailed

    however, based on this vmstat output and your min/maxfree setting I dont see any reason to say that vmm does not work as expected

    and you really increased minfree from 960 (default) to 8192??
    minfree, not minperm!
    #AIX-Forum


  • 10.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 08:46 AM

    Originally posted by: higby


    Forgot to answer about AIX 6.1. We migrated from 5.3
    #AIX-Forum


  • 11.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 09:14 AM

    Originally posted by: alethad


    Have you checked out this White paper?? It's very good. Particularly pages 26-27 would be of some interest to you. It's been updated for AIX6.1 & 10G also.

    http://www-03.ibm.com/support/techdocs/atsmastr.nsf/84279f6ed9ffde6f86256ccf00653ad3/bae31a51a00fa0018625721f00268dc4/$FILE/Oracle%20Architecture%20and%20Tuning%20on%20AIX%20(v%201.3).pdf

    I have kept this around for a while and has really helped whenever we run into performance issues with Oracle. We run 9.2 & 10G.
    good luck.
    #AIX-Forum


  • 12.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 09:09 AM

    Originally posted by: higby


    Bear with me, I'm a little slow. I'm not quite following you and am not quite seeing your point. Are you implying our minfree and maxfree are too big then? I still don't understand why it's paging. Sorry??
    #AIX-Forum


  • 13.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 09:18 AM

    Originally posted by: higby


    No, I hadn't seen that one. I'll definitely read it over. Thank you very much.
    #AIX-Forum


  • 14.  Re: Memory paging during Oracle RMAN backups

    Posted Tue May 19, 2009 09:33 AM

    Originally posted by: alethad


    I forgot to send this link also. Take a look at it too. Just a quick blog.

    http://jaffardba.blogspot.com/2008/12/how-to-boost-rman-performance-on-aix5l.html

    It might be worth trying but not sure how you're environment is set.
    #AIX-Forum


  • 15.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 07:47 AM

    Originally posted by: higby


    We're at 14 GB of memory now. When we were on AIX 5.3, it was running fine without paging and using 9.5 GB. As soon as we upgraded to AIX 6.1, it started to page and I've slowly added 4.5 GB over the weeks and now it's up to 14....and it's still paging during the RMAN backups.

    The svmon shows tons of free pages even during the worst paging just like vmstat does. vmstat -v shows about what you would expect:
    3670016 memory pages
    3542689 lruable pages
    53205 free pages
    5 memory pools
    2111440 pinned pages
    80.0 maxpin percentage
    1.0 minperm percentage
    90.0 maxperm percentage
    16.3 numperm percentage
    580997 file pages
    0.0 compressed percentage
    0 compressed pages
    16.3 numclient percentage
    90.0 maxclient percentage
    580997 client pages
    0 remote pageouts scheduled
    0 pending disk I/Os blocked with no pbuf
    36270 paging space I/Os blocked with no psbuf
    2484 filesystem I/Os blocked with no fsbuf
    0 client filesystem I/Os blocked with no fsbuf
    217 external pager filesystem I/Os blocked with no fsbuf
    After reading some of the suggestions above, I chopped the minfree down from 12288 originally, then to 8192, and last night to 4096. But none of the settings make much difference. Last night at 4096, it was very bad and paged out so much it added 13% to the 4 GB paging space. The reason it was so big was because IBM told me to make it that big a couple years ago when this happened last with AIX 5.3 TL4, and it has been big ever since.
    #AIX-Forum


  • 16.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 07:54 AM

    Originally posted by: Holgervk


    it would be useful to see the output of vmstat -Iw after you changed minfree to 4096
    (which, imho is way too much. we run all our systems with the default of 960)
    #AIX-Forum


  • 17.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 08:36 AM

    Originally posted by: higby


    OK, here's a short vmstat sample from last night with minfree at 4096 and maxfree at 4608. This was taken using 1-second intervals. The 9th column is 'po':
    System configuration: lcpu=8 mem=14336MB ent=2.30

    kthr memory page faults cpu time

    ---------------------
    ------------------
    --------
    r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se

    1 10 0 3030011 41465 25886 11188 7 65 46511 64132 1201 7254 17745 10 60 15 15 1.80 78.3 19:13:37
    2 7 0 3038863 38775 20644 10514 0 124 29628 35139 1594 10824 20601 7 44 22 26 1.39 60.3 19:13:42
    4 8 0 3044020 38353 22051 10749 0 175 34899 128860 1504 10429 22667 6 53 20 21 1.58 68.7 19:13:44
    2 8 0 3045059 39319 20104 11220 1 138 27993 30586 970 3481 16965 14 41 22 23 1.45 63.0 19:13:45
    0 0 0 3049177 45492 24536 10392 2 103 44940 57803 1090 5568 23452 4 49 29 17 1.38 60.0 19:13:46
    0 15 0 3052260 39359 30108 12464 3 181 44241 75781 1370 7259 21255 5 49 28 19 1.38 60.2 19:13:47
    3 13 0 3056364 36679 27220 9733 3 161 43008 56500 1138 5354 24368 15 52 23 11 1.67 72.5 19:13:48
    #AIX-Forum


  • 18.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 08:49 AM

    Originally posted by: Holgervk


    ok... when you had minfree at 5*8192 = 40960 pages AIX started paging at 82000 free frames having a demand of 40485 pages
    3 7 2 2998306 81376 24570 8212 7 54 40485 83635 640 7308 15756 5 50 29 16 1.41 61.2

    now you have minfree at 5*4096 = 20480 and AIX started paging at 39000 free frames having a demand of 29682 pages
    2 7 0 3038863 38775 20644 10514 0 124 29628 35139 1594 10824 20601 7 44 22 26 1.39 60.3 19:13:42

    its hard to believe that this is the same workload
    earlier aix had to look at 83635 pages to free 40485 pages, fr:sr ratio 0.48
    now aix had to look at 29628 pages to free 35139 pages, fr:sr ratio 0.84

    but in any case it does not look that wrong to me what aix´s vmm does
    #AIX-Forum


  • 19.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 08:58 AM

    Originally posted by: higby


    It's the same workload every night...it's just the Oracle RMAN backup of the same databases.

    The bottom line is that I still do not understand why it has to page out or what to do to make it stop. What should I do?
    #AIX-Forum


  • 20.  Re: Memory paging during Oracle RMAN backups

    Posted Wed May 20, 2009 10:38 AM

    Originally posted by: SystemAdmin


    It appears you are double buffering your file data since you do not use cio on the data mounts.
    When using cio mounted filesystem you can only have oracle processes manage the data on the filesystems since oracle will manage the locking of the data. DO not use system commands to cp or mv data.

    I use a setup that has ONLY my data on the filesystems /data/project/files00, /data/project/files01 ...
    and used the mount option cio on the filesystems., Also, an oracle option was set to setall...... This prevented the file blocks being read in to the OS buffers and then into the Oracle Buffers. Data is read directly into the oracle buffers. HUGE performance benefit !!!!

    Also, Is your oracle using pinned shared memory or non pinned shared memory. With version 5.3 and 6.1 there is no need to pin oracle SGA.
    #AIX-Forum


  • 21.  Re: Memory paging during Oracle RMAN backups

    Posted Thu May 21, 2009 05:57 AM

    Originally posted by: MarkTaylor


    Mount the backup filesysttem -o rbrw so you dont cache file pages on that filesystem .. no need, unless you need to re-read the backup directly after you have written it !!!

    HTH
    Mark Taylor
    #AIX-Forum


  • 22.  Re: Memory paging during Oracle RMAN backups

    Posted Thu May 21, 2009 05:59 AM

    Originally posted by: MarkTaylor


    I am assuming you have a filesystem specifically for backups ? if not, then create one and mount it rbrw ..

    Rgds
    Mark Taylor
    #AIX-Forum


  • 23.  Re: Memory paging during Oracle RMAN backups

    Posted Thu May 21, 2009 03:29 PM

    Originally posted by: kah00na


    On a busy Oracle AIX server, the lru daemon scans and free up a lot of pages a lot of the day. However, most of them seem to be persistent data (aka files). When persistent data is freed in memory, it does not go to the paging space, rather it is paged to the filesystem. You can see the "pages/sec to File System" in the memory section of nmon. It looks like this:

    pages/sec In Out
    to Paging Space 0.0 0.0
    to File System 606.5 1293.6
    Page Scans 1638.1
    Page Cycles 0.0
    Page Steals 1512.1
    Page Faults 18879.4


    Only the working storage (or "computational") pages go to the paging space. Therefore, when we see the paging space being used, it is because the computational pages are being freed from the memory and they go to the paging space. persistent data could still be in the process of being paged out, but since it goes to the filesystem, you don't see the paging space usage going up.

    When would the system page out computational pages instead of files? That is dependent upon the minperm% and the maxperm%. If the lru_file_repage is set to "0", then the only time computational pages should be sent to the paging space is when the total number of free pages goes below the minperm%. With lru_file_repage set to 0, anytime the numperm% is ABOVE the minperm%, only file pages should be freed from memory. Once the system has less than the minperm% of free pages available, then the computational pages are just as likely to be paged out to memory as the file pages.

    Run "svmon -G" and look at the "memory" row and "size" column. Take that number times your minperm% (1%). When the number of free pages goes below the resulting number, the working storage (aka computational pages) become candidates to be paged out to the paging space.

    Assuming your memory/size field in "svmon -G" is 3670016, once the amount of "fre" column goes below 36,700.16 (3670016*0.01), computation pages can be sent to the paging space and probably are.

    If the "fre" pages can be kept above that threshold, you "shouldn't" see any activity to the paging space. Review your vmstat files and see if the paging is occuring when your "fre" pages are less than that threshold. You may not actually see a lower number than the threshold because you are looking at averages and the lru is working to free up pages, but you can make a good guess that it is by subtracting the "fr"eed pages from the "fre" total.

    This begs the question, how do you keep it below that threshold? You can try playing with the minfree/maxfree settings some. Perhaps spreading out the gap between them from 512 to 768. Maybe?
    #AIX-Forum


  • 24.  Re: Memory paging during Oracle RMAN backups

    Posted Thu May 21, 2009 03:31 PM

    Originally posted by: kah00na


    Sorry, this is how it looks in nmon.
    
    |        pages/sec  In     Out | | to Paging Space   0.0    0.0 | | to File System  606.5 1293.6 | | Page Scans     1638.1        | | Page Cycles       0.0        | | Page Steals    1512.1        | | Page Faults   18879.4        |
    

    #AIX-Forum