AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.

 View Only
Back to discussions
Expand all | Collapse all

High memory utilization

  • 1.  High memory utilization

    Posted Tue July 18, 2023 06:07 AM
    Edited by Anup Regmi Tue July 18, 2023 06:11 AM
    Is there something wrong with the system, in the picture below it is seen that about 54% of allocated memory (146G) in the system is used as FS Cache and also there is pagespace utilization.
    For more information, Oracle DB is running on the particular server.
    While observing other performance evaluations, lrud process is utilizing a massive amount of CPU resources and there is a significant performance impact due to the same issues.
    I am even more confused by memory management by AIX, why it is using paging space and not releasing real memory from FSCache.
    There is 54% of 146G used by FS caching and also there is paging space utilization

    One more query, is it possible to flush the FileSystemCache manually? The %usage is not changed after I increased the physical memory.

    ------------------------------
    Anup Regmi
    ------------------------------



  • 2.  RE: High memory utilization

    Posted Tue July 18, 2023 07:31 AM
    On Tue, Jul 18, 2023 at 10:07:22AM +0000, Anup Regmi via IBM TechXchange Community wrote:
    > Is there something wrong with the system, in the picture below it is
    > seen that about 54% of allocated memory (146G) in the system is used
    > as FS Cache and also there is pagespace utilization. For more
    > information Oracle DB is running in the particular server. While
    > observing other performance evaluations, lrud process by roor is
    > utilizing a massive amount of CPU resources and there is significant
    > performance impact due to the same issues. I am even more confused
    > by memory management by AIX, why it is using paging space and not
    > releasing real memory from FSCache.

    If LRUD is high with Oracle, check if the database is configured for
    file options ALL. This enables CIO mode which should stop using the
    filesystem cache for database files, reducing LRUD traffic.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 3.  RE: High memory utilization

    Posted Tue July 18, 2023 12:08 PM

    Hi Russell

    As I am unaware of the database things, I asked the database admin to check as you mentioned and he is also unaware of these configurations.



    ------------------------------
    Anup Regmi
    ------------------------------

    Original Message


  • 4.  RE: High memory utilization

    Posted Tue July 18, 2023 02:48 PM
    Anup,

    In Oracle database's init.ora:

    filesystemio_options = SETALL

    Remember to read up on this first. I just Googled it from prior experience.

    Thanks.

    On Tue, Jul 18, 2023 at 04:08:18PM +0000, Anup Regmi via IBM TechXchange Community wrote:
    > Hi Russell
    >
    >
    > As I am unaware of the database things, I asked the database admin to check as you mentioned and he is also unaware of these configurations.
    >
    >
    > ------------------------------
    > Anup Regmi
    > ------------------------------
    > -------------------------------------------
    > Original Message:
    > Sent: Tue July 18, 2023 07:30 AM
    > From: Russell Adams
    > Subject: High memory utilization
    >
    > On Tue, Jul 18, 2023 at 10:07:22AM +0000, Anup Regmi via IBM TechXchange Community wrote:
    > > Is there something wrong with the system, in the picture below it is
    > > seen that about 54% of allocated memory (146G) in the system is used
    > > as FS Cache and also there is pagespace utilization. For more
    > > information Oracle DB is running in the particular server. While
    > > observing other performance evaluations, lrud process by roor is
    > > utilizing a massive amount of CPU resources and there is significant
    > > performance impact due to the same issues. I am even more confused
    > > by memory management by AIX, why it is using paging space and not
    > > releasing real memory from FSCache.
    >
    > If LRUD is high with Oracle, check if the database is configured for
    > file options ALL. This enables CIO mode which should stop using the
    > filesystem cache for database files, reducing LRUD traffic.
    >
    > ------------------------------------------------------------------
    > Russell Adams Russell.Adams@AdamsSystems.nl <russell.adams@adamssystems.nl>
    > Principal Consultant Adams Systems Consultancy
    > https://adamssystems.nl/ <https: adamssystems.nl/="">
    >
    >
    > Original Message:
    > Sent: 7/18/2023 6:07:00 AM
    > From: Anup Regmi
    > Subject: High memory utilization
    >
    > Is there something wrong with the system, in the picture below it is seen that about 54% of allocated memory (146G) in the system is used as FS Cache and also there is pagespace utilization.For more information, Oracle DB is running on the particular server.While observing other performance evaluations, lrud process is utilizing a massive amount of CPU resources and there is a significant performance impact due to the same issues.I am even more confused by memory management by AIX, why it is using paging space and not releasing real memory from FSCache.
    >
    >
    > One more query, is it possible to flush the FileSystemCache manually? The %usage is not changed after I increased the physical memory.
    >
    > ------------------------------
    > Anup Regmi
    > ------------------------------
    >
    >
    > Reply to Sender : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=373987&SenderKey=01a5236b-ec58-4e7e-a579-ecf97601f720
    >
    > Reply to Discussion : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6049&MID=373987
    >
    >
    >
    > You are subscribed to "AIX" as Russell.Adams@AdamsSystems.nl. To change your subscriptions, go to http://community.ibm.com/community/user/preferences?section=Subscriptions. To unsubscribe from this community discussion, go to http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=7b554d78-d4dc-417a-b4dc-017e309e5c91.


    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 5.  RE: High memory utilization

    Posted Tue July 18, 2023 09:16 AM

    Hi Anup,

    Your paging space usage is tiny, I wouldn't worry about that. File cache is important, don't flush it.

    I'm pretty sure you don't need more memory on that system, you are only using a quarter of it. What do you mean %usage didn't change? And how much did you add?

    Regards,

    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------

    Original Message


  • 6.  RE: High memory utilization

    Posted Tue July 18, 2023 12:02 PM

    Hi Henrik

    Also, LRUD and VITOL processes are running increasing the CPU utilization by the system itself up to 50% of 6cores in Power 9 server. From my understanding, these processes are using so many resources due to a lack of memory and I thought it can be minimized by flushing the FSCache.

    Previously, the allocated memory was 130GB, performance degradation was noticed, there was paging space utilization which was in increasing order so real memory was increased by 16GB. And in both cases, memory utilized by FSCache was around 54%.

    Regards



    ------------------------------
    Anup Regmi
    ------------------------------

    Original Message


  • 7.  RE: High memory utilization

    Posted Tue July 18, 2023 09:49 AM

    I suggest you also check out the recommended best practices in the document:

    https://www.ibm.com/support/pages/system/files/inline-files/Managing%20the%20Stability%20and%20Performance%20of%20current%20Oracle%20Database%20versions%20running%20AIX%20on%20Power%20Systems%20including%20POWER10%20%2008_04_2022_0.pdf



    ------------------------------
    Rafael Cezario
    ------------------------------

    Original Message


  • 8.  RE: High memory utilization

    Posted Wed July 19, 2023 02:45 AM

    Hi Anup,

    can you run few commands when you encounter the problem:

    uptime 
mount 
ipcs -bm
svmon -G
svmon -G -O unit=GB 
vmstat -v
vmstat -s 
echo "mempsum *" | kdb | sed -n '/mempsum/,$p'
vmstat -p ALL 1 60
vmstat -IWwt 1 60


    ------------------------------
    Jakub Pacowski
    ------------------------------

    Original Message


  • 9.  RE: High memory utilization

    Posted Wed July 19, 2023 03:51 AM

    > Is there something wrong with the system, in the picture below it is seen that about 54% of allocated memory (146G)
    > in the system is used as FS Cache and also there is pagespace utilization.

    The screenshot you posted shows no paging (ie performing I/O from/to pagings space) - look at "pages/sec to Paging Space".
    Yes, a part of paging space is ocupied by some code paged out, but it is quite OK. This is what paging space is for: storing unused data so that more real memory is available for used data.

    I would call this system state perhaps a bit strange (why so much data - 3GB - went from real mem to paging space?) but still quite safe and OK (assuming paging space I/O is really non-existent).



    ------------------------------
    Lech Szychowski
    ------------------------------

    Original Message


  • 10.  RE: High memory utilization

    Posted Wed July 19, 2023 05:24 AM
    Edited by Henrik Morsing Wed July 19, 2023 05:25 AM

    Hi Anup,

    I have worked with AIX for 24 years, Linux for 27 years, and I have spent all of that time trying to explain to DBAs (especially) and application people that there is nothing wrong with paging space being used.

    What you need to look at is paging in vs out, IBM's white paper actually has a ratio for when it starts going bad. But, even better is looking at the memory page scanner stats, that's what truly tells you if there is contention for memory pages.

    I wrote a script last year to look at memory stats and tell you if you need to add memory. It looks at more than just memory also processor, disks and soon fibre adapters. You can find it here:

    Capacity and performance check script | AIXperts Consultancy ltd.

    Regards,

    Henrik



    ------------------------------
    Henrik Morsing
    ------------------------------

    Original Message


  • 11.  RE: High memory utilization

    Posted Wed July 19, 2023 07:27 AM

    Kudos Henrik,

    I haven't run your script but looks nice. Have to test it on live LPAR. Earl rocks and his recommendations are imho the best way to start with performance review and tunning. 

    I would suggest also to extend memory monitoring with memory pools check. It could be also reason for paging and potential bottleneck. Maybe large pages also? I had one real life example when client asked me about paging activity on idle lpar. Just OS was running with large number of assigned memory. This was LPAR used for Oracle some time ago and they haven't clean large pages assignment causing most of the memory to be pinned (not the Anup case as there is 11.9% only;)) 

    I



    ------------------------------
    Jakub Pacowski
    ------------------------------

    Original Message


  • 12.  RE: High memory utilization

    Posted Wed July 19, 2023 11:32 AM

    Hi

    Please which AIX version is that ??



    ------------------------------
    Cesar Daniel Delgado Ponce
    ------------------------------

    Original Message


  • 13.  RE: High memory utilization

    Posted Wed July 19, 2023 11:50 PM

    Hi

    OS level is 7200-05-02



    ------------------------------
    Anup Regmi
    ------------------------------

    Original Message


  • 14.  RE: High memory utilization

    Posted Wed July 19, 2023 11:55 PM

    Thank you, everyone, for your valuable response. On monitoring the I/O stats read max serv time was quite high on production disks, and after stopping some reporting services in the same server every seems Ok and the system is operating normally.



    ------------------------------
    Anup Regmi
    ------------------------------

    Original Message


  • 15.  RE: High memory utilization

    Posted Thu July 20, 2023 08:32 AM

    How Anup,

    I assume you're on a SAN? Run 'fcstat -D fcsX' and check the values:

    FC SCSI Adapter Driver Information
      No DMA Resource Count: 0
      No Adapter Elements Count: 0
      No Command Resource Count: 0

    They should really be 0. Also check:

      FC SCSI Adapter Driver Queue Statistics
        Number of active commands:   0
        High water mark  of active commands:   375
        Number of pending commands:   0
        High water mark of pending commands:   336

    Adding the two High water mark counters should be less than the num_cmd_elems value on the adapter, found with 'lsattr -El fcsX -a num_cmd_elems -F value'.

    I have added these checks to my script, but still testing.

    Regards,

    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------

    Original Message


  • 16.  RE: High memory utilization

    Posted Thu July 20, 2023 12:26 PM

    By default, AIX will use any free memory for cache. This is normal and expected.

     

    Some page space (2 pages) is RESERVED for each process. This is part of the paging space 'in use'.

     

    I suspect that at some point in time, AVM (Active Virtual Memory) approached very close to the real memory size OR exceeded it. That is why you may have real paging space used. To understand why this has happened, you need to capture data at the exact time the pageout activity occurs. Page in activity really tells us very little about why the page out happened.

     

    Adding more memory will simply result in the cache growing larger at this point because we do not understand what caused the pageout activity.

     

    AIX performance support has tools that can identify what happened and help you correct it. Please open a ticket with them. Tuning without data to support the changes may result In bad things happening.

     

    Grover Davidson

    Sent from Mail for Windows

     




    Original Message


  • 17.  RE: High memory utilization

    Posted Thu July 20, 2023 04:27 AM

    Is the Oracle database using AMS, jfs2 filesystems or RAW volumes.  If it's using jfs2 are the data filesystems (not the redo or archive) mounted with CIO?  Using CIO will stop AIX double buffering/caching of database data.  Otherwise, it is perfectly normal for AIX to use any memory that is not used by applications/database for filesystem cache and to page out memory pages that have not be used in a long time in preference to memory pages that are more beneficial to be kept in RAM.

    The number of times I've needed to explain to DBAs how modern Unix based systems manage memory and that it's entirely normal for there to be very little free memory and a large amount of filesystem cache..........



    ------------------------------
    Phill Rowbottom
    ------------------------------

    Original Message


  • 18.  RE: High memory utilization

    Posted Thu July 20, 2023 04:35 AM

    > The number of times I've needed to explain to DBAs how modern Unix based systems
    > manage memory and that it's entirely normal for there to be very little free memory
    > and a large amount of filesystem cache..........

    Yep, MS contributed a lot to this situation...



    ------------------------------
    Lech Szychowski
    ------------------------------

    Original Message


  • 19.  RE: High memory utilization

    Posted Thu July 20, 2023 05:18 AM
    On Thu, Jul 20, 2023 at 08:26:35AM +0000, Phill Rowbottom via IBM TechXchange Community wrote:
    > Is the Oracle database using AMS, jfs2 filesystems or RAW volumes.
    > If it's using jfs2 are the data filesystems (not the redo or
    > archive) mounted with CIO? Using CIO will stop AIX double
    > buffering/caching of database data.

    This is the old solution which forces all files on that filesystem to
    open with CIO. Oracle's set file options ALL uses CIO mode to open the
    database files in each file open call. Then you don't need CIO on the
    AIX mount.

    CIO can make backup/restore, file listing and more difficult and
    slow. It's a bonus to have Oracle manage it internally.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 20.  RE: High memory utilization

    Posted Thu July 20, 2023 05:43 AM

    I've seen the Oracle set option not work often enough to always set the option on the filesystem - which works every time.



    ------------------------------
    Phill Rowbottom
    ------------------------------

    Original Message


  • 21.  RE: High memory utilization

    Posted Fri August 04, 2023 09:41 AM

    FS cache is useful, except when it is not, IMHO.

    For example this kind of a scenario.

    • backup of the databases are done to a filesystem, disk space used by the files created by the backup is for example 100 GiB
    • You keep 4 copies of the backups in the filesystem
    • the filesystem is backed up daily, using incremental backup

    If the memory pressure is low, then there is a good chance that good percentage of the buffer cache is being used by files that are written and read once, possibly several hours after the backup has completed.

    Lets assume that last 30 GiB of the files written by the backup are still in buffer cache.

    When the backup to TSM, Networker etc comes to do its jobs, it may or may not start at the files that have blocks in buffer cache.

    I personally don't see this as useful use of buffer cache.

    If the filesystem and logical volume are not fragmented then there is a good chance majority of the IO that is done to write and read the backups is sequential, meaning that AIX and possibly storage subsystems read ahead functionality can compensate for the lack of data in the buffer cache.

    The JFS2 filesystem has three mounting options rbr (release-behind-when-reading), rbw (release-behind-when-writing) and rbrw (release-behind-when-reading and release-behind-when-writing). These mount options should not be used on filesystems that have binaries, these are IMHO useful on backups, archive logs, by and large files that are written once and read once, most likely by backup.

    AIX releases buffer cache when the file has been deleted and when there are no open file handles to that particular inode or when the filesystem is unmounted, which can be done only when there are no opened files in the filesystem in question.



    ------------------------------
    Esa Kärkkäinen
    ------------------------------

    Original Message


  • 22.  RE: High memory utilization

    Posted Fri August 04, 2023 10:26 AM
    Esa,
    if you feel rbw would benefit you on your backups, but don’t want to use it on a system with binaries, have you considered splitting out your file-systems so backups are on their own filesystem, and thus can benefit from the rbw/rbrw mount option?

    I’m also curios why rbw isn’t something to use on a disk with binaries.. how often are binaries written?

    Tom


    Original Message


  • 23.  RE: High memory utilization

    Posted Mon August 07, 2023 01:47 AM

    Hi Tom,

    Yes, that's what I've been doing for years and years, separate filesystems for the application etc, what ever it might be.

    In case of Oracle DB server, separate filesystems for redo logs, archive logs, data files and backups. I've given the DBA what the file system layout could be, discussed about the layout with the DBA, made the adjustments to the the layout etc.

    The only additional option I usually use with the filesystems that I've created is noatime, which is the only additional option that I've used with file systems that contain executables, shared libraries etc.

    I've used "rbrw,noatime" with archive log and backup filesystems, "cio,noatime" with redo and data filesystems and just "noatime" with all the rest.

    By and large the Oracle binaries are written/updated quarterly, so backups are updated about hundred times more often than Oracle binaries.



    ------------------------------
    Esa Kärkkäinen
    ------------------------------

    Original Message


  • 24.  RE: High memory utilization

    Posted Tue August 08, 2023 09:35 AM

    Esa,

    since Oracle 12.2 we do not recommend to utilize the "cio" mount option but instead utilize the init.ora parameter "filesystemio_options=setall". With that setting Oracle will open all the relevant files with the CIO option itself.

    Ralf.



    ------------------------------
    Ralf Schmidt-Dannert
    ------------------------------

    Original Message