Power

 View Only

 nmem64 example to utilize up to 36TB memory

Lukas Schmid's profile image
Lukas Schmid posted 11/21/24 10:14 AM

Hi, im looking for an example to utilize a big amount of memory wthin an AIX LPAR. 

I know nmem64 from Nigel, but I struggle a bit with addressing 2GB emory junks.  I also know I need to run multiple nmem64 to address a huge amount of Memory. 
But this first simple process isn't working. 


see here: # ./nmem64 -m 2000 -s 120
malloc: Not enough space 

Any idea whats wrong?

thanks for a short feedback, Lukas  

Yves-Sandro Foltys's profile image
Yves-Sandro Foltys

Hi Lukas,
I don't have an alternative to nmem64, but the error message looks like hitting a ulimit to me. 
Maybe setting "ulimit -d unlimited" helps.

Regards,

Yves

Nigel Griffiths's profile image
Nigel Griffiths IBM Champion

First 36 TB of memory is extremely large. You have a serious server.

I looked up the Power10 memory bandwidth and found:

The IBM Power10 processor has a maximum memory bandwidth of 409 GB/sec per socket for 32 GB and 64 GB memory cards, and 375 GB/sec per socket for 128 GB and 256 GB memory cards. The Power10's memory bandwidth is 2.6 times higher than scalable x86 processors.

With with 3 TB of memory per Power10 socket with up to 15 CPUs.

I assume you are trying some sort of "burn in" test to check the memory is fully working - IBM will have already done that but I understand the need.

For 36TB, you have at least 9 sockets with (guessing at least) 120 to the full 240 CPUs.

You will have to use all of the CPU to test memory or you will be waiting days.

As nmem64 is single threaded, a starting point would be one nmem64 process per CPU and up to eight nmem64 processes per CPU. Due to SMT=8.

There is no getting around this. So you are into hundreds of copies of nmem64.

At 36 TB and say 100 processes you will need 36 x 1024 / 100 = 368 GB per process.

Testing this much memory is not a trivial task.

I have never has such a huge machine to "play" with - please let me know how you get on.

Ask if you get stuck, I will try to help out.

With 100 to 800 processes, I would guess you have to send the output to log files and then run checks on the log files to fine out if it is working as expected.

Even starting 800 processes, getting the kernel to allocate the space and then write to each memory page to force the allocation of real memory could take a few hours!

Good luck, cheers, Nigel

Lukas Schmid's profile image
Lukas Schmid

Hi, here is my update about the Memory utilization.
The reason behind is to test it in combination with ncpu and see the "electrical Power" consumption of a P10 box under full load. 
As nmem64 may be the wrong approach I tried to follow a different path:

create a ramdisk with the big amount of Memory
#mkramdisk -u 35000G 
#./ndisk64 -R -f /dev/rramdisk0 -s 35000G -M 174 -t 3600 

plus I ran
#./ncpu -p 174 (174CPUs in the profile / dedicated/donating) in parallel to stress the CPUs!

Result: the one LPAR with these big resources (174CPUS/ 35TB Memory) is fully utilized, but the electrical Power is only 8kWH instead of 14kWH, calculated by
the IBM Power estimator.
Any idea how to stress this machine? br Lukas
**** Update

HI all, found the answer of the missing electrical Power consumption gap:

Power Estimator= 14.5 kW
- PCI-Card estimate= 0.27kW (24 cards) 6.48kW (see ...- My stress test (CPU & Memory = 8kW)
The math now is ok, br Lukas