Originally posted by: Balaji-OHIO

hi,

We have two LPARs, both have same capacity and believe same configuration. ulimit settings for oracle user is unlimited for both LPARs. Installed oracle databases with same configurations on both LPARs, both databases sync every second so volume is same. Both LPARs/databases have identical jobs running at exact time.

We run net-backup process at same time on both LPARs everynight. LPAR1 finishes in 30 min but LPAR2 takes 2 hours to complete.

When I observed, i noticed backup processes running on LPAR1 consuming 35 to 40% CPU for every PID. And i noticed backup processes running on LPAR2 consuming only 5 to 10% CPU for every PID.

I'm understanding that backup process in LPAR2 delays because it's not utilizing available CPU, please correct me if 'm wrong? Secondly how we can make or instruct the AIX to utilize more CPUs for the backup processes running on LPAR2?

-Thanks in advance

-Bala

Originally posted by: luverofpeanuts

It does seem odd that LPAR2 takes longer, and I would agree, it seems likely that it's just not processing as fast because it's not getting as much CPU.   Trying to find out why that is, is what I would be doing as well.   Here are some followup questions that come to mind about your configuration...

1) Are the LPARS on the same physical system, or different?  If so, are the physical systems identical, or different? 

2) When you say LPARs are identical, assuming the underlying physical system is the same, processors assigned dedicated or shared?   If shared, I assume the entitlement, virtual processors assigned, and processing weight factor is the same as well.  (memory assigned is also identical?).  

3) Is VIOS involved?  The same VIOS for both?  Is VIOS SEA utilized for networking?    Is the VIOS processor/memory configuration identical as well, if they are on different VIOS?  

4) Aside from the backup processes, are the database equally utilized? In which case, when they are synced, they are both send/receiving equivalent amount of data to the other?  i.e.  One DB is not 'primarily' getting most of the work/changes.  

5) Is the backup server another physical machine?  Is there any priority scheme that that system uses when doing backup processing?  Could it be prioritizing one over the other?

6) What tool are using to compare CPU %?  topas? nmon? vmstat? ..etc.

Originally posted by: Balaji-OHIO

Thanks for the response.

1)  LPARS are on different physical servers.

2) Processor details below. I see small difference for both as below.

LPAR1:
Online virtual CPUs : 18
Maximum Virtual CPUs : 16
Minimum Virtual CPUs : 8
Maximum Physical CPUs in system: 24
Active Physical CPUs in system:24
Shared Physical CPUs in system:24
Active CPUs in pool : 24
Physical CPU percentage: 100%
Desired Virtual CPUs:8

LPAR2:
Online virtual CPUs : 18
Maximum Virtual CPUs : 16
Minimum Virtual CPUs : 8
Maximum Physical CPUs in system: 24
Active Physical CPUs in system:23
Shared Physical CPUs in system:23
Active CPUs in pool : 23
Physical CPU percentage: 100%
Desired Virtual CPUs:8

3) not sure, need to check with SA

4) yes, mostly database is utilized equally. One weird thing is, every Saturday full backup runs, both completes at same time, i noticed CPU utilization was almost equal. I see this CPU utilization difference happening only for the incremental backups running on week days.

5) net backup runs on different physical server.

6) i use nmon.

Originally posted by: luverofpeanuts

Great, that is helpful info.  

Since you're running nmon....from the 'p' display..... can you verify the 'entitlement' is the same on each?  And that it is 'uncapped' in each?  'PoolCPUs' should be check to see if they are the same too.   See attached graphic.

Since they are on different physical servers, (assuming they have the same physical specs, like processor speed), it 'appears' to me, that the slower LPAR my have more contention in the shared pool during the incremental backup times, whereas on the weekend full backup, there is no contention and they operate at a more similar rate.   The larger the difference between the virtual processors assigned and entitlement, the greater chance of other LPARs on a physical system affecting the share of the CPU that is available to your LPAR when it wants to use more than it's entitlement.  Assuming each of your LPARs have the same entitlement and are uncapped, then you would also need to look at the all of the other LPARs that are competing for processor in the same processor pool as the LPARs you are concerned with.  

I hope this helps!  Keep us updated on what you find ;-) 

Originally posted by: Balaji-OHIO

Attached screenshot for both lpa1 & lpar2, pls let me know if you need more info?

Attachment(s)

entitlement_det.docx   269 KB 1 version

Originally posted by: luverofpeanuts

That screen shot showing both LPARs shows alot of what I'd want to see from a processor/memory config.  It confirms that both LPARs are assigned the same amount of processing.  

They are 'capped', and the virtual processors online match the entitlement assigned.   So, both should be able use up to 8 processing units.  Even if other LPARs on each physical system are busy, they should be guaranteed to get similar processing resource.  

Memory assigned is the same, that is good ;-) 

One never rules out anything, but from a processor, memory, I'd expect those to not be causing a difference in the two throughputs.  

Next thing to look at is network configuration.   Given these LPARs on on different physical systems, there could be differences unseen by anyone but system admins.  

To start with, what do these show?  

ifconfig -a

lsdev | grep Eth

Originally posted by: Balaji-OHIO

topas output taken while RMAN incr backup running, converted into table format.

CPU  ---- LPAR1 --  LPAR2
user%  ---- 12.7 --  6.4
kern% ---- 11.2 -- 6
wait%  ---- 1.1 --  3.8
idle%  ---- 75  -- 83.9
Physc  ---- 2.91 -- 1.34
Entc%  ---- 36.43 -- 16.73

Network  ---- LPAR1  --  LPAR2
BPS  ---- 30.7M  -- 6.41M
I-Pkts  ---- 2.43k   -- 825.2
O-pkts  ---- 21.8k  -- 4.62K
B-in  ---- 395k  -- 116K
B-out ---- 30.3M  -- 6.30M

Disk  ---- LPAR1  -- LPAR2
Busy%  ---- 8  -- 10.7
BPS  ---- 3.14G   -- 944M
TPS  ---- 6.79K  -- 2.39K
B-Read  ---- 3.14G  -- 942M
B-write  ---- 1.14M  -- 2.32M

FileSystem  ---- LPAR1 -- LPAR2
BPS  ---- 784M -- 250M
TPS  ----- 2.24K  -- 2.13K
B-Read  ---- 784M -- 250M
B-write  ---- 428K --  269K

Thanks

Originally posted by: luverofpeanuts

Aside from checking the network setup, especially if using VIOS.... since you mentioned RMAN, it'd be worth double checking the RMAN settings as well.   If one was set to use more streams/sessions/threads than the other, that could explain CPU differences.  And the RMAN settings could be different for incremental versus full backup...i.e.. perhaps they are configured the same for full backup on the weekend, but the incremental, for some reason, could be different.