Originally posted by: salzo

Hi.

We have a 5300-06-01-0000 AIX machine that was installed in local disks and had a Power5 processor. One of the services running there is a Apache HTTP Server. The disks were cloned to SAN and the processor upgraded to Power6.

After that, the HTTP Server began to use 99% CPU. The service was isolated and is not receiving requests, but it's consuming the proc.

I did some TPROF tests (attached: tprof.httpd.txt) that show that the HTTPD processor consume is done mainly in Kernel libs (umem_move, mtrchook1, sc_msr_2_point, sys_call_end, lock_done, lock_read from 64/low.s) and Sharedl libs (divu64 from libc.ahttp://shr.o).

I haven't found incidents or reports related to these libraries.

I'd like to know if anyone knows other ways in which I can know "what is doing" the HTTPD or have some advice about the libraries.

Thank you very much.

Best regards,
Esteban.

Originally posted by: The_Doctor

A couple of quick things off the top:

1. 5300-06 is a little old to be running on a new p6. Best you plan to get that updated, especially for device driver support on some of the newer adapters...... like the LHEA & the 8GB FC adapter. But that's not your question, so I'll move on.

2. Are you running micro-partitioning or dedicated processors on the p6 ? What model of p6 ? how many processors ? I suspect you're running micro-partitioning, in which case the 99% utilization you quote is meaningless. Look for the PHYSICAL CONSUMED number.... it's the only value most can easily relate to.

For example, in SAR it is the "physc" column.... or in VMSTAT it's the "pc" column.

Look at the output from --> vmstat -w -t 1 30 <-- and I'm sure you'll see the "pc" column substantially < 1.0

BTW, in micro-partitioning, the 99% is a good indication that your LPAR has ceded the spare/excess cpu cycles back to the hypervisor.... to be used by other LPARs during the measured time period. This is a good thing.

..... and of course if you're running dedicated processors, something else is going on that is yet to be understood.

Originally posted by: salzo

Hi.

Thanks for your answer.

The processors are dedicated.

According to nmon: Processor: PowerPC_POWER6, MHz CPU clock rate: 4204, CPUs configured: 2, CPUs currently: 2, Hardware: 64 bit, Kernel: 64 bit, Logical Partition: Dynamic, AIX Kernel Version: 5.3.0.61 ML06 (lscfg output attached: lscfg-httpd-trhead.txt).

The "vmstat -w -t 1 30" commands shows the "pc" value always as 1.0 (vmstat-httpd-thread.txt).

topas shows:
Name PID CPU% PgSp Owner
httpd 217240 99.4 23.6 oracle

Kernel 53.2 |################
User 46.6 |##############
Wait 0.0 |
Idle 0.2 |#
Physc = 1.00

Thank you.

Originally posted by: The_Doctor

Are you sure you're running dedicated processors ? vmstat appears to show you are doing micro-partitioning with an LPAR set up like this:

Memory = 8 GB , Virtual Processor = 1, CPU Entitlement = 0.5, Uncapped Partition..... or am I mis-reading something? Wouldn't be the first time :)

Unable to tell what model of p6..... but I don't believe any of the above is really relevant now. I'd still suggest a minimum level of TL 9 for a p6. TL11 is out, you might want to consider something a little more current.

But, bottom line... yes, you are correct. It does look like the "httpd" process is consuming 100% of 1 Physical CPU.

Sorry, I'm not able to really help you. I'm not familiar enough with "httpd" to suggest a root cause why it might run 100% of 1 CPU. I assume the Apache logs are not giving you any clues ?

Originally posted by: cd3lgado

Hi

I would install the latest available tech levels for AIX 5.3. Remember that some work must have be done to AIX to work well with Power6. After that test you server's CPU behavior.

Originally posted by: salzo

Hi. Thanks for your answers. I will try the S.O update and update with the results.

After some investigation, I have found additional information and it seems the cause is a custom Module (not an official Apache module) that was working OK before the machine updates.

I have use:
"lsof -a -p PID" to see the file descriptors that are being used by the process.
"procstack PID" to see the current stack trace for each thread of the process.
"tprof -kvlsu -P PID -x sleep 60" to see detailed proc consume info for each thread of the process.

Originally posted by: salzo

Hi. The OS was downgraded to a somewhat old previously version (5300-05-05-0000) and the problem is gone. We're investigating a recommended level to update the OS. Thank you very much for your answers.