Originally posted by: haubi

Hi,

my problems started with "fork: Resource temporarily unavailable".

Thing is, there are more than 1400 "<defunct>" processes from different users around, each of them having parent-pid 1, the [i]init[/i].

So I wanted to know why [i]init[/i] does not do its job and removes them:

[i]truss -p 1[/i] repeats only two lines always:codeksetcontext_sigreturn(0x2FF20EB0, 0x2000115C, 0x2FF20EB0, 0x2FF3B820, 0x1000452C, 0x0000D0B2, 0x024A8400, 0x2FF3AEE0)
Received signal #11, SIGSEGV caught[/code]Huh, segfaulting init ?

Ok, [i]init[/i] currently is still segfaulting, and I've already collected output of these commands, called with argument "1" or "-p 1":
[i]proccred[/i], [i]procfiles[/i], [i]procfiles -n[/i], [i]procflags[/i], [i]procldd[/i], [i]procmap[/i], [i]procsig[/i], [i]procstack[/i], [i]procwdx[/i], [i]truss[/i], [i]lsof[/i]

What else could be of interest for IBM support if we eventually make a call on monday (which naturally will be post-mortem) ?

Thank you!
#AIX-Forum

Originally posted by: SystemAdmin

Hello:

This could be a memory issue. How much Physical (prtconf and vmstat) and paging space (lsps -s) is being used? What is the application thats residing on the system?
Also, try checking your system's max user processes

1. lsattr -E -l sys0 -a maxuproc

To change the value of maxuproc

1. chdev -l sys0 -a maxuproc=

My guess is its a memory related issue.

Good luck...


#AIX-Forum

Originally posted by: haubi

Thanks, have collected them.

> (prtconf and vmstat) and paging space (lsps -s) is

Hmm, doesn't look like memory related issue to me:codePaging Space Information
Total Paging Space: 16384MB
Percent Used: 1%

kthr memory page faults cpu

---

-----------

---

------------

---

r b avm fre re pi po fr sr cy in sy cs us sy id wa
3 1 523074 384884 0 0 0 4 9 0 263 73609 28614 13 6 81 0 [/code]
> being used? What is the application thats residing on
> the system?

This is a developer- and automated-build-system, and is up for ~50 days now...

> # lsattr -E -l sys0 -a maxuproc
codemaxuproc 500 Maximum number of PROCESSES allowed per user True[/code]Including ~480 defunct's, some users now have reached that 500 processes.

> Good luck...

Thank you ;)
#AIX-Forum

Originally posted by: SystemAdmin

You may have something there. I am having the same problem as the original poster, and I have just noticed that the init process has a large memory footprint:
$ ps -l -p 1
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
200003 A 0 1 0 107 113 20 74039000 261252 - 4165:56 init

Compared to another box running AIX 5.3 (but a 32-bit kernel):
$ ps -l -p 1
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
200003 A 0 1 0 0 60 20 7007c 756 - 0:16 init
#AIX-Forum

Originally posted by: SystemAdmin

I am having exactly the same problem. init catches sig 11, returns from the handler and so on. Reboots do not help.

At the moment I am up to 2374 zombies.

Have you been able to resolve this problem? If yes, can you please provide some more info?
#AIX-Forum

Originally posted by: haubi

> I am having exactly the same problem. init catches
> sig 11, returns from the handler and so on. Reboots
> do not help.
>
> At the moment I am up to 2374 zombies.
>
> Have you been able to resolve this problem? If yes,
> can you please provide some more info?

Well, because reboot did help here, our sysadmins were not interested in the details...

But as you can reproduce the problem, this is a good chance for aix devs (?) to find it - maybe you have a daemon being started which triggers that bug in init ?
#AIX-Forum

Originally posted by: Carlyle

Hello. We are having exactly the same problem here as well. Did you get anywhere on this?
#AIX-Forum

Originally posted by: IBMFORUMS

It seems like a memory issue.
Do you have performance issues as well?
Did you have any dead processes?
#AIX-Forum

Originally posted by: Carlyle

IBM has told us that this is a known issue in oslevel -r < 5300-05.

There is apparently a slumbering memory leak in initd that was awakened yesterday when we rolled our a faulty script which was entered into the inittab as respawn. The command respawning too rapidly apparently was able to cause some of our systems' initd processes to loose control of their memory management and begin spewing defunct process list entries.

If the faulting process is fixed or eliminated, a reboot certainly does help. But the problem can only really be solved by updating the OS.
#AIX-Forum