Originally posted by: Bhoju

Dear All, 

We have this particular issue several times where one of our process ( A shell script which does extensive DB operations, financial calculations etc) brings down the AIX server. 

We are trying to find a solution when such process runs and consumes pretty much all resources on the server, it should kill itself and not the server itself. It causes serious business problems and we need to find a solution to it on priority basis. 

Any expert opinion on this would be helpful. 

AIX information

Thanks,

Bhushan

Originally posted by: Wouter Liefting

Is your script running as root, and if so, why? If it's not running as root you can limit the resources with the ulimit command, or via the file /etc/security/limits. If the script then tries to allocate resources beyond its limit, the syscall will fail and the script will most likely kill itself (if it was written properly and checks return codes in the first place).

But I wonder why you would want the script to kill itself in the first place. If the script is not that important, don't run it. If the results are important, then the script itself should be investigated, not killed. Maybe doing stuff like this in a shell script is not the right method (*), and the script could be converted to run in perl, or something like that.

(*) Actually I'm pretty sure that if you do extensive DB operations, that they should not be done from a shell script.

Originally posted by: Bhoju

Hi Wouter, 

Thanks for the reply. Below is more explanation on the issue and some more considerations.

The script does not run as a root, it runs using one of the application id.  We will take the suggestions of limiting the resources in the consideration. 

About doing heavy operations from shell script, Technically, the script hits sybase open server (we have them configured as financial servers which does numerous financial calculations) so technically, the script is pretty simple but just make a sybase open server call which in turn does lot of sybase db operations. Now this financial servers are something like a black box to us and we cannot do much inside it. (They are delivered with product). 

Also the issue happens sometimes due to heavy data which also we cannot control, so we are hoping that we can kill the process if its about to bring down the machine and rerun it again later when there is no other load on system. 

Also the process is very important as it generates some financial reports for our customers and so very critical.

Thanks for your inputs, we will think about the limit  by setting per process limit for this process, if there is any other suggestion you can think of. please let us know.

Thanks again and have a nice day.

Bhushan

Originally posted by: Wouter Liefting

Okay, so it's not the script which brings the machine down, but it's the Sybase DB which brings the system down while it's running the query that was submitted by the script. That's an entirely different matter. From the AIX side you can limit the amount of resources that the Sybase DB uses (using ulimit like I described above) but there's a significant risk that your whole DB will crash if the DB hits those limits. Which is probably just as bad as AIX crashing.

The best option is to ask this question in a Sybase forum, because in all likelihood you can limit the resources that a single query is using, from within Sybase. Or you may need to so some sort of query optimization.

As a temporary measure, from your post I understand that it is possible to rerun the query later. This means it's not very time-critical. You could consider running these queries daily from cron at a time when the database is quiet, and then making the results available via a web server or similar. Yes, it means your customers won't have real-time results, but they don't get that either when the DB crashes.

Originally posted by: j.cortez

Hello,

First Question, has this script runned before fine without causing this crash? If yes, what had been changed before this issue starts? changes? If no, i should start with:

Check the total I/O systems with NMON, taking care with:

CPU, Memory, Swap, Discs and Network. See if there's something strange and what starts getting high first, to trace the problem. Is there any problems logged on application logs? System logs?

Is your capacity well adjusted?

Hope it helps,
João Bosco Cortez Filho