Informix

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark

These can be tough, but you have to start somewhere.

First thing - check to see if the high usage oninit processes are CPU VPs and not some other class of VP.  Run onstat -g glo and match the PID there with the PIDs shown in "top".

Are you using any PDQ?  Have any procedures been compiled with PDQPRIORITY set?

You can use a query like this against sysmaster to see what sessions have consumed the most CPU:

SELECT
s.sid ,
s.username ,
s.uid ,
s.pid ,
s.hostname ,
t.tid,
t.name,
t.statedesc ,
t.statedetail,
t.cpu_time
FROM syssessions s, systcblst t, sysrstcb r
WHERE t.tid = r.tid AND s.sid = r.sid
order by t.cpu_time desc;

Running this twice, say 60 seconds apart, and comparing the results may help determine what has used the most CPU in that interval.

Does the timing of these slowdowns coincide with an archive?

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark

Mark:

Check into the btree cleaner threads (onstat -C)

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark

Mark:

Check into the btree cleaner threads (onstat -C)

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark

Hi all, in the past few days we've encountered a situation where we see Informix run high CPU across all assigned CPU cores (16) for extended periods of time (up to 2 hours).  When we run TOP it shows all the oninit processes running at ~97% while user sessions etc. are only running with 1-2%.  We've tried to identify which process or system activities might be causing this, but nothing appears to be showing consistently as a long running process or activity.   We've run onstat -g act -r 2 but can't identify any threads that are running consistently.  We're struggling to see how a user processes could affect ALL the CPUs so wondering if there is a more fundamental issues with tables, extents or indexes?  or something else?  Any suggestions for additional commands or tools we could use to investigate the high CPU or tools we can use assess health of the DB?  It's the first time these issues have occurred for us, but happened twice now in 3 days.  We're on 12.10 FC13.  

Any suggestions or help appreciated.

Kind regards, Mark