DataPower

Hi all, 

I am seeing suddent spike up in CPU on IDG 2018.x.x version on VMWare. Any idea what could be the major cause? What checks can be done? 
I have tried shutdown reboot which did not work. I did stylesheet & document cache clearing as well which failed to resolve issue ultimately. 

The issue started suddenly after the midnight and all before that the appliances were pretty stable. 

Please note that we do have 6 other datapowers on IDG 2018.x.x and all are very stable.

Any help would be appreaciated. 

Thanks ! 
Prathamesh

The first question will be, has anything changed in the affected machine?

A good place to start is the "Hypervisor Status".  See if something happened to the #cores or # of virtual CPUs.  Since it suspiciously changed overnight, if there were no other known changes, maybe something changed there.

If that isn't it, try running a "System Usage" status?  I cannot exactly remember, because 2018 FW is pretty old, but I think the newer version of this status shows the CPU per task.   What we're looking for is maybe a particular task (more than likely 'main') taking up a lot of CPU.

If the task taking up the CPU is "webgui" or "Quota Enforcement", then we have to go another direction.

If it is the 'main' task, then run the 'Stylesheet Executions' on all domains (go to default domain, search for "StyleSheet Executions", then select "Show All Domains".  This will lead you to the stylesheet more than likely consuming the most resources.  See if one or more stand out in the 10sec or 1min measure.

Another status that might prove helpful is the "Stylesheet Status" (again for all domains).  You're looking for failed status or huge compile times, which could lead to problems.

I wouldn't think it is a document issue with respect to traffic, because after restarting, that traffic should have shown up on another appliance (I suspect you're clustered) and you'd have seen the CPU spike there.   So, if it is a document problem, it is related to some kind of XML used for configuration.  In this case you'd be looking for differences in your XML or other config type documents between the affected appliance and those not affected.

I'll stop here.  This can be difficult to figure out.  But this should get you started.

Hi all, 

I am seeing suddent spike up in CPU on IDG 2018.x.x version on VMWare. Any idea what could be the major cause? What checks can be done? 
I have tried shutdown reboot which did not work. I did stylesheet & document cache clearing as well which failed to resolve issue ultimately. 

The issue started suddenly after the midnight and all before that the appliances were pretty stable. 

Please note that we do have 6 other datapowers on IDG 2018.x.x and all are very stable.

Any help would be appreaciated. 

Thanks ! 
Prathamesh

Hi Joseph, 

Thanks for these details. Much appreciated. I did verified initial suggestions. Nothing related to it. 

However, when it comes to stylesheet status. I do see compiler logs for many stylesheets. Mostly saying "Error at local:///file: Construct is not streamable." 
Any idea what this error is? 

Also in compile options I do see message as : XSLT 1.0, failed to stream input 

I am not sure if these were occuring before as this is the first time I am exploring this part. 

Thanks again ! 
Prathamesh 

The first question will be, has anything changed in the affected machine?

A good place to start is the "Hypervisor Status".  See if something happened to the #cores or # of virtual CPUs.  Since it suspiciously changed overnight, if there were no other known changes, maybe something changed there.

If that isn't it, try running a "System Usage" status?  I cannot exactly remember, because 2018 FW is pretty old, but I think the newer version of this status shows the CPU per task.   What we're looking for is maybe a particular task (more than likely 'main') taking up a lot of CPU.

If the task taking up the CPU is "webgui" or "Quota Enforcement", then we have to go another direction.

If it is the 'main' task, then run the 'Stylesheet Executions' on all domains (go to default domain, search for "StyleSheet Executions", then select "Show All Domains".  This will lead you to the stylesheet more than likely consuming the most resources.  See if one or more stand out in the 10sec or 1min measure.

Another status that might prove helpful is the "Stylesheet Status" (again for all domains).  You're looking for failed status or huge compile times, which could lead to problems.

I wouldn't think it is a document issue with respect to traffic, because after restarting, that traffic should have shown up on another appliance (I suspect you're clustered) and you'd have seen the CPU spike there.   So, if it is a document problem, it is related to some kind of XML used for configuration.  In this case you'd be looking for differences in your XML or other config type documents between the affected appliance and those not affected.

I'll stop here.  This can be difficult to figure out.  But this should get you started.

Hi all, 

I am seeing suddent spike up in CPU on IDG 2018.x.x version on VMWare. Any idea what could be the major cause? What checks can be done? 
I have tried shutdown reboot which did not work. I did stylesheet & document cache clearing as well which failed to resolve issue ultimately. 

The issue started suddenly after the midnight and all before that the appliances were pretty stable. 

Please note that we do have 6 other datapowers on IDG 2018.x.x and all are very stable.

Any help would be appreaciated. 

Thanks ! 
Prathamesh

Also to add on, for most of these stylesheets I can see execution time between 10 sec, 1 min, 10min and even for 1 day. If this should not be the case, then I wonder what causing them to be here even after stylesheet cache clearing.

Hi Joseph, 

Thanks for these details. Much appreciated. I did verified initial suggestions. Nothing related to it. 

However, when it comes to stylesheet status. I do see compiler logs for many stylesheets. Mostly saying "Error at local:///file: Construct is not streamable." 
Any idea what this error is? 

Also in compile options I do see message as : XSLT 1.0, failed to stream input 

I am not sure if these were occuring before as this is the first time I am exploring this part. 

Thanks again ! 
Prathamesh 

The first question will be, has anything changed in the affected machine?

A good place to start is the "Hypervisor Status".  See if something happened to the #cores or # of virtual CPUs.  Since it suspiciously changed overnight, if there were no other known changes, maybe something changed there.

If that isn't it, try running a "System Usage" status?  I cannot exactly remember, because 2018 FW is pretty old, but I think the newer version of this status shows the CPU per task.   What we're looking for is maybe a particular task (more than likely 'main') taking up a lot of CPU.

If the task taking up the CPU is "webgui" or "Quota Enforcement", then we have to go another direction.

If it is the 'main' task, then run the 'Stylesheet Executions' on all domains (go to default domain, search for "StyleSheet Executions", then select "Show All Domains".  This will lead you to the stylesheet more than likely consuming the most resources.  See if one or more stand out in the 10sec or 1min measure.

Another status that might prove helpful is the "Stylesheet Status" (again for all domains).  You're looking for failed status or huge compile times, which could lead to problems.

I wouldn't think it is a document issue with respect to traffic, because after restarting, that traffic should have shown up on another appliance (I suspect you're clustered) and you'd have seen the CPU spike there.   So, if it is a document problem, it is related to some kind of XML used for configuration.  In this case you'd be looking for differences in your XML or other config type documents between the affected appliance and those not affected.

I'll stop here.  This can be difficult to figure out.  But this should get you started.

Hi all, 

I am seeing suddent spike up in CPU on IDG 2018.x.x version on VMWare. Any idea what could be the major cause? What checks can be done? 
I have tried shutdown reboot which did not work. I did stylesheet & document cache clearing as well which failed to resolve issue ultimately. 

The issue started suddenly after the midnight and all before that the appliances were pretty stable. 

Please note that we do have 6 other datapowers on IDG 2018.x.x and all are very stable.

Any help would be appreaciated. 

Thanks ! 
Prathamesh

Hi all, 

I am seeing suddent spike up in CPU on IDG 2018.x.x version on VMWare. Any idea what could be the major cause? What checks can be done? 
I have tried shutdown reboot which did not work. I did stylesheet & document cache clearing as well which failed to resolve issue ultimately. 

The issue started suddenly after the midnight and all before that the appliances were pretty stable. 

Please note that we do have 6 other datapowers on IDG 2018.x.x and all are very stable.

Any help would be appreaciated. 

Thanks ! 
Prathamesh

------------------------------
Prathamesh Dixit
------------------------------