View Only
  • 1.  High CPU utilization in IDG 2018.x.x

    Posted Thu April 18, 2024 10:11 AM

    Hi all, 

    I am seeing suddent spike up in CPU on IDG 2018.x.x version on VMWare. Any idea what could be the major cause? What checks can be done? 
    I have tried shutdown reboot which did not work. I did stylesheet & document cache clearing as well which failed to resolve issue ultimately. 

    The issue started suddenly after the midnight and all before that the appliances were pretty stable. 

    Please note that we do have 6 other datapowers on IDG 2018.x.x and all are very stable.

    Any help would be appreaciated. 

    Thanks ! 

    Prathamesh Dixit

  • 2.  RE: High CPU utilization in IDG 2018.x.x

    IBM Champion
    Posted Thu April 18, 2024 12:01 PM

    The first question will be, has anything changed in the affected machine?

    A good place to start is the "Hypervisor Status".  See if something happened to the #cores or # of virtual CPUs.  Since it suspiciously changed overnight, if there were no other known changes, maybe something changed there.

    If that isn't it, try running a "System Usage" status?  I cannot exactly remember, because 2018 FW is pretty old, but I think the newer version of this status shows the CPU per task.   What we're looking for is maybe a particular task (more than likely 'main') taking up a lot of CPU.

    If the task taking up the CPU is "webgui" or "Quota Enforcement", then we have to go another direction.

    If it is the 'main' task, then run the 'Stylesheet Executions' on all domains (go to default domain, search for "StyleSheet Executions", then select "Show All Domains".  This will lead you to the stylesheet more than likely consuming the most resources.  See if one or more stand out in the 10sec or 1min measure.

    Another status that might prove helpful is the "Stylesheet Status" (again for all domains).  You're looking for failed status or huge compile times, which could lead to problems.

    I wouldn't think it is a document issue with respect to traffic, because after restarting, that traffic should have shown up on another appliance (I suspect you're clustered) and you'd have seen the CPU spike there.   So, if it is a document problem, it is related to some kind of XML used for configuration.  In this case you'd be looking for differences in your XML or other config type documents between the affected appliance and those not affected.

    I'll stop here.  This can be difficult to figure out.  But this should get you started.

    Joseph Morgan

  • 3.  RE: High CPU utilization in IDG 2018.x.x

    Posted Thu April 18, 2024 12:15 PM

    Hi Joseph, 

    Thanks for these details. Much appreciated. I did verified initial suggestions. Nothing related to it. 

    However, when it comes to stylesheet status. I do see compiler logs for many stylesheets. Mostly saying "Error at local:///file: Construct is not streamable.
    Any idea what this error is? 

    Also in compile options I do see message as : XSLT 1.0, failed to stream input 

    I am not sure if these were occuring before as this is the first time I am exploring this part. 

    Thanks again ! 

    Prathamesh Dixit

  • 4.  RE: High CPU utilization in IDG 2018.x.x

    Posted Thu April 18, 2024 12:18 PM

    Also to add on, for most of these stylesheets I can see execution time between 10 sec, 1 min, 10min and even for 1 day. If this should not be the case, then I wonder what causing them to be here even after stylesheet cache clearing.

    Prathamesh Dixit

  • 5.  RE: High CPU utilization in IDG 2018.x.x

    IBM Champion
    Posted Thu April 18, 2024 02:27 PM

    Did HyperVisor status give you information?   Some time back, again I cannot remember, they changed it to Virtual Platform Status.  So if HyperVisor status isn't available to you, try the other.

    I'm curious, it is actually just "local:///file:"?  It seems like you may have old configurations lingering.  That, I wouldn't think, should affect CPU.

    Clearing stylesheet cache shouldn't affect status.  The best thing clearing stylesheet cache does for you is to enable you to force the update of a stylesheet when the stylesheet is changed directly through the file editor, or by either the local or remote file changing without explicitly changing the object referencing the stylesheet.  That is, if the stylesheet is local and you edit it using the DataPower file editor, or if you upload a new file over it, you may have to clear the stylesheet cache.  If it is remote and is changed at the remote location, then again, clearing the caches forces DataPower to fetch the new copy.

    Do you use GatewayScript by chance?

    Joseph Morgan

  • 6.  RE: High CPU utilization in IDG 2018.x.x

    Posted Fri April 19, 2024 09:31 AM
    Edited by Hermann Stamm-Wilbrandt Fri April 19, 2024 09:31 AM

    Regardless of the problem you stated, you should not run DaatPower firmware that is past end of service.
    2018.4.1 firmware EOS was 4/30/2023 with last fixpack 2018.4.1.26:

    You missed a big number of high severity CVE fixes and many other issues fixed since then.
    Please upgrade to a supported firmware level.

    EOS for 10.0.1 firmware is end of this months, so please do upgrade to a newer firmware level:

    Since you are past EOS, you cannot create a support ticket on whatever issue you have.

    Hermann Stamm-Wilbrandt
    Compiler Level 3 support, IBM DataPower Gateways
    Boeblingen Germany