IBM Verify

IBM Verify

Join this online user group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  ISVG IM SW 10.0.0.4 Pending processes - concurrency_lock

    Posted Tue September 19, 2023 08:15 AM

    ISVG IM SW 10.0.0.4 

    very often (daily) we have requests that remain in pending status (for example, when modifying an account). In the WAS log SystemOut there are SIBMJMSRAThreadPool warning messages about hanging threads. This behavior is not known from any of the previous ISVG versions. There are lower hundreds of requests in the application and the only solution is to restart the application.Has anyone solved something similar? There is nothing non-standard deployed in the application.

    W   WSVR0605W: Thread "SIBJMSRAThreadPool : 119" (00000330) has been active for 637372 milliseconds and may be hung.  There is/are 37 thread(s) in total in the server that may be hung.
            at com.ibm.itim.concurrency.ConcurrentLock.lock(ConcurrentLock.java:295)
            at com.ibm.itim.remoteservices.ejb.mediation.Connector.modify(Connector.java:2552)
            at com.ibm.itim.workflowextensions.RemoteServicesAdapter.modifyAccount(RemoteServicesAdapter.java:462)
            at com.ibm.itim.workflowextensions.PolicyFulfillmentAdapter.modifyAccount(PolicyFulfillmentAdapter.java:451)
            at com.ibm.itim.workflowextensions.PolicyFulfillmentAdapter.fulfillComplianceEnforcement(PolicyFulfillmentAdapter.java:194)
            at com.ibm.itim.workflowextensions.AccountExtensions.modifyAccount(AccountExtensions.java:1017)
            at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
            at java.lang.reflect.Method.invoke(Method.java:508)
            at com.ibm.itim.workflow.engine.ApplicationActivityExecutor.execute(ApplicationActivityExecutor.java:172)
            at com.ibm.itim.workflow.engine.WorkflowEngine.executeActivity(WorkflowEngine.java:2948)
            at com.ibm.itim.workflow.engine.WorkflowEngine.processMessage(WorkflowEngine.java:619)
            at com.ibm.itim.workflow.engine.ExecutionContext.processMessage(ExecutionContext.java:1088)
            at com.ibm.itim.workflow.engine.MessageRouter.onMessage(MessageRouter.java:86)
            at com.ibm.itim.messaging.mdb.MessageHandlerBean.handleMessage(MessageHandlerBean.java:166)
            at com.ibm.itim.messaging.mdb.EJSLocalStatelessenroleejb_ContainerManagedMessag_ae956b4e.handleMessage(Unknown Source)
            at com.ibm.itim.messaging.mdb.TransactedMessageListenerBean.handleMessage(TransactedMessageListenerBean.java:272)
            at com.ibm.itim.messaging.mdb.TransactedMessageListenerBean.onMessage(TransactedMessageListenerBean.java:197)
            at com.ibm.itim.messaging.mdb.MDBProxyenroleejb_LocalWorkflowMDB_7198f191.onMessage(MDBProxyenroleejb_LocalWorkflowMDB_7198f191.java)
            at com.ibm.ws.sib.api.jmsra.impl.JmsJcaEndpointInvokerImpl.invokeEndpoint(JmsJcaEndpointInvokerImpl.java:258)
            at com.ibm.ws.sib.ra.inbound.impl.SibRaDispatcher.dispatch(SibRaDispatcher.java:919)
            at com.ibm.ws.sib.ra.inbound.impl.SibRaSingleProcessListener$SibRaWork.run(SibRaSingleProcessListener.java:597)
            at com.ibm.ejs.j2c.work.WorkProxy$RunWork.run(WorkProxy.java:283)
            at java.security.AccessController.doPrivileged(AccessController.java:717)
            at javax.security.auth.Subject.doAs(Subject.java:490)
            at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:133)
            at com.ibm.ejs.j2c.work.WorkProxy$RunWork.run(WorkProxy.java:286)
            at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
            at com.ibm.ejs.j2c.work.WorkProxy.run(WorkProxy.java:668)
            at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909)



    ------------------------------
    Jakub Nejdl
    ------------------------------


  • 2.  RE: ISVG IM SW 10.0.0.4 Pending processes - concurrency_lock

    Posted Wed September 20, 2023 02:16 AM

    It is very difficult to backtrack these kind of errors based on the SIB hang exception only. There can be a lot of reasons why this pops up - but normally this is caused by having too many connections from WAS to Db2. And this again can be caused either by load on the ISVG IM side (many workflows running concurrently) or Db2 not performing well.

    I would check the database first too see if there is anything obvious there - then the WAS Db2 connection parameters to see if they are adequate. You may also need to get some help from IBM Support through a case....

    Have you cleaned the pending transactions (https://www.ibm.com/support/pages/technique-removing-pending-transactions-isim-6x-7x-or-svg-im-10x-system) ? 

    That is normally needed after such errors as the pending requests will still exist in the SIB tables after a restart of the environment

    HTH



    ------------------------------
    Franz Wolfhagen
    WW IAM Solution Engineer - Certified Consulting IT Specialist
    IBM Security Expert Labs
    ------------------------------

    Original Message


  • 3.  RE: ISVG IM SW 10.0.0.4 Pending processes - concurrency_lock

    Posted Sun September 24, 2023 08:10 AM

    Thank you for your response.

    it looks like there is no problem with the database. db2diag log does not print any error messages when the application is stuck. At the same time, the problem always occurs e.g. during AD service reconciliation, when AD returns a larger number of errors e.g. on AD group assignments that no longer exist. The SIBJMSRAThreadPool is exhausted - the adapter does not receive any new requests from IdM and the requests remain hanging in the pending state in the audit. In previous versions of IdM such as ISIM 6.0, this was never a problem and the application was able to deal with it.



    ------------------------------
    Jakub Nejdl
    ------------------------------

    Original Message


  • 4.  RE: ISVG IM SW 10.0.0.4 Pending processes - concurrency_lock

    Posted Mon September 25, 2023 03:17 AM

    I would guess a lot of the problems is coming from the point that sub processes fails and is not catched in the workflow due to groups not existing any more.

    You should of course fix those problems - but there is probably a deeper problem of some uncaught exceptions that may deserve a case to IBM Support... 



    ------------------------------
    Franz Wolfhagen
    WW IAM Solution Engineer - Certified Consulting IT Specialist
    IBM Security Expert Labs
    ------------------------------

    Original Message


  • 5.  RE: ISVG IM SW 10.0.0.4 Pending processes - concurrency_lock

    Posted Wed September 27, 2023 09:57 AM

    We had this same exact problem with 10.0.0.4.  There seems to be an internal issue with the way concurrency is handled with that version. We had all kinds of issues with AD, and some other adapters pending for eternity.   We were able to pinpoint the issue by creating a custom adapter that did nothing but sleep for a few seconds, to simulate an account "processing."  When submitting transactions for the same account before the first one competed, it would end up pending forever.

    After spending months troubleshooting with IBM support, they were finally able to replicate the issue in that version.  However, later versions of SVG did not showcase the same issue.  We ended up having to do two upgrades (one to get to 10.0.1, then another to get to 10.0.1.4), which resolved the issue.

    My colleague posted about this here, and updated it when we found a resolution: https://community.ibm.com/community/user/security/discussion/ibm-security-verify-governance-identity-manager-10004-concurrency-issue

    Hope that helps, and good luck!



    ------------------------------
    Abbie Harrison
    ------------------------------

    Original Message


Related Content