Very interesting. Thanks!
I got close to the root cause and confirmed with IBM L2 support. I'm not entirely sure if it's truly deadlocking on CONCURRENT_LOCK, but the usage of that table in v10.0.0.4 does not behave well when two operations are pending for the same account.
Steps to replicate:
- Take any existing adapter that can be modified easily (or create your own). I recommend an RMI/SDI based adapter.
- In the adapter and for a given operation (Add, Modify, etc) put in somewhere to sleep transaction thread for a specific amount of time. Sixty seconds should be sufficient. I recommend the modify operation.
- Submit a request in SVG for that adapter and that operation (add, modify, etc). This should tell SVG to fire off a request to the adapter and the adapter hold off on doing anything for the time you put in. This causes SVG to keep the transaction in pending status.
- While the first transaction is still pending, do the same operation for the same account again.
This causes both transactions to remain in pending status indefinitely. Interestingly, the account operations do complete on the managed resource. They just never get noted as completed in SVG's DB2 and thus remain in pending.
Small update: v10.0.1.4 and v10.0.1.5 do not showcase this issue. In either version, the above series of events cause the second operation to wait on the first before firing. We are migrating from v10.0.0.4 to v10.0.1.4 to remove this inconsistency.
------------------------------
Brandon
------------------------------
Original Message:
Sent: Wed July 12, 2023 02:51 AM
From: Felipe Risalde Serrano
Subject: IBM Security Verify Governance Identity Manager 10.0.0.4 Concurrency Issue
Hi Brandon,
we had a similiar issue working with ISIM 6 in 2017, but it was not relative to the concurrency, just some operations kept pending forever and never finished, although all their child's had done.
It was fixed with a FP, but meanswhile we had a scheduled process which was responsable to finish the activities and processes by updating the database.
In our case, it happened with MODIFYPERSON and HUMAN RECONCILIATION operations and ENFORCEPOLICYFORPERSON and RECONCILIATION activities.
Find enclosed the queries if it could be for your interest.
------------------------------
Felipe Risalde Serrano
Security Expert
Banco de España
Original Message:
Sent: Tue July 11, 2023 10:27 AM
From: Brandon
Subject: IBM Security Verify Governance Identity Manager 10.0.0.4 Concurrency Issue
I have been working with IBM support with an issue regarding request concurrency.
Series of events:
- A person record change is submitted and causes account1 to be created.
- While the first event is midflight (waiting on account1 to be created), the person record is updated again - spawning a new root process and subsequent account creation.
This appears to cause a race condition or deadlock- neither requests finish and are left in pending status indefinitely.
When looking at db2 locks, i see requests to insert into the CONCURRENT_LOCKS table, but i don't see any data in that table. I've confirmed that the permissions are correct for SVG on the entire database and that table (DBADM).
We've been struggling with this for a few months now and with multiple support cases. Prior I could not figure out what events led to the pending transactions, but it wasn't until recent i noticed the pattern. It's a low chance scenario with higher than expected impact. In ISIM 6, this did not occur. Only after we upgraded to v10.0.0.4 did we start seeing this.
Hoping to see if anyone else has seen this or if it's better to just let support handle the situation.
------------------------------
Brandon
------------------------------