Planning Analytics

Planning Analytics

Get AI-infused integrated business planning

 View Only
Back to discussions
Expand all | Collapse all

CA threads causing lock contention

  • 1.  CA threads causing lock contention

    Posted Fri January 24, 2020 04:47 PM
    Can anyone shed more light on what CA does when it connects to and queries TM1?

    We assumed CA would be relatively harmless to connect to our production server, but we were very wrong. Within minutes CA threads were causing a large number of waits, blocking perspectives/pax reports, TI runs, and other connections. Plus none of the CA threads could be canceled. 

    Cognos Analytics reports and dashboards should simply read data from TM1, what else is it doing? 

    Does someone have a document that lays out what CA is doing to TM1 when it runs a report, establishes the first connection, or other common interactions?

    ------------------------------
    Ryan Clapp
    ------------------------------

    #PlanningAnalyticswithWatson


  • 2.  RE: CA threads causing lock contention

    Posted Mon January 27, 2020 09:05 AM
    Hi Ryan,

    I would suggest that you log a support case and we can investigate a bit further. 
    I would agree that CA should be read only.  Only thing I can think of is if the user logs in for the first time, we may need to update the }ClientProperties. But it shouldn't have this result.

    Is this with any CA reports? Haven't heard this issue from any other clients so far.

    Kind Regards,

    ------------------------------
    Robert Vautour
    IBM Business Analytics Advocate
    ------------------------------

    Original Message


  • 3.  RE: CA threads causing lock contention

    Posted Mon January 27, 2020 11:15 AM

    While working on building some interactive reports, our PA server became non-responsive and could not stop the server gracefully.  Had to "end process" to restart the service.

    Data source cube had no rules/feeders pointing to/from other PA cubes, it was essentially a stand alone cube with "copy/pasted" values.  Only thing it shares with the other cubes in the server are some dimensions.

    Support review of the logs suggest contentions and rollbacks but why would it cause the server to be non-responsive?

    Support is still doing its review of the logs on this issue.

    I also have another PMR with another CA issue which I suspect is CAPA integration related.



    ------------------------------
    Jose Balitactac
    ------------------------------

    Original Message


  • 4.  RE: CA threads causing lock contention

    Posted Mon January 27, 2020 11:21 AM
    Very similar results here too. My guess is that since the connector is owned by the CA team it does not get the level of review or testing that it should.

    Let me know what comes of your open tickets. We plan on testing again after a few upgrades.


    Ryan Clapp
    AWS Business Systems
    Sr. Manager

    Sent from my mobile device



    Original Message


  • 5.  RE: CA threads causing lock contention

    Posted Tue January 28, 2020 07:18 AM
    Hi Ryan,

    Please see: https://www.ibm.com/support/pages/planning-analytics-server-logging-odataplanning-analytics-workspace

    That debug logging will tell us exactly what Cognos Analytics is asking of the TM1 database.  We should also take a lock at the Lock Exception debug logging to understand what objects are involved.  The answer to what Cognos Analytics is doing will completely depend on the report design.  I can't answer your question without seeing the REST API debug logging (and maybe TM1Top logging) but I can give you an example of why a read only type report can cause lock contention.  When dynamic subsets are referenced in MDX the TM1 Server is actually building (writting) a metadata object.  This can lead to lock contention depending on what dimension the dynamic subset is being used on.  These lock might be less granular than you might expect.  While a dynamic subset is built very quickly, the write lock will remain for the duration of the view execution that used the subset.  For example, the subset may determine which elements to include in a matter of seconds, but if the view that used the subset takes 5 minutes to run, the lock is actually held for 5 minutes.  Again, this is just one potential example of how a read only report can cause lock contention.


    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------

    Original Message


  • 6.  RE: CA threads causing lock contention

    Posted Wed January 29, 2020 03:36 AM
    Hi all,

    I'm not entirely sure if this could be connected to CA locking described by @Ryan Clapp but I reported few weeks ago a defect which will be fixed in 2.0.9.1.
    In this case, I'm using the REST API call, which is only reading data. Please see below the very shortened case description.


    1.  ​​[APAR PH19611 - 2.0.9.1] TM1 Server is dead locking when REST API Dimensions GET is called by TI
     
    We introduced a new process that should run after the server has been restarted.
Unfortunately the TM1 server is deadlocking when a REST API "/api/v1/Cubes('CubeName')/Dimensions?$select=Name" request is called 
directly after the source view definition in the prolog section in the TI.
This situation happens only if the model was restarted, and the process is running the first time.
If we kill the deadlocking tread and re-run the process, then it is not locking anymore till the next restart.

The process which is locking the model is running a REST Request using cURL to get the dimension names of the cube.
Please be aware that cURL is here not the problem, because we are able to reproduce the problem with python as well.


    Additionally, to the above described case, I'm also experiencing crashes when using the REST API in a similar scenario. According to IBM, it should also be fixed in 2.0.9.1.

    1. [APAR PH19605 - 2.0.9.1] TM1 Server is crashing when REST API combined with RunProcess

    We introduced a new process that should run after the server has been restarted.
Unfortunately, the TM1 server is crashing if we using the new RunProcess() function and
executing a rest call, which is writing back data to the same model.

The process which is crashing the model is running a REST Request using cURL to write data to the same model.
Please be aware that cURL is here not the problem, because we are able to reproduce the problem with python as well.

9708 [] ERROR 2019-10-28 13:01:00.296 TM1.Lock Before image is NOT NULL Client, server is not stable
    ​
    Kind Regards,

    ------------------------------
    Tomasz Brzoza
    Linde AG
    Munich
    ------------------------------

    Original Message


  • 7.  RE: CA threads causing lock contention

    Posted Wed January 29, 2020 07:20 AM
    Hi Tomasz,

    For APAR PH19611 - Please try setting ReduceCubeLockingOnDimensionUpdate=T and retest your TI process.  We believe that should work around the problem.  We are advising not to make REST API calls from within Ti processes using scripts.  This has the potential to lead to deadlock situation (as you seem to have encountered).  I suggest following up with support on the status of that APAR, they should confirm the same.

    APAR PH19605 has a codefix that is under review and should make the 2.0.9.1 update.

    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------

    Original Message


  • 8.  RE: CA threads causing lock contention

    Posted Thu February 06, 2020 05:55 AM

    Hi Stuart,

    finally, I found a few minutes to check the ReduceCubeLockingOnDimensionUpdate = T parameter.
    It solved the APAR PH19611 problem.
    Thanks for that.

    I just wondering why this lock is happening and what gets updated if I'm only reading using the GET "/api/v1/Cubes('CubeName')/Dimensions?$select=Name".





    ------------------------------
    Tomasz Brzoza
    Linde AG
    Munich
    ------------------------------

    Original Message


  • 9.  RE: CA threads causing lock contention

    Posted Wed January 29, 2020 02:20 PM

    Stuart,

     

    A clarification please.

    In a theoretical case where there are 2 independent cubes "A" (non-Cognos Reporting cube) & "B" (Cognos Reporting cube), no rules/feeders referencing each other except for a few common dimensions shared between the cubes let's say Account and Cost Center.

    If a user opens up a view in cube "A" that uses a dynamic subset in the Cost Center dimension, will that cause a lock contention in cube B?

    Likewise, when a user opens a view in cube "B" that uses a dynamic subset in a dimension that only exists in cube "B" will that cause a contention.

    Reason I ask is that when I run a CA report our server log makes many entries of "CommitActionLog::Rollback: Called for thread '15724' of user 'CAMID("pans:u:joeblow@ancestry.com")' executing request 'GET /api/v1/Cubes('Rpt Cognos Analytics')/Dimensions('Rpt Period')/Hierarchies('Rpt Period')/Members('Prior12Months')'."

    "Rpt Period" is a dimension used only in the Cognos Reporting cube and "Prior12Months" is a dynamic subset.

     

    Thank you,



    ------------------------------
    Jose Balitactac
    ------------------------------

    Original Message


  • 10.  RE: CA threads causing lock contention

    Posted Thu January 30, 2020 07:23 AM
    Given the log message Prior12Months is a member in this case, not a subset.  Keep in mind that it's possible to have a subset with the same name as a member.  

    If Prior12Months was a subset in this case you would see this:

    GET /api/v1/Cubes('Rpt Cognos Analytics')/Dimensions('Rpt Period')/Hierarchies('Rpt Period')/Subsets('Prior12Months')

    The message you are seeing would happen if the member Prior12Months was removed from the Rpt Period' hierarchy, but your report still references that member name.

    ------------------------------
    Stuart King
    IBM Planning Analytics Offering Manager
    ------------------------------

    Original Message