Cognos Analytics

 View Only
Expand all | Collapse all

Authentication stops working for certain users

  • 1.  Authentication stops working for certain users

    Posted Mon February 01, 2021 07:19 PM
    I have a service account that I used for building cubes on a tools server.  The account authenticates to our Production environment (11.1.7 FP2) nightly during the build process. Recently this account stops being authenticated by Cognos and the cubes fail to build.  The account hasn't been locked out in AD, only Cognos doesn't like it. I tested the account against our DEV server and it works fine (it connects to the same AD system).  This authentication issue happened 3 days ago and to fix it I had to restart the Cognos server. It happened again last night.  Is there a way to fix the issue without restarting the Cognos service?  Does anyone know why this is happening? BTW the service account has been used for over 5 years, never having issues like this.

    I hope someone has a clue to this issue as I don't like taking the system down regularly to fix authentication.
    Thanks in advance everyone.

    ------------------------------
    Scott
    Cognos Administrator
    Wenatchee WA
    ------------------------------

    #CognosAnalyticswithWatson


  • 2.  RE: Authentication stops working for certain users

    IBM Champion
    Posted Tue February 02, 2021 12:47 AM
    Enable debug level on logging of AAA sessions in order to understand details?

    Up to 11.0.7 you could use the ipfclientconfig.xml files for debugging AAA issues.
    Since then this has moved to the portal: Manage > Configuration > Diagnostic Logging > Enable AAA

    This will log lots of lines per login to your cognosserver.log whenever authentication is involved:
    cat /opt/IBM/cognos/crn0/logs/cognosserver.log | grep -i ".AAA" | grep -i trace

    I would schedule the activation/de-activation of AAA logging to the time when the cube is build. This would prevent performance issues and also strip down the log size a bit.

    Enable logging (on Unix)  by:
    * echo '{"logsEnabled":[{"topicName":"AAA","topicType":"BUILTIN"}],"socketAppenderHostname":"localhost","meta":{"type":"glugLoggingConfiguration","version":"0.0.1"},"socketAppenderPort":-1}' > /opt/IBM/cognos/crn0/logs/cognosserver.log

    Back to default:
    * echo '{"logsEnabled":[{"topicName":"DEFAULT","topicType":"BUILTIN"}],"socketAppenderHostname":"localhost","meta":{"type":"glugLoggingConfiguration","version":"0.0.1"},"socketAppenderPort":-1}'  > /opt/IBM/cognos/crn0/logs/cognosserver.log

    ​* Link to documentation 


    ------------------------------
    Ralf Roeber
    https://linkedin.com/in/ralf-roeber-470425a/
    ------------------------------



  • 3.  RE: Authentication stops working for certain users

    Posted Tue February 02, 2021 10:37 AM
    Scott,

    You could be running out of resources for the authentication service due to increased usage patterns or a new bug. Try checking what the Java process is using over time in the administration console. We had to increase the Maximum memory for the Java process on our primary Content Manager to 12 GB to prevent the occasional fail over when processing a lot of batch jobs including data set refreshes.

    ------------------------------
    Robert Hofstetter
    ------------------------------



  • 4.  RE: Authentication stops working for certain users

    Posted Wed February 03, 2021 03:03 PM
    Thanks for the ideas Robert.  Just wondering where in the admin console you would see the Java usage?  Each of our two Cognos servers have 8 cores and 32 GB ram which I would think shouldn't be an issue. After I posted the question I had another day that the authentication stopped working for my service account. IBM hasn't responded to my SR, so that is still out there.  
    To fix the issue this time I stopped the Cognos service then rebooted the server. This time after the reboot the number of Cognos services running were 15 but prior to rebooting the count was 17. Still only 3 JAVA processes in each case.

    Thanks for pointing me to the admin console area to check.

    ------------------------------
    Scott
    Cognos Administrator
    Wenatchee WA
    ------------------------------



  • 5.  RE: Authentication stops working for certain users

    IBM Champion
    Posted Thu February 04, 2021 12:25 PM
    Hi Scott,

    Have you had any Cryptographic Key issues before, I have had random issues with access before due to the Crypto key expiry lifetime kicking in. Can you save the configuration on the Active Content Manager server without any errors?

    Regards
    Andy

    ------------------------------
    Andrew Copeland
    ------------------------------



  • 6.  RE: Authentication stops working for certain users

    Posted Thu February 04, 2021 01:37 PM

    Thanks for the idea Andy,

    I was successful in saving the configuration.

    • Checking for errors and configuration integrity
    • Generating cryptographic information
    • Checking integrity of encrypted data
    • Checking upgrade status

     

    No issues were noted.

     

    I have found that when the authentication has stopped on the primary server I happen to have 2 additional processes running that are not there after a restart.  I have one extra conhost.exe and one extra BIBusTKServerMain.exe.  Not sure if that should be happening but just something I noted.

     

    Thanks,

    Scott

     






  • 7.  RE: Authentication stops working for certain users

    IBM Champion
    Posted Thu February 04, 2021 06:06 PM
    Hi Scott,

    The additional BIBUSTKSERVERMAIN process will be a rogue report process that has collapsed and chances are created a very large *.dmp file in your bin or bin64 folder. It sounds like one of your daily processes in terms of reporting is causing such an issue that it is crashing and causing subsequent instabilities.

    As a baseline go to the bin64 directory and delete any rubbish out of there like dmp files. I wrote an article on C10 BI Server cleanup a few years ago at Maintenance of IBM Cognos 10 BI Servers – allthingscognos (wordpress.com) so if you follow the steps for section 1 that will ensure you start with a clean slate.

    Restart your services and see when *.dmp files get created, this will help you identify a faulting report, the dmp files can also be useful for IBM Support to work out defects but they do fill up your hard drive if not managed.

    Regards
    Andy

    ------------------------------
    Andrew Copeland
    ------------------------------



  • 8.  RE: Authentication stops working for certain users

    Posted Thu February 04, 2021 07:37 PM

    Andrew,

    Thanks for this information.  I remember your article and have on a regular basis checked the BIN and BIN64 folders for dump files.  This time I did not find any, nor any of the other files you suggested. 

     

    Since I have a primary and standby content server, I switched them to see if they would solve the problem. It did for awhile today, allow me to build several cubes.  After lunch authentication stopped working again and the transformer build hung up.  Basically it was not authenticating to the Cognos server.  Checked the task manager on primary and standby and now I don't see any BIBusTKServerMain.exe process. This process has just dropped off.  This is really strange what is happening to the processes.  I also double checked the report processes and there are no jobs or tasks running.  Most work is interactive with accounting using the Excel add-in and cubes sources.

     

    I appreciate your response. Thanks.

    Scott

     

    Bar chart

    Scott Taylor

    Cognos Administrator | Senior BI Analyst

    Desk: (509) 661-4332

    Mobile: (509) 630-1733

     






  • 9.  RE: Authentication stops working for certain users

    Posted Fri February 05, 2021 11:07 AM
    Scott,

    Depending on how you have the environment tuned, 32gb is not very much. Since you mentioned having 8 cores I will step out on a limb and assume you increased maximum processes to utilize those cores. IBM does document the recommended memory settings. Let's walk through a scenario where you've increased those to 8 each for interactive and batch:

    2gb for OS + 8gb for WLP (aka dispatcher) + 8gb for Query Service + 8gb for Dataset Service + 2gb for Graphics Service + 2gb * 16 for BIBus = 60gb.

    Even if you had not increased the maximum processes you would run the risk of running out. On top of that, if you are like many of us, you've had to increase the Query Service maximum allocation. I think you should monitor the system resources closely to see if memory is running out.

    ------------------------------
    Robert Hofstetter
    ------------------------------



  • 10.  RE: Authentication stops working for certain users

    Posted Fri February 05, 2021 06:20 PM

    Robert,

    I got a response from IBM Support stating I should make the following changes in Cognos Configuration.

    Under Advance properties of the namespace add the following:

    handleExpiry=600

    cacheTimeToLive=300

    cacheSize=1048576

    save and restart service. 

     

    I will have to wait until after hours to implement. Requested from IBM to explain what these changes are doing but I'm sure that will be a day or two before I hear.

    Just thought I'd share what I have learned and wait to see what happens after the changes.

     

    Scott

     

    Bar chart

    Scott Taylor

    Cognos Administrator | Senior BI Analyst

    Desk: (509) 661-4332

    Mobile: (509) 630-1733

     






  • 11.  RE: Authentication stops working for certain users

    IBM Champion
    Posted Fri February 05, 2021 06:36 PM
    Hi Scott,
    I haven't come across those settings before but would be interested to know if they work and if so what they do. Appreciate you sharing this.
    Regards
    Andy

    ------------------------------
    Andrew Copeland
    ------------------------------