Db2

Expand all | Collapse all

An Internal error occours - summary page not working after upgrading to 3.1.1

  • 1.  An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Thu February 06, 2020 02:57 PM
    Edited by System Fri January 20, 2023 04:45 PM
    Hi DMC team

    Have just upgraded DMC to 3.1.1,  after re-connecting to my existing previous repository db, getting this error in the summary page. 

    And none of the graphs in summary pages works anymore.  They were working fine under 3.1.0 release. 

    In ds_system.0*  logs I can see this only error, but not sure if its related or not. 
    DS_System       (4430)                  Feb 6, 2020 5:39:44 PM          INFO    Use intenral token for default service /internal/credentials/monitor/NOXGR
    DS_System       (3672)                  Feb 6, 2020 5:39:44 PM          ERROR    The integration internal token can only be used in allowed services./internal/credentials/operation/NOXGR
    DS_System       (3672)                  Feb 6, 2020 5:39:44 PM          ERROR    "URLAccessHandler:: JWTValidation failed.
    com.auth0.jwt.exceptions.JWTVerificationException: Invalid internal token.
            at com.ibm.aps.tools.http.util.jwt.InternalTokenAgent.verifyInternalToken(InternalTokenAgent.java:291)
            at com.ibm.aps.tools.http.util.jwt.JWTValidationUtil.validateInternalToken(JWTValidationUtil.java:411)
            at com.ibm.aps.tools.http.util.jwt.JWTValidationUtil.validateToken(JWTValidationUtil.java:158)
            at dsweb.filter.URLAccessServletFilter.isAccessAllowed(URLAccessServletFilter.java:737)
            at dsweb.filter.URLAccessServletFilter.handleURLAccess(URLAccessServletFilter.java:461)
            at dsweb.filter.URLAccessServletFilter.doFilter(URLAccessServletFilter.java:175)
            at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
            at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:90)
            at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:993)
            at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1131)
            at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1002)
            at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:75)
            at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:938)
            at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:279)
            at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1136)
            at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:417)
            at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:376)
            at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:547)
            at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:481)
            at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:346)
            at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:317)
            at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1077)
            at com.ibm.ws.channel.ssl.internal.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:656)
            at com.ibm.ws.channel.ssl.internal.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1803)
            at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:503)
            at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:573)
            at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:954)
            at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1043)
            at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.lang.Thread.run(Thread.java:818)
    ​





    ------------------------------
    Samuel Pizarro
    ------------------------------
    #Db2


  • 2.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Thu February 06, 2020 08:51 PM
    Hi Samuel,
    We've received your problem, and will investigate and update with you ASAP. Thanks.

    ------------------------------
    Li Hui Yan (Autumn Yan)
    From DMC Team
    ------------------------------



  • 3.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Thu February 06, 2020 09:40 PM
    Hi Samuel,

    Could you please help to provide following info? thanks.

    1) Please open browser "developer tools", and refresh the summary page again, so that to locate which request caused the error;

    2) The detailed error log may be found in "monitor.*" log files, please help provide the info (in the same folder as DS_System.*).

    ------------------------------
    KAI DING
    ------------------------------



  • 4.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Fri February 07, 2020 07:11 AM
    Hi @KAI DING

    1. I am not sure what part of ​"Developer Tools" section details you want..  I hope it`s the console log.  I have saved the console output.  Please, let me know if this is not what you are looking for,  and provide more details on how to get it.  I have uploaded the file in this post. 

    2. The "monitor.*" files are all empty.   here is the current log files inside logs subdir 
    PS C:\IBM_DB2_DMC\ibm-datamgmtconsole\logs> dir
    
    
        Directory: C:\IBM_DB2_DMC\ibm-datamgmtconsole\logs
    
    
    Mode                LastWriteTime         Length Name
    ----                -------------         ------ ----
    d-----       06/02/2020     09:53                ffdc
    -a----       06/02/2020     09:32              0 admin.0
    -a----       06/02/2020     09:32              0 admin.0.lck
    -a----       06/02/2020     17:03              0 alert.0
    -a----       06/02/2020     09:32              0 alert.0.lck
    -a----       06/02/2020     17:03       10518184 alert.1
    -a----       06/02/2020     09:31              0 console.log
    -a----       06/02/2020     09:33            225 console.token
    -a----       06/02/2020     09:33            695 DS_System.0
    -a----       06/02/2020     09:32          84535 DS_System.0.1
    -a----       06/02/2020     09:32              0 DS_System.0.1.lck
    -a----       06/02/2020     09:33            695 DS_System.0.2
    -a----       06/02/2020     09:32              0 explain.0
    -a----       06/02/2020     09:32              0 explain.0.lck
    -a----       06/02/2020     09:33            233 integration.token
    -a----       06/02/2020     09:33              0 LOAD.0
    -a----       06/02/2020     09:33              0 LOAD.0.lck
    -a----       06/02/2020     09:31              0 messages.log
    -a----       06/02/2020     09:32              0 monitor.0
    -a----       06/02/2020     09:32              0 monitor.0.lck
    -a----       06/02/2020     09:48              0 repoUtil.0
    -a----       06/02/2020     09:48              0 repoUtil.0.lck
    -a----       06/02/2020     09:32              0 runsql.0
    -a----       06/02/2020     09:32              0 runsql.0.lck
    -a----       06/02/2020     09:33              0 sample.0
    -a----       06/02/2020     09:33              0 sample.0.lck
    -a----       06/02/2020     09:33           1045 setup.0
    -a----       06/02/2020     09:33            170 setup.0.1
    -a----       06/02/2020     09:32              0 tuning.0
    -a----       06/02/2020     09:32              0 tuning.0.lck​

    Do you want them all ?

    ------------------------------
    Samuel Pizarro
    ------------------------------



  • 5.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Sun February 09, 2020 08:30 PM
    Hi Samuel,

    1) About "Developer Tools", I mean the browser tool which can show all HTTP requests in browser. Take Chrome as an example: View -> Developer -> Developer Tools. The purpose is to find out which HTTP request caused the "Internal Errors" shown in page.

    2) About the empty files, seems DMC is not working as expected, and it may be related to the DMC upgrade. Could you provide more details about how you did the upgrade, and was there anything wrong during the upgrade?

    ------------------------------
    KAI DING
    ------------------------------



  • 6.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Mon February 10, 2020 12:09 PM
    Hi @KAI DING

    Yes,  that was the log I have provided...  from Console output ,  in developer tools. 
    but here is a new picture of it.. 

    ​​
    2)  I have described my upgrade scenario,  in other thread in this forum . 

    Basically,  my original 3.1.0 setup was done by windows GUI installer,  and we have no option to upgrade that. The only option is by script-installer interface.   So,  when the zip was extracted in the original setup.  none of the files got replaced,  as they dir layout does not match.  
    so, when setup of 3.1.1 was executed,  it performed a clean-setup instead of upgrading.    

    Later on,  when I first activated DMC and logged in,  it prompted me to setup the repository (so it really did not detect the previous one during the upgarde).  when I pointed my original repository,  it asked me something like this : 

    "The repository was used by a previous version of DMC,  do you want to keep existing setup,  or start a fresh new ? "

    I asked to preserve the settings 

    since them,  monitor is not working properly. 

    I have stopped and restarted DMC one more time today.   and now it looks it wrote somehing to the logs...  

    I have uploaded them  logs.zip

    Regards



    ------------------------------
    Samuel Pizarro
    ------------------------------



  • 7.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Tue February 11, 2020 05:55 AM
    Hi Samuel,

    Thanks for your update. We are investigating the issue, and will let you know the update. thanks.

    ------------------------------
    KAI DING
    ------------------------------



  • 8.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Wed February 12, 2020 10:45 PM
    Hi Samuel,

    We've re-validated the upgrade scenario just as you described, and it works well. So the process should be fine, and the issue should be specific in your env.

    Do you still hit the "-404" error in home page after console restart? And is the "internal error" message gone?

    Also in your logs I find a lot of "Db2 -964 transaction log full" error. It would make monitor functionality down. Please consider to fix it and restart Db2 if necessary. thanks.

    ------------------------------
    KAI DING
    ------------------------------



  • 9.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Thu February 13, 2020 11:01 AM
    Hi @KAI DING

    Thanks for pointing me to the -964 sql error​.  I have increased the log space in the repo db.  
    Note: I did not find any where the logs and db parameters required in DMC documentation for the repository db.  That's why it was unset.  
    I now used the recommendations from DSM docs,  and have increased the log space.  hope this will be enough. 

    Do you still hit the "-404" error in home page after console restart? And is the "internal error" message gone?
    After increasing the logs in the repo,  I have restarted DMC,  but still receiving -401 error 
    :11090/dbapi/v4/internal/users/roles:1 Failed to load resource: the server responded with a status of 401 (Unauthorized)
    
    TEXTLOG 12:53:28 [INFO] [EventService] - token expired or logged out
    :11090/dbapi/v4/internal/settings_ext:1 Failed to load resource: the server responded with a status of 401 (Unauthorized)​


    and the issue should be specific in your env.
    Well,  this was working exactly fine with DMC 3.1.0 .  Same server, same repo,  same everything,  just DMC got updated to 3.1.1 release. 
    I can accept that it could be something in my env, but not being able to identify it. 

    What else I can do, or what else I can provide you in order to troubleshoot this ? 

    Regards


    ------------------------------
    Samuel Pizarro
    ------------------------------



  • 10.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Thu February 13, 2020 12:14 PM
    Looking at monitor and ds_system logs,  found this particular one which caught my attention

    monitor	(192)			Feb 13, 2020 1:16:25 PM		INFO    Mymon MonitorCollectorRunnable::run:  The collection elapsed time for NOXGR is 5627ms
    monitor	(192)			Feb 13, 2020 1:16:25 PM		INFO    >>>>>>>>>>>>>>>>>>>>> send collectDone ThirtyMin:NOXGR
    monitor	(278)			Feb 13, 2020 1:16:25 PM		INFO    <<<<<<<<<<<<<<<<  get collectDoneNOXGR:ThirtyMin
    monitor	(336)			Feb 13, 2020 1:16:25 PM		INFO    [createConnectionInternal] url is: jdbc:db2://localhost:50000/DCMREPO:
    monitor	(331)			Feb 13, 2020 1:16:25 PM		ERROR    HomeQueueConsumer::whetherDbProfileStatusChanged: homeDataLast1h is null. So don't push data to UI
    monitor	(337)			Feb 13, 2020 1:16:25 PM		INFO    [createConnectionInternal] url is: jdbc:db2://localhost:50000/DCMREPO:
    monitor	(193)			Feb 13, 2020 1:16:25 PM		INFO    NOXGR_193:TableSpace DMC1001 after collect()
    monitor	(193)			Feb 13, 2020 1:16:25 PM		INFO    NOXGR_193:Connections DMC1001 before collect()​
    specially the "So don't push data to UI" part. 




    ------------------------------
    Samuel Pizarro
    ------------------------------



  • 11.  RE: An Internal error occours - summary page not working after upgrading to 3.1.1

    Posted Thu April 02, 2020 09:37 AM
    Hi DMC Team

    after working with other DMC team support in background,  we have identified the real root cause of this issue.  and its related to java OOM condition being hit. 

    More details can be found in another thread of this forum 

    Timeout Errors - fresh new install



    so,  I believe we can close this one.

    ------------------------------
    Samuel Pizarro
    ------------------------------