IBM License Metric Tool (ILMT)

 View Only
Expand all | Collapse all

After update ILMT to 9.2.36.0 does not start

  • 1.  After update ILMT to 9.2.36.0 does not start

    Posted Mon July 01, 2024 09:56 AM

    Hi, just updated (through BigFix) ILMT to V9.2.36 but unfortunately does not work any more.

    When calling the website of ILMT

    https://ilmtsrv2.acegas.loc:9081/

    an error comes back

    Something went wrong.
For more information, see the tema.log file.

    In tema.log I find some msg, like to following:

    [01/07/24 13.39.27:237 UTC] 0000005a com.ibm.ws.webcontainer.webapp                               I SRVE0292I: Messaggio servlet - [tema]:.DEBUG: due a previous initialization failure application instance can not be returned
[01/07/24 13.39.27:237 UTC] 0000005a com.ibm.ws.webcontainer.webapp                               I SRVE0292I: Messaggio servlet - [tema]:.INFO: resetting rack response due exception: org.jruby.rack.RackInitializationException: uninitialized constant ReportSubscription::User
	from org/jruby/RubyModule.java:4331:in `const_missing'

    Will continue searching, but did anyone encounter this problem and knows how to solve it?

    Thanks for your help, Igor



    ------------------------------
    Igor P. Merkù
    ------------------------------


  • 2.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Mon July 01, 2024 09:59 AM

    Hello,

    Please check if you are affected by the following issue:

    https://www.ibm.com/support/pages/node/7158851


    Thank you.



    ------------------------------
    Andrei Ionescu
    IBM
    Bucharest
    ------------------------------

    Original Message


  • 3.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Mon July 01, 2024 10:20 AM

    Hello Andrei,

    thanks a lot

    a) for the fast reply

    b) for suggesting a fix

    c) make ILMT work again

    Thank you very much!

    Kind regards, Igor



    ------------------------------
    Igor P. Merkù
    ------------------------------

    Original Message


  • 4.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Tue July 02, 2024 06:39 AM

    Hi Andrei,

    is the fix of https://www.ibm.com/support/pages/node/7158851 already implemented in version 9.2.36.1
    in LMT-server-9.2.36.1-20240627-1047-windows-x86_64.zip?

    Best regards
    Harald



    ------------------------------
    Tooling42
    ------------------------------

    Original Message


  • 5.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Tue July 02, 2024 06:49 AM

    Hi Harald,

    Yes, the fix is implemented in version 9.2.36.1. 

    Thank you.



    ------------------------------
    Andrei Ionescu
    IBM
    Bucharest
    ------------------------------

    Original Message


  • 6.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Tue July 02, 2024 02:51 PM

    Hello @Andrei Ionescu!

    I´ve waited for 9.2.36.1 but after installing it (the fixlet had the status "fixed") the App on Port 9080 won´t start and I see the following message in the tema.log, is this related? Thanks!

    [7/2/24 14:11:52:125 UTC] 00003b00 com.ibm.ws.kernel.feature.internal.FeatureManager            E CWWKF0004E: An unknown exception occurred while installing or removing features. Exception: java.lang.IllegalStateException: Method called outside of provisioining operation or without a registered service
        at com.ibm.ws.kernel.feature.internal.subsystem.SubsystemFeatureDefinitionImpl.getConstituents(SubsystemFeatureDefinitionImpl.java:294)
        at com.ibm.ws.kernel.feature.internal.FeatureResolverImpl.processSelected(FeatureResolverImpl.java:383)
        at com.ibm.ws.kernel.feature.internal.FeatureResolverImpl.processRoots(FeatureResolverImpl.java:322)
        at com.ibm.ws.kernel.feature.internal.FeatureResolverImpl.processCurrentPermutation(FeatureResolverImpl.java:296)
        at com.ibm.ws.kernel.feature.internal.FeatureResolverImpl.doResolveFeatures(FeatureResolverImpl.java:259)
        at com.ibm.ws.kernel.feature.internal.FeatureResolverImpl.resolveFeatures(FeatureResolverImpl.java:182)
        at com.ibm.ws.kernel.feature.internal.FeatureResolverImpl.resolveFeatures(FeatureResolverImpl.java:114)
        at com.ibm.ws.kernel.feature.internal.FeatureManager.callFeatureResolver(FeatureManager.java:1321)
        at com.ibm.ws.kernel.feature.internal.FeatureManager.resolveFeatures(FeatureManager.java:1305)
        at com.ibm.ws.kernel.feature.internal.FeatureManager.updateFeatures(FeatureManager.java:1430)
        at com.ibm.ws.kernel.feature.internal.FeatureManager.update(FeatureManager.java:777)
        at com.ibm.ws.kernel.feature.internal.FeatureManager.processFeatureChanges(FeatureManager.java:890)
        at com.ibm.ws.kernel.feature.internal.FeatureManager$1.run(FeatureManager.java:677)
        at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:280)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:825)

[7/2/24 14:11:52:133 UTC] 00003f05 com.ibm.ws.tcpchannel.internal.TCPChannel                    I CWWKO0220I: TCP Channel defaultHttpEndpoint has stopped listening for requests on host localhost  (IPv4: 127.0.0.1) port 9080.
[7/2/24 14:11:52:135 UTC] 0000412a com.ibm.ws.tcpchannel.internal.TCPPort                       I CWWKO0219I: TCP Channel tema-ssl has been started and is now listening for requests on host *  (IPv6) port 9081.
[7/2/24 14:11:52:206 UTC] 00003b00 com.ibm.ws.logging.internal.impl.IncidentImpl                I FFDC1015I: An FFDC Incident has been created: "java.lang.IllegalStateException: Method called outside of provisioining operation or without a registered service com.ibm.ws.kernel.feature.internal.FeatureManager checkInstallStatus" at ffdc_24.07.02_14.11.52.0.log
[7/2/24 14:11:52:208 UTC] 00003b00 com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0012I: The server installed the following features: [].
[7/2/24 14:11:52:208 UTC] 00003b00 com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0008I: Feature update completed in 0.086 seconds.


    ------------------------------
    Stephan Dietl
    ------------------------------

    Original Message


  • 7.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Tue July 02, 2024 03:03 PM

    Hello,

    This one is not related...

    Have you manually modified the server.xml after ILMT Installation/Upgrade ?
    In particular the     <featureManager> stanza ?


    To go further I'd need to look into your server.xml + ffdc_24.07.02_14.11.52.0.log file.



    ------------------------------
    Thank you,
    Oktawian

    Oktawian Powązka, L3 Support
    IBM License Metric Tool
    ------------------------------

    Original Message


  • 8.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 03:00 AM

    Hello again,

    I was not able to reply to your private email...

    Anyhow,
    That path:
    /opt/ibm/LMT/wlp/server/usr/servers/server1/server.xml

    is definitely not correct...
    It looks like some aftermath of the failed upgrade process.
    To go further I'd need to look into the upgrade logs (If those are not available just send us the whole '/opt/ibm/LMT/resume' folder content).

    At last,
    Due to the bigger amount of files which needs to be exchanged I'd strongly suggest to open support ticket to us....



    ------------------------------
    Thank you,
    Oktawian

    Oktawian Powązka, L3 Support
    IBM License Metric Tool
    ------------------------------

    Original Message


  • 9.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 04:04 AM

    Hello @Oktawian Powązka !

    > I was not able to reply to your private email...

    I´ve today accepted your contact request and when this message is posted it seems the problems I had yesterday posting here are gone :) !

    > Anyhow,
    > That path:
    > /opt/ibm/LMT/wlp/server/usr/servers/server1/server.xml
    > is definitely not correct...

    Ok, I see!

    > It looks like some aftermath of the failed upgrade process.

    Yes, but what puzzles me is that the fixlet in BigFix just says "fixed" and had no indication that something went wrong.

    > To go further I'd need to look into the upgrade logs (If those are not available just send us the whole '/opt/ibm/LMT/resume' folder content).

    I have both, yes:

    -rw-r--r--.  1 root root 209941 Jul  2 16:12 LMT_9.2.36.1_20240702-161100_1798090_logs.tar.gz

    [0][PROD][root@ilmt03p:/opt/ibm/LMT] ls -alrt resume/
total 60
drwxr-----.  3 root root 4096 Jul 16  2021 LMT9.2.23.0
drwxr-----.  3 root root 4096 Jul 22  2021 LMT9.2.24.0
drwxr-----.  3 root root 4096 Oct  4  2021 LMT9.2.25.0
drwxr-----.  3 root root 4096 Dec 17  2021 LMT9.2.26.0
drwxr-----.  3 root root 4096 Jun 22  2022 LMT9.2.28.0
drwxr-----.  3 root root 4096 Oct  4  2022 LMT9.2.29.0
drwxr-----.  3 root root 4096 Jan 13  2023 LMT9.2.30.0
drwxr-----.  3 root root 4096 Mar 31  2023 LMT9.2.31.0
drwxr-----.  3 root root 4096 Jun 28  2023 LMT9.2.32.0
drwxr-----.  3 root root 4096 Sep 21  2023 LMT9.2.33.0
drwxr-----.  3 root root 4096 Dec 13  2023 LMT9.2.34.0
drwxr-----.  3 root root 4096 Mar 19 14:47 LMT9.2.35.0
drwxr-----.  3 root root 4096 Jul  2 16:11 LMT9.2.36.1
drwxr-----. 15 root root 4096 Jul  2 16:11 .
drwxr-----. 16 root root 4096 Jul  2 16:12 ..
[0][PROD][root@ilmt03p:/opt/ibm/LMT] ls -alrt resume/LMT9.2.36.1/
total 12
drwxr-----. 15 root root 4096 Jul  2 16:11 ..
drwxr-----.  3 root root 4096 Jul  2 16:11 .
drwxr-----.  2 root root 4096 Jul  2 16:12 20240702_1611
[0][PROD][root@ilmt03p:/opt/ibm/LMT] ls -alrt resume/LMT9.2.36.1/20240702_1611/
total 992
drwxr-----. 3 root root   4096 Jul  2 16:11 ..
-rw-------. 1 root root    249 Jul  2 16:11 action_list.sav
-rw-------. 1 root root   2488 Jul  2 16:11 action_0.sav.bac
-rw-------. 1 root root   2791 Jul  2 16:11 action_0.sav
-rw-------. 1 root root   1860 Jul  2 16:11 action_1.sav.bac
-rw-------. 1 root root   2048 Jul  2 16:11 action_1.sav
-rw-------. 1 root root   1868 Jul  2 16:11 action_2.sav.bac
-rw-------. 1 root root   2057 Jul  2 16:11 action_2.sav
-rw-------. 1 root root   1797 Jul  2 16:11 action_3.sav.bac
-rw-------. 1 root root   1872 Jul  2 16:11 action_3.sav
-rw-------. 1 root root   1680 Jul  2 16:11 action_4.sav.bac
-rw-------. 1 root root   1755 Jul  2 16:11 action_4.sav
-rw-------. 1 root root   1981 Jul  2 16:11 action_5.sav.bac
-rw-------. 1 root root   2056 Jul  2 16:11 action_5.sav
-rw-------. 1 root root   1660 Jul  2 16:11 action_7.sav.bac
-rw-------. 1 root root   1735 Jul  2 16:11 action_7.sav
-rw-------. 1 root root   1980 Jul  2 16:11 action_9.sav.bac
-rw-------. 1 root root   3922 Jul  2 16:11 action_9.sav
-rw-------. 1 root root   6263 Jul  2 16:11 action_10.sav.bac
-rw-------. 1 root root  10031 Jul  2 16:11 action_10.sav
-rw-------. 1 root root   1644 Jul  2 16:11 action_11.sav.bac
-rw-------. 1 root root   1719 Jul  2 16:11 action_11.sav
-rw-------. 1 root root   1733 Jul  2 16:11 action_6.sav.bac
-rw-------. 1 root root   1808 Jul  2 16:11 action_6.sav
-rw-------. 1 root root   1750 Jul  2 16:11 action_12.sav.bac
-rw-------. 1 root root   1825 Jul  2 16:11 action_12.sav
-rw-------. 1 root root   1856 Jul  2 16:11 action_13.sav.bac
-rw-------. 1 root root   1986 Jul  2 16:11 action_13.sav
-rw-------. 1 root root   1926 Jul  2 16:12 action_14.sav.bac
-rw-------. 1 root root   2366 Jul  2 16:12 action_14.sav
-rw-------. 1 root root   1676 Jul  2 16:12 action_15.sav.bac
-rw-------. 1 root root   1836 Jul  2 16:12 action_15.sav
-rw-------. 1 root root   1939 Jul  2 16:12 action_16.sav.bac
-rw-------. 1 root root   2069 Jul  2 16:12 action_16.sav
-rw-------. 1 root root   1988 Jul  2 16:12 action_18.sav.bac
-rw-------. 1 root root   2294 Jul  2 16:12 action_18.sav
-rw-------. 1 root root   2492 Jul  2 16:12 action_19.sav.bac
drwxr-----. 2 root root   4096 Jul  2 16:12 .
-rw-------. 1 root root   2933 Jul  2 16:12 action_19.sav
-rw-------. 1 root root 842299 Jul  2 16:12 Server.log


    > At last,
    > Due to the bigger amount of files which needs to be exchanged I'd strongly suggest to open support ticket to us....

    Agreed, I was going to do that anyways, thanks for your help so far!

    With kind regards,

    Steve



    ------------------------------
    Stephan Dietl
    ------------------------------

    Original Message


  • 10.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 04:56 AM

    Hello @Oktawian Powązka !

    > Due to the bigger amount of files which needs to be exchanged I'd strongly suggest to open support ticket to us....

    I´ve now opened TS016642688 :) !


    Thanks,

    With kind regards,

    Steve



    ------------------------------
    Stephan Dietl
    ------------------------------

    Original Message


  • 11.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 06:00 AM

    Hello Stephan,


    Just looked into it...

    Pretty interesting behavior of the mv command:
    dbg - com.ibm.license.mgmt.install.ia.util.ExecuteCmd.execute - Executing command: [mv] [-v] [/opt/ibm/LMT/server] [/opt/ibm/LMT/wlp]
    dbg - com.ibm.license.mgmt.install.ia.util.ExecuteCmd.waitFor - timeot = 0 means wait indefinitely
    dbg - com.ibm.license.mgmt.install.ia.util.ExecuteCmd.logResult - Command completed --------------------------
    dbg - com.ibm.license.mgmt.install.ia.util.ExecuteCmd.logResult - return Code 0
    dbg - com.ibm.license.mgmt.install.ia.util.ExecuteCmd.logResult - Stdout renamed '/opt/ibm/LMT/server' -> '/opt/ibm/LMT/wlp/server'

    It looks like that somehow mv command has got a mind on its own and placed the server subfolder under wlp rather than renaming into that name.
    I can't give you any logical explanation to this conundrum...

    I can see that you've got a dedicated mount point '/opt/ibm/LMT' for ILMT server location, but such scenario has already been tested by us and other Customers.

    Anyhow, there is a pretty quick workaround to this whole mess:
    Rename (just to have it as a backup) the existing ILMT folder '/opt/ibm/LMT',
    Rename or delete the '/var/.com.zerog.registry.xml' file.
    run the fresh installer and connect into the existing temadb database (at the setup database page please point out the existing temadb location)



    ------------------------------
    Thank you,
    Oktawian

    Oktawian Powązka, L3 Support
    IBM License Metric Tool
    ------------------------------

    Original Message


  • 12.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 06:32 AM

    Hello @Oktawian Powązka!

    Thanks for your quick response!

    > It looks like that somehow mv command has got a mind on its own and placed the server subfolder under wlp rather than renaming into that name.
    > I can't give you any logical explanation to this conundrum...

    SkyNet awakening? ;)

    > I can see that you've got a dedicated mount point '/opt/ibm/LMT' for ILMT server location, but such scenario has already been tested by us and other Customers.

    > Anyhow, there is a pretty quick workaround to this whole mess:
    > Rename (just to have it as a backup) the existing ILMT folder '/opt/ibm/LMT',

    Wouldn`t that be affected by the fact that this is the mount point of the dedicated filesystem?

    > Rename or delete the '/var/.com.zerog.registry.xml' file.
    > run the fresh installer

    Via BigFix?

    > and connect into the existing temadb database (at the setup database page please point out the existing temadb location)

    The one mentioned in /opt/ibm/LMT/wlp/usr/servers/server1/config/database.yml ? How to find out the location please?

    Thanks!

    With kind regards,

    Steve



    ------------------------------
    Stephan Dietl
    ------------------------------

    Original Message


  • 13.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 07:21 AM

    SkyNet awakening? 
    Could be....


    Anyhow, I think that this weird mv behavior is somehow associated with particular RHEL release (8.9), as you've stated in the ticket:
    Weve upgraded to 9.2.35 in March without a problem and later we had to upgrade our ILMT RHEL Server from     7.9 to 8.9
    This is the first upgrade of ILMT which happened after the release change of the RHEL server it is running on!


    Wouldn`t that be affected by the fact that this is the mount point of the dedicated filesystem?
    You are absolutely correct, I've got ahead of my self...forgetting that this path is a dedicated mount point.
    Just move the current ILMT folder (/opt/ibm/LMT) content to somewhere else...

    It doesn't really matter if the fresh installer is via BigFix or from command line...

    By 'connect into the existing temadb database' I've meant to fill in the existing ILMT database name on the setup database page.
    From the upgrade log I can see that yours ILMT database is called TEMADB 



    ------------------------------
    Thank you,
    Oktawian

    Oktawian Powązka, L3 Support
    IBM License Metric Tool
    ------------------------------

    Original Message


  • 14.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Thu July 04, 2024 07:40 AM

    Hello @Oktawian Powązka !

    After "finding" the locally hidden "Download ILMT" fixlet I was able to reinstall it and after connecting it again with my existing TEMADB everything looks fine now!

    Thanks also here for your support,

    With kind regards,

    Steve



    ------------------------------
    Stephan Dietl
    ------------------------------

    Original Message


  • 15.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Wed July 03, 2024 10:09 AM

    Hi Andrei,

    thanks for the confirmation. So the implementation of version 9.2.36.1. was done today.
    Until now, everything is looking fine.

    Best regards,
    Harald



    ------------------------------
    Tooling42
    ------------------------------

    Original Message


  • 16.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Thu July 18, 2024 09:13 AM

    Regarding this 'failing' mv command  ([mv] [-v] [/opt/ibm/LMT/server] [/opt/ibm/LMT/wlp])


    I've found a root cause,
    It happens due to race condition with the previous upgrade step (Backup the server directory):
    Executing command: [mv] [-v] [/opt/ibm/LMT/wlp] [/opt/ibm/LMT/wlp_9.2.xxx.0]

    Somehow, it only occurs  in cases with a separated mount point for ILMT server location (like '/opt/ibm/LMT' in this example)
    If the mv command from 'Backup the server directory' step doesn't finish on time and the wlp folder is not fully moved the subsequent mv operation creates a server subfolder under wlp instead renaming into it.

    To be fixed in next ILMT release.
    A dedicated technote is ongoing....

    There is only one workaround:
    run the fresh installer and connect into the existing temadb database.



    ------------------------------
    Thank you,
    Oktawian

    Oktawian Powązka, L3 Support
    IBM License Metric Tool
    ------------------------------

    Original Message


  • 17.  RE: After update ILMT to 9.2.36.0 does not start

    Posted Tue July 30, 2024 07:54 AM

    Hello @Oktawian Powązka !

    Sorry for my late answer, was on vacation :) !

    > To be fixed in next ILMT release.
    > A dedicated technote is ongoing....

    Perfect, thanks!

    > There is only one workaround:
    > run the fresh installer and connect into the existing temadb database.

    Did that, thanks for your help, worked fine and everything is back to normal again :) !

    With kind regards,

    Stephan Dietl



    ------------------------------
    Stephan Dietl
    ------------------------------

    Original Message