Maximo

Maximo

Come for answers, stay for best practices. All we're missing is you.

 View Only
Back to discussions
Expand all | Collapse all

Issue while Maximo to MAS Upgrade

  • 1.  Issue while Maximo to MAS Upgrade

    Posted Wed September 20, 2023 12:39 PM

    Hi Team,

    I am getting the below error while the all pod is trying to connect to database. Also the UpdateDB was successful .

    BMXAA6453I - The server is connecting to database version: V8600-60
    [err] javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
    [err] at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:1028)
    [err] at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1109)
    [err] at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:906)
    [err] at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
    [err] at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
    [err] at psdi.util.MXCipher.decData(MXCipher.java:293)
    [err] at psdi.server.MXServer.packagingCheck(MXServer.java:1952)
    [err] at psdi.server.MXServer.boot(MXServer.java:1631)
    [err] at psdi.server.MXServer.start(MXServer.java:3858)
    [err] at psdi.servlet.MAXIMOStartupServlet.init(MAXIMOStartupServlet.java:116)
    [err] at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:301)
    [err] at [internal classes]
    [err] at com.ibm.ws.webcontainer.osgi.WebContainer.access$100(WebContainer.java:112)
    [err] at com.ibm.ws.webcontainer.osgi.WebContainer$3.run(WebContainer.java:994)
    [err] at [internal classes]
    [err] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    [err] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    [err] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    [err] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    [err] at java.base/java.lang.Thread.run(Thread.java:839)
    20 Sep 2023 16:04:03:683 [FATAL] [maximo] BMXAA6473E - The MXSERVER server could not be started. Check the log file for other errors to determine the cause.
    psdi.util.MXSystemException: BMXAA4077E - The format for the start date is not valid. The system install date that is stored on the database is corrupt. Contact your system administrator to ensure that the database was installed properly.
    at psdi.server.MXServer.packagingCheck(MXServer.java:1956) ~[businessobjects.jar:?]
    at psdi.server.MXServer.boot(MXServer.java:1631) ~[businessobjects.jar:?]
    at psdi.server.MXServer.start(MXServer.java:3858) [businessobjects.jar:?]
    at psdi.servlet.MAXIMOStartupServlet.init(MAXIMOStartupServlet.java:116) [mboweb.war:?]
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:301) [com.ibm.ws.webcontainer_1.1.76.jar:?]
    at [internal classes]

    I have already performed the ./resetcrypto.sh command but still the same problem.

    Appreciate if someone can provide some pointers here.

    Thanks,

    Dhiraj Panjwani



    ------------------------------
    Dhiraj Panjwani
    ------------------------------


  • 2.  RE: Issue while Maximo to MAS Upgrade

    Posted Thu September 21, 2023 03:03 AM
    Edited by Jan-Willem Steur Thu September 21, 2023 03:04 AM

    Hi Dhiraj,

    According to the stack trace it seems that the date format on the database you want to migrate is invalid according to the date format that is required on MAS8. There are multiple reasons that can cause this issue. Did you have installed more language packs / locales on the MX database that are not yet available on your MAS8 instance?
    The stack trace is also mentioning this error:

    BMXAA4077E - The format for the start date is not valid. The system install date that is stored on the database is corrupt. Contact your system administrator to ensure that the database was installed properly.


    Can you try to sync / repair or update the system install date on the database you want to migrate, just before you start the migration procedure again? That might solve the issue.



    ------------------------------
    Jan-Willem Steur
    Jan-Willem
    ------------------------------

    Original Message


  • 3.  RE: Issue while Maximo to MAS Upgrade

    Posted Thu September 21, 2023 06:21 AM

    Hi Jan-Willem-Steur,

    Thanks for your inputs and suggestions.

    We have not installed additional language packs/locales. We tried to convert Oracle database to db2 and the updatedb was successful.

    So if there was database issue, we could have received error during updatedb. But as you pointed out the issue with System Install date, I would request your help to understand which table I would need to update to fix this issue. Also, the logs also mention about error related to bad padding: err] javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

    Appreciate your help .

    Thanks,

    Dhiraj Panjwani



    ------------------------------
    Dhiraj Panjwani
    ------------------------------

    Original Message


  • 4.  RE: Issue while Maximo to MAS Upgrade

    Posted Thu September 21, 2023 07:18 AM

    Hi Dhiraj,

    An updatedb is different then a data (content) migration. I am investigating the system install date issue now on one of our MAS8 instances and see if I can reproduce the issue. Will come back to you when I found some possible causes / solutions.



    ------------------------------
    Jan-Willem Steur
    Jan-Willem
    ------------------------------

    Original Message


  • 5.  RE: Issue while Maximo to MAS Upgrade

    Posted Thu September 21, 2023 07:26 AM
    Edited by Jan-Willem Steur Thu September 21, 2023 07:35 AM

    Hi Dhiraj,

    Did you tried to retrieve the crypto keys from the MX76 database (in some cases they exist, but sometime not). You can remove the old crypto keys in your MAS Manage project on OpenShift (secret xxxxx-manage-encryptionsecret-operator) and leave them empty or copy in the crypto keys you retrieved from your MX76 database. At least we had a similar case (not exactly the same error stack trace) where this approach helped us to successful connect to the database after updatedb.

    I found this procedure step back in our internal WIKI space which we use to store all our procedures and instructions:

    In OpenShift to avoid Crypto Keys errors, edit the secrets: maximopropertysecret, encryptionsecret and encryption-secret operator, with empty values for MXE_SECURITY_CRYPTO_KEY, MXE_SECURITY_CRYPTOX_KEY, MXE_SECURITY_OLD_CRYPTO_KEY and MXE_SECURITY_OLD_CRYPTOX_KEY properties



    ------------------------------
    Jan-Willem Steur
    Jan-Willem
    ------------------------------

    Original Message


  • 6.  RE: Issue while Maximo to MAS Upgrade

    Posted Wed September 27, 2023 04:04 AM

    Thanks for your help Jan-Willem.

    We tried the above approach of making the crypto keys null but still we have bad padding exception.



    ------------------------------
    Dhiraj Panjwani
    ------------------------------

    Original Message


  • 7.  RE: Issue while Maximo to MAS Upgrade

    Posted Wed September 27, 2023 11:02 AM

    Thank you for your inputs, Jan-Willem. 

    We tried the above steps and still we have the same issue.
    Made all these secrets as null but still getting bad padding exception.



    ------------------------------
    Dhiraj Panjwani
    ------------------------------

    Original Message


  • 8.  RE: Issue while Maximo to MAS Upgrade

    Posted Thu September 21, 2023 08:10 AM

    Dhiraj

    I ran into an issue on an upgrade that looks the same. What appears to be happening is the encrypt runs the first time successfully and is intended to set the old crypto keys to be equal to the new crypto keys. For some reason this fails. Then on sequential runs the code is trying to use the old keys but the data is actually encrypted with the new keys. I did report this to IBM and it was already identified as an internal defect that should be fixed in 8.10.4. 

    Not trying to force you into a new version. The work around that works well for me is modifying the crypto secret. I first confirmed that my crypto values were changed from the original values I provided in my database then I followed these steps. 

    1. Go to Secrets for the project in Open Shift
    2. Select the secret that has both the old and new crypto keys
    3. Edit the secret and copy the current keys as the old keys
    4. Save and reload
    5. Wait for the pods to re-process 

    Hope this helps



    ------------------------------
    Phillip Runion
    ------------------------------

    Original Message


  • 9.  RE: Issue while Maximo to MAS Upgrade

    Posted Wed September 27, 2023 04:06 AM

    Thank you, Phillip, for your inputs. 

    In our case both keys old and current are updated with same values, still we see bad padding exception.



    ------------------------------
    Dhiraj Panjwani
    ------------------------------

    Original Message


  • 10.  RE: Issue while Maximo to MAS Upgrade

    Posted Wed September 27, 2023 04:48 PM

    Dhiraj,

    Dunno if you have been able to fix this issue yet or not.  If not please consider this:  The bad padding error is likely due to the DB conversion routine you went through.  When I first saw this just now I thought " I wonder if they did a DB conversion?"  As I read the thread through I see that is in fact the case.  Likely what you need to do is reconvert the DB using a vargraphic conversion and "near" Oracle compatibility. There is a bit setting of 8 on the full compatibility mode.  Set it to 7 and see what happens.  In doing this before with other customers, the full Oracle mode compatibility causes some issues.  The dates are the worst.  The manner in which DB2 physical stores dates is very different from the way that Oracle or even SQL Server does.  As such the padding issue comes form the algorithm trying to convert the date from one storage pattern to the other.

    Hope this helps



    ------------------------------
    Bradley K. Downing , MBA
    Senior Brand Technical Specialist
    IBM
    Bakersfield CA
    ------------------------------

    Original Message


  • 11.  RE: Issue while Maximo to MAS Upgrade

    Posted Tue October 03, 2023 04:59 AM

    Hi Bradley,

    Thanks for your inputs and suggestions. Yes you are right here we are performing a database conversion from Oracle to DB2 and then trying to connect to this upgraded database. We tried changing the Oracle Compatibility to 7 ( in the Upgraded database) but still no luck. The other option you suggested to re-run the whole database conversion upgrade will take some time but will let you know if that helps.

    But is there any other place we can check on the reason for this failure? Any logs other than all-bundle pod which says there is problem with database conversion?

    Thanks,

    Dhiraj Panjwani



    ------------------------------
    Dhiraj Panjwani
    ------------------------------

    Original Message


  • 12.  RE: Issue while Maximo to MAS Upgrade

    Posted Tue October 03, 2023 10:22 AM

    Hi Dhiraj,

    We changed the bit during the follow-on conversion routine, in other words we set it to that value then executed conversion. Also, one test we did was DB conversion of 7.6 in 7.6.  When you point the MAS Manage instance to the Converted DB, it will try to initiate upgrade as well.  One step at a time.  First convert. Test and ensure it works in 7.6 then upgrade to 8.x.

    Since we did the conversion first and brought it up on 7.6 the logs we had were the standard.out file from WAS ND.  After success in ringing up the converted DB then we pointed the MAS instance to the converted DB, and brought it up there.  This approach seems to be a bit cleaner in that we separated the conversion from the upgrade.

    It appears you have having a lot of help come in from a number of different resources.  I believe you will get a solution soon.  This is a great community! 😁👍🏻



    ------------------------------
    Bradley K. Downing , MBA
    Senior Brand Technical Specialist
    IBM
    Bakersfield CA
    ------------------------------

    Original Message


  • 13.  RE: Issue while Maximo to MAS Upgrade

    Posted Fri September 29, 2023 04:23 AM

    Hi Dhiraj

    I had this one in the past when i tried to insert a crypto field inside a customization repository. Although it past through the update db fine, it wouldnt start the server because of it. In my case it referenced the field in the logs though, which made it a bit easier to find out what it was causing the issue.

    To fix it, I went to the encryption secret which stores the old and the new ones, and blanked out the values of the two OLD keys. I then deleted the manage entity ws pod and waited for that to recreate the maxinst and server pods, rather than trying to delete the maxinst pod directly which still had reference to the old secret.

    Thanks 



    ------------------------------
    Paul Irving
    Product Development Manager
    BPD Zenith Ltd
    ------------------------------

    Original Message


  • 14.  RE: Issue while Maximo to MAS Upgrade

    Posted Tue October 03, 2023 04:56 AM

    Hi Paul,

    Thanks and Appreciate your help and suggestion for this issue. I tried to perform the steps you mentioned . The maxinst and all bundle pod started . There was no issue with maxinst( earlier as well ran fine) and when the all bundle tried connecting to database , it gave the same padding exception error. I am not sure if the issue is with decrypting while it tries to connect to DB or it has different issue. 

    P.S.: The maxinst pod runs fine and we are able to see the database connecting in the  maxinst pod .

    Thanks,

    Dhiraj Panjwani



    ------------------------------
    Dhiraj Panjwani
    ------------------------------

    Original Message


  • 15.  RE: Issue while Maximo to MAS Upgrade

    Posted Tue January 23, 2024 01:58 PM

    Hi, did you solve this issue? I had the same problem when converting a maximo DB from MS SQL Server to DB2. I had to delete the dummy_table content and then do a insert select from an existing dummy_table natively in DB2 and it worked. My server started afterwards . Hope this heps.



    ------------------------------
    Alina Vlase
    ------------------------------

    Original Message