Informix

Informix

Connect with Db2, Informix, Netezza, open source, and other data experts to gain value from your data, share insights, and solve problems.

 View Only

  • 1.  Backup in object store

    Posted 02/18/20 02:49 PM
    Edited by System Admin 01/20/23 04:48 PM
      |   view attached
    Hello,

    I'm using Informix in a K8S cluster. As backup, I use an S3 object store (the IBM one). I use the storage class vault or cold vault.
    My dbspaces and backup files are encrypted.

    I followed the documentation for setup the backup with the PSM. The backup works well but I have problems with verify and sometimes restore.

    The problem with verify (onbar -v) is the following:
     2020-02-18 09:18:49.212748 5901  5899 /opt/ibm/informix/bin/onbar_d -v
     2020-02-18 09:18:49.466129 5901  5899 Using IBM's PSM version 14.10.FC3 as the Storage Manager. XBSA API version is 1.0.3.
     2020-02-18 09:18:49.948736 5901  5899 Successfully connected to Storage Manager.
     2020-02-18 09:18:51.088223 5901  5899 Begin backup verification of level 0 for rootdbs (Storage Manager copy ID: 0 632).
     2020-02-18 09:24:51.494699 5901  5899 XBSA Error: (BSAGetData) A system error occurred. Aborting XBSA session.
     2020-02-18 09:24:51.495622 5901  5899 (-43391) Skipped backup/restore of space 'rootdbs'.


    The problems with restore are the following but I'm not able to actually reproduce its into logs files. However, I kept the errors ;-):
    XBSA Error: (BSAGetData) Exceeded available resources.
    (-43391) Skipped backup/restore of space 'db1_sbs'.
    or
    Unable to write storage space restore data to the database server: .
    (-43391) Skipped backup/restore of space 'db2_sbs'.
    or
    the restore get stuck on Begin cold level 0 restore db1_sbs (Storage Manager copy ID: 0 39). (bar_act.log) / Clearing encrypted chunk 5 before restore... (online.log) and never give hand back.

    I join to a zip file with the logs from the backup (with no error), verify (with error) and restore (no error) operations.

    Could you please help me understand why I'm unable to verify the backups when I can restore them without error?
    I need to periodically verify the backup to demonstrate my customer that its data is safe.

    Thanks a lot.
    Regards,
    Renaud

    ------------------------------
    Renaud Demarneffe
    ------------------------------
    #Informix

    Attachment(s)

    zip
    onbar.zip   4.05 MB 1 version


  • 2.  RE: Backup in object store

    Posted 02/18/20 03:39 PM
      |   view attached
    Hello,

    I just got to be able to reproduce the error during a restore but with the DEBUG level to 0.
     2020-02-18 19:40:28.779457 3862  3860 Begin cold level 0 restore rootdbs (Storage Manager copy ID: 0 632).
     2020-02-18 19:45:28.029499 3862  3860 88 percent of rootdbs has been restored.
     2020-02-18 19:46:54.648648 3862  3860 XBSA Error: (BSAGetData) Exceeded available resources.
     2020-02-18 19:46:54.648674 3862  3860 (-43391) Skipped backup/restore of space 'rootdbs'.
     2020-02-18 19:46:55.422337 3862  3860 Unable to close the storage space restore: buc_fe.c : Archive API processing failed at line 684 for msgtype.
     2020-02-18 19:47:00.422490 3862  3860 (-43140) Due to the previous error, logical restore will not be attempted.

    I join the collected logs.
    After that, I do a new restore with DEBUG level to 9 and it worked successfully. I also join the logs.

    It's a very surprizing behaviours. Can the debug level have an influence on the result of a restore (do the traces slow down certain operations, making them run without error?) ?

    Regards,
    Renaud

    ------------------------------
    Renaud Demarneffe
    ------------------------------

    Attachment(s)

    zip
    3 - restore with error.zip   5.28 MB 1 version


  • 3.  RE: Backup in object store

    Posted 02/19/20 05:40 AM
    Hi Renaud,

    first: would you be able, in the future, to log a support case for such - well defined and well prepared, i.e. obvious - defect descriptions, just to allow for some bookkeeping and stuff around the work that's required?

    I nonetheless proceeded and logged a defect for the onbar verification problem as I think both bar_dbug.log and psm_dbug.log provide sufficient information to understand the problem and, hopefully, get it fixed: some return code 3 is produced somewhere and propagated all the way up to BSAGetData() (for which it gets reported as "A system error occurred.", corresponding to BSA_RC_ABORT_SYSTEM_ERROR (s. https://pubs.opengroup.org/onlinepubs/9657499/chap4.htm) and ultimately ends up being onbar's return code.

    What I'm not so sure about is whether your newly found restore problem is the same issue. Unfortunately all we really know about it is "XBSA Error: (BSAGetData) Exceeded available resources."  -  which, at first glance, looks different  (BSA_RC_NO_RESOURCES 0x13 = 19).
    -> have you tried with lower debug level, or differing BAR_DEBUG / PSM_DEBUG levels, so at least some debug output can be gathered?

    BR,
     Andreas

    ------------------------------
    Andreas Legner
    ------------------------------

    Original Message


  • 4.  RE: Backup in object store

    Posted 02/19/20 05:50 AM
    Well, the psm_act.log seems to clearly say it ("different problem"):

    bar_act.log:
    2020-02-18 19:45:28.029499 3862 3860 88 percent of rootdbs has been restored.
    2020-02-18 19:46:54.648648 3862 3860 XBSA Error: (BSAGetData) Exceeded available resources.

    psm_act.log:
    2020-02-18 19:40:27.780990 3862 3860 Informix Primary Storage Manager session 1380 opened.
    2020-02-18 19:46:54.646707 3862 3860 Error -88105: Cannot lock catalog table 'lock'.
    ISAM Error = 4.
    OS Error = 4 (Interrupted system call).

    ------------------------------
    Andreas Legner
    ------------------------------

    Original Message


  • 5.  RE: Backup in object store

    Posted 02/19/20 03:37 PM
    Hi Andreas,

    Thank you for your help.

    I did not log a support case because I was not sure if the problem was due to a bug or to an error in my configuration. I did not found a lot of information on internet about PSM and S3, mainly the IBM one. Moreover, I did not understand much in the result of my tests, mainly because of the erratic behavior of the restores compared from the debug level.

    I will redo a battery of restore tests with different debug levels for BAR and PSM.
    Once the results obtained, do you want me to post them here or open a support ticket?

    Regards,
    Renaud

    ------------------------------
    Renaud Demarneffe
    ------------------------------

    Original Message


  • 6.  RE: Backup in object store

    Posted 02/20/20 09:21 AM
    Case please, esp. if it looks like a new defect, and certainly if it nicely reproduces.

    Not wanting to deter from posting problems here (letting us know what you're trying certaily is nice testament to Informix being well alive), but for support to work such problem a case should be the correct vehicle.

     Andreas

    ------------------------------
    Andreas Legner
    ------------------------------

    Original Message


Related Content