IBM i Global

 View Only
  • 1.  Backup Issue

    Posted Wed August 14, 2024 11:34 AM

    Good day,

     

    I am having an issue where my backup on my Dev system has always been running under TAPMLB03, in Media Library Services. For some reason after the Friday night back and when the weekly backup is to start Saturday night, something causes a switch to TAPMLB02. I go into BRMS and make my changes to TAPMLB02 and then by the weekend it switches back to looking for TAPMLB03. The tape drive is TAP04, which I'm sure no relevance.  When I look at the adapter card for the failing device and it says not operational, and it flip flops back and forth. Don't really see anything in the log.  Ideas?

     

    Michael Ruth

    iSeries Support – SiteOps IT North American MQ & RD

     

     

    CONFIDENTIALITY NOTICE: This email message (including all attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, copying or distribution is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

    PRIVACY NOTICE: Your privacy is important to us. To find out more about the information that Elanco may collect, how we use it, how we protect it, and your rights and choices with respect to your Personal Data, go to privacy.elanco.com

     



  • 2.  RE: Backup Issue

    Posted Wed August 14, 2024 08:20 PM

    Dear Michael

    Which "log" did you look into?  Apart from IBM i job log, there are a few more such logs.  BRMS has its own event log. You should also look at IBM i history log or QSYSOPR message queue.   Also event log of the tape library itself.



    ------------------------------
    Satid S
    ------------------------------



  • 3.  RE: Backup Issue

    Posted Thu August 15, 2024 04:24 AM

    Another point that comes to mind is whether or not this issue has to do with Logical Library Mode ( https://www.ibm.com/docs/en/ts4300-tape-library?topic=functions-random-sequential-logical-library-modes ) ?  Please check if your tape library is currently set to random or sequential mode?  If it is now in random mode, you may try changing it to sequential mode to see if this addresses the issue or not?



    ------------------------------
    Satid S
    ------------------------------



  • 4.  RE: Backup Issue

    Posted Thu August 15, 2024 02:04 PM

    Michael, 

    Questions that come to mind:

    1. Is the Library shared amongst other systems?  (BRMS has a networking feature that flips libraries in a pool between systems)
    2. What "type" of library is it?
      1. like LTO / VTL / Cloud backup? 
    3. Is this a standalone BRMS install? or a hub and node option? (like Server / Client configuration?)

    Some Links you can peruse (if you have the capacity and some time to troubleshoot through it) 

    https://www.ibm.com/docs/en/i/7.3?topic=brms-last-active-device <== This shows you that the "Last Active/ Good known Device" will be selected, so probably fixing the network/adapter card failure will resolve your issue 

    ---> further to that, look at the control group settings of the devices used, there might be two libraries configured as failover perhaps? (normally you can have many tape drives for multiple parallel saves)

    https://helpsystemswiki.atlassian.net/wiki/spaces/IWT/pages/888963080/BRMS+SQL+Services <--- good insight for the "log" of BRMS with SQL statements

    https://helpsystemswiki.atlassian.net/wiki/spaces/IWT/pages/165642242/Tape+Library+Resource+Status+Tool <---- this will give you helpful diagnostics  (might want to consider asking an admin to help you run these commands)

    https://helpsystemswiki.atlassian.net/wiki/spaces/IWT/pages/165642273/Troubleshooting+Network+Connectivity+Problems+in+Enterprise <--- also a good way to see, if these libraries are connected on a network basis, to see why they fail. 

    https://helpsystemswiki.atlassian.net/wiki/spaces/IWT/pages/221904929/BRMS+Partners <--- if you need professional support then you can contact one of these companies,. 

    As Satid S said, best to go through the History Log of the system / QSYSOPR log (close to the time of the failure) & BRMS log itself for helpful indications (also the WRKPRB - Problem Log) might give some clues as well. 

    HTH

    Marius



    ------------------------------
    Marius le Roux
    Owner
    MLR Consulting
    ------------------------------



  • 5.  RE: Backup Issue

    Posted Thu August 15, 2024 02:47 PM

    I did a little further digging (after consulting with a fellow admin): 

    something also to consider : Reference : https://www.ibm.com/support/pages/ibm-i-removable-media-tape-and-tape-library-devices#TAPMLB

    Control Path and Data Path Failover

    When saving to a tape library, data flows across 2 channels: the data path and the control path. The actual data travels along the data path. The tape library control commands "mount tape", "eject tape", "inventory library" and so on, flow along the control path. In all libraries except the 3494, the control path and data path use the same cable.

    Control Path Failover:

    IBM i supports control path failover on systems that are running IBM i 7.1 or higher and using IOPless fibre cards for their tape or virtual tape. If a save is running and the IBM i detects a problem with the control path it is using, it automatically starts sending the library commands down another control path that is attached to the same fibre port and is associated with another drive in the same tape library.  The save continues without interruption.  If multiple control paths fail, the save continues to fail over until there are no more operational control paths.  Note: This function does not support failover to a control path on a different Fibre Channel port or a different library.

    Data Path Failover

    Tape multi-path support is available for IBM i 7.2 and later releases.   See Tape Multipath

    Redundant tape connections (multi-path) are not supported on IBM i 7.1 and earlier



    ------------------------------
    Marius le Roux
    Owner
    MLR Consulting
    ------------------------------



  • 6.  RE: Backup Issue

    Posted Thu August 15, 2024 03:52 PM

    Marius

     

    1. Is the Library shared amongst other systems?  (BRMS has a networking feature that flips libraries in a pool between systems) No, this is not shared.

     

    1. What "type" of library is it?  
      1. like LTO / VTL / Cloud backup?  VTL

     

        1. Is this a standalone BRMS install? or a hub and node option? (like Server / Client configuration?)  Stand alone

     

    I have 3 other partitions and all are configured the same with no issues. Puzzling to say the least.

     

     

    Michael Ruth

    iSeries Support – SiteOps IT North American MQ & RD

     

     

    CONFIDENTIALITY NOTICE: This email message (including all attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, copying or distribution is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

    PRIVACY NOTICE: Your privacy is important to us. To find out more about the information that Elanco may collect, how we use it, how we protect it, and your rights and choices with respect to your Personal Data, go to privacy.elanco.com

     






  • 7.  RE: Backup Issue

    Posted Fri August 16, 2024 06:45 AM

    ok I see, interesting. 

    "highly unlikely" but something to consider, Are the volumes perhaps in different medial libraries? (e.g. VOL1 is in TAPMLB03 and VOL2 is in TAPMLB02)

    that could also explain why it's switching, perhaps the move of the volumes or the operator misplaced some tapes? 

    your BRMS log will show you a golden trove of treasures in this conundrum. perhaps if you are allowed to share (you may even email me privately if you wish, I can look at it to spot something odd or not) 



    ------------------------------
    Marius le Roux
    Owner
    MLR Consulting
    ------------------------------



  • 8.  RE: Backup Issue

    Posted Mon August 19, 2024 02:03 PM

    Ok

    1 - This has nothing to do with whether or not your library is set to random or sequential.

    2 - This has nothing to do with whether or not volumes are dedicated to one library or the other.

    What this has to do with is your switch process.

    Something during the switch process is causing the controller for the VTL to get whacked and a new one is generated.

    First, let's make sure the terminology is right.  I am guessing you are talking about VTL as Virtual Tape Library and are NOT talking about simple Virtual Tape with the volumes on IFS files, right?  I have used both.

    What is happening is you have these items in WRKHDWRSC *STG.  Some may be disk and some are for your VTL.  The "whacked" ones may show up in STRSST, 1. Start a service tool, 7. Hardware service manager, 4.  Failed and non-reporting hardware resources.

    You can match the entries in WRKHDWRSC *STG with the resource name in WRKDEVD *MLB.

    If, during your switch, you dynamically allocate one FC card from one lpar to another this will likely cause your issue.  You might be better served by having VIOS serve up your FC card and have VIOS virtualize it to both lpars.  I do switches all the time and do not have this issue.  If you are not using FC but instead are using SCSI or whatever the interface is and you are dynamically allocating it from one lpar to another this will cause your issue also.  Better to use FC and VIOS.  Some may object to the additional cost of a FC switch, VIOS, etc.  But I fail to find how having an operator work on the weekend to go into SST and clean up the mess and reconfigure the media library would be any less expensive.  Especially as finding an operator you'd trust to go into SST would likely not be on the low end of a living wage position.

    One alternative might be to use the "Change Resource Entry Information API" at https://www.ibm.com/docs/en/i/7.5?topic=ssw_ibm_i_75/apis/hw1.html to rename TAPMLBxx to the right name.  Or you could use https://www.ibm.com/docs/en/i/7.5?topic=services-media-library-info-view to get a list of your current media libraries, find whatever one is current today, vary it off, rename it from TAPMLBxx to TAPMLB01 (or whatever you are calling it in BRMS) and vary it back on.  Heck you might even want to start calling it something unique so as to never worry about autoconfig using the same name.  For example even if the entry in WRKHDWRSC *STG is TAPMLB07, there's nothing stopping you from using script to vary off TAPMLB07, change MYBRMSMLB to use resource name TAPMLB07, then vary MYBRMSMLB on.  IDK if all of this, along with "Media and Storage Extensions" trapping stuff would start to confuse the daylights out of BRMS.

    I do switches, use BRMS, etc and with the FC virtualized by VIOS I have no such issues.

    But I do think this may be what your problem is.



    ------------------------------
    Robert Berendt IBMChampion
    ------------------------------