Global Storage

Global Storage Forum

Connect, collaborate, and stay informed with insights from across Storage

 View Only
Expand all | Collapse all

Flashsystem 5045 Ports in DEGRADED mode

  • 1.  Flashsystem 5045 Ports in DEGRADED mode

    Posted Sun March 10, 2024 07:54 PM

    Hello:

    We have a Flashsystem 5045 storage directly attached to a IBM POWER9 that has 2 4ports 16Gbps fiber channel cards, and each card is attached to a VIOS, so we have 2 VIOS using 4 port fiber-channel ports.

    The 8 fiber-channel ports (4 for each controller) are connected this way (in order to have no SPOF):

    Port 1 controller 1 is connected to port 1 first FC POWER9 card

    Port 2 controller 1 is connected to port 2 first FC POWER9 card

    Port 3 controller 1 is connected to port 1 second FC POWER9 card

    Port 4 controller 1 is connected to port 2 second FC POWER9 card

    Port 1 controller 2 is connected to port 3 first FC POWER9 card

    Port 2 controller 2 is connected to port 4 first FC POWER9 card

    Port 3 controller 2 is connected to port 3 second FC POWER9 card

    Port 4 controller 2 is connected to port 4 second FC POWER9 card

    The issue raises when we start the VIOS and try to connect each VIOS to the FS5045. We creatd two hosts, VIOS1 and VIOS2 and the FC cards are autodetectd, so in each hosts we add 4 WWN corresponding to each port of the FC cards of the POWER9. Then we create volumes, attach them to each VIOS and then we boot and install VIOS OS and everything without issues. Ports are direct attached and are negotiating at 16 Gbps.

    BUT, if we list the HOST on the FS5045 GUI, the Hosts are online with the green "OK" but if we enter to properties, and select "Ports" tab, the 4 ports appear as DEGRADED and yellow sign. 

    The access is normal, we installed several LPARs and VIOS don't show any errors, nor errpt issues, but the FS5045 is reporting DEGRADED status in the ports. FS5045 also don't report any issues. The FS5045 has 8.6.0.0 firmware. The machine is working in NPIV mode.

    So, my question is: How can we solve the DEGRADED status of each ports and get the ONLINE and "green" status?

    Thank you in advance.

    Best Regards,

    Urtzi Larrieta



    ------------------------------
    Urtzi Larrieta Alvarez
    ------------------------------


  • 2.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Mon March 11, 2024 03:11 AM

    Hi Urtzi,

    have you tired this: click Monitor > Events in the management GUI. This action displays a list of unfixed errors that are currently in the event log. For these unfixed errors, select the error name to begin a guided maintenance procedure to resolve them.

    Bests

    Justyna



    ------------------------------
    Justyna Kamionka
    IBM System Storage Product Manager
    justyna.kamionka@arrow.com
    Arrow ECS
    ------------------------------



  • 3.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Mon March 11, 2024 08:38 AM

    Hello, Justyna:

    That is the first thing I've done, but no, there are NO events recorded. Everything is just working fine, but ports appear as Degraded and we don't know why. Do you have another advise in order to check?

    Thank you ina dvance.

    Regards,

    Urtzi Larrieta



    ------------------------------
    Urtzi Larrieta Alvarez
    ------------------------------



  • 4.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Mon March 11, 2024 12:56 PM

    Hi Urtzi,

    would need more info from the system.

    At least:

    svcinfo lsportfc -delim , -gui

    svcinfo lshost

    and

    svcinfo lshost -delim : X

    where X is for each host id that You have in the system

     

    OR please use IBM Support:

    SNAP option 1 (Standard logs)
    Upload to:
    https://www.secure.ecurep.ibm.com/app/upload_hw
    For MTM use machine type for Serial Number use Your machine serial.

    Bests

    Justyna



    ------------------------------
    Justyna Kamionka
    IBM System Storage Product Manager
    justyna.kamionka@arrow.com
    Arrow ECS
    ------------------------------



  • 5.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Mon March 11, 2024 08:43 PM

    Are you having any trouble with the NPIV/Virtual FC client lpars?  

    The reason I ask is that I am pretty sure NPIV over a direct connection to PowerVM (VIOS) is not supported.  See https://www.ibm.com/support/pages/direct-attached-storage-vios-npiv-clients

    The issue is the FlashSystems incorporate a limited function NPIV controller, but it is not a full function NPIV controller as you'd get in a SAN switch.  This tricks the VIOS into thinking that all is well, including response to the lsnports command, but it does not support the full function required for NPIV.

    Checking your configuration on the SSIC link in the support page referenced above is a good step as it may also depend on the specific adapters you are using.  In any case you really should open a support ticket -- this is not the sort of question you're likely to get a definitive answer to in this forum.

    I have spent a fair amount of time researching this issue due to restrictions on direct connections in the non-LTS 8.6.1+ firmware levels.  I know that in small configurations, direct connect is often desirable to eliminate the cost of a switch, but building a small switch (or two) into your configuration smooths over a lot of potential problems.  It also gives you a monitoring point for traffic and error statistics and makes future system and storage migrations much easier.



    ------------------------------
    Vincent Greene
    IT Consultant
    Technology Expert labs
    IBM
    Vincent.Greene@ibm.com


    The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
    ------------------------------



  • 6.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Tue March 12, 2024 08:50 AM

    Thank you for your reply, Vincent:

    We actually use VIOS and Flashsystems volumes as a virtual disks (that is, we have vhosts with hdiskX disks mapped from the FlashSystem) and no NPIV volumes really mapped, so we have no issues. This question arised because the customer had a 5030 with NPIV deactivated and ports worked properly in online stattus, and we migrated the 5030 to the 5045 with NPIV enabled by default and with this result, ports degraaded. But the Flashsystems are working well, I think it's only a "cosmetic" degradation...

    But you are right, better open a Case in IBM to help me solve this situation.

    Thank you for all.

    Best Regards,

    Urtzi Larrieta



    ------------------------------
    Urtzi Larrieta Alvarez
    ------------------------------



  • 7.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Mon March 25, 2024 04:37 AM
    Hello Urtzi,
     
    depending on the Virtualize version, NPIV and Direct Attached are no longer supported or tested. The same applies to the VIO version, FC yes NPIC, FC no. As Vincent wrote. And no, it's not cosmetic.
     
    https://www-50.ibm.com/systems/support/storage/ssic/interoperability
    This environment is not in a supported state.
    Regards Patrik


    ------------------------------
    Patrik Groß
    ------------------------------



  • 8.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Wed March 27, 2024 12:55 AM

    Hi Urtzi

    Earlier, I configured the same with FS5045 storage and a P8 server with a single VIOS. In a similar configuration, I faced the same issue with the online host and degraded ports for all hosts in P8 servers (VIOS and multiple LPARS).

    As per my knowledge, this is a limitation (or say, normal behavior) for the directly connected storage as the single port of a host won't be able to communicate to both controllers of storage in any case. Also, as a host can communicate with both controllers of storage using multiple different ports there is no SPOF.



    ------------------------------
    Anup Regmi
    ------------------------------



  • 9.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Wed March 27, 2024 09:38 AM

    Hello:

    Thank for your reply. I opened a case and they told me that IBM is working in a new APAR HU02431 in order to solve the degraded status, because NPIV "is" supported in direct attach configurations also with new 8.6.0.X firmwares, and the issue is only a "cosmetic" problem in version 8.6.0.3 and newer versions that, as they told me, are working to solve.

    So thank you for all your replies, and this thread can be closed for now.

    Best regards,

    Urtzi Larrieta



    ------------------------------
    Urtzi Larrieta Alvarez
    ------------------------------



  • 10.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Mon September 30, 2024 09:30 AM

    Hi Urtzi, 

    Just a follow up on this thread.  Did IBM successfully close your support ticket and resolve the issue in new spectrum virtualize release ? 

    We have a client asking for the same question in a very similar configuration, a Power 10 server directly attaches to a FS5045, without SAN switches.  NPIV is expected to be enabled.   

    Would like to see if your case is a good example.  Thanks a lot!  

    Best regards,

    Larry Hao



    ------------------------------
    Larry Hao
    ------------------------------



  • 11.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted Tue October 01, 2024 05:08 AM

    Hello:

    IBM cloesd the ticket, but they only send an RPQ to newer versions, but as far as I know, the "issue" (that not an issue because ports are working normally) degraded mode is still on.

    No good news at the moment.

    Best regards,

    Urtzi Larrieta



    ------------------------------
    Urtzi Larrieta Alvarez
    IT Manager
    ECNA Informatica
    Bilbao
    944159688
    ------------------------------



  • 12.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 10 days ago

    Hello All 

    Could anyone please let me know if this issue is resolved.

    because we got a new IBMFS5045 few weeks and having the PowerEdge Servers directly connected to the Storage.

    we have 2 hosts defined as Boot from SAN and its working, I have installed Hyper-V2022, no issues observed sofar.

    please check the below and let me know if this cab be fixed ( degraded status on the Host WWPN's)

    IBM_FlashSystem:IBMFS01:superuser>svcinfo lsportfc -delim , -gui
    id,fc_io_port_id,port_id,type,port_speed,node_id,node_name,WWPN,nportid,status,switch_WWPN,fpma,vlanid,fcf_MAC,attachment,cluster_use,adapter_location,adapter_port_id,fabric_WWN
    0,1,1,fc,16Gb,1,node1,500507680D06C99E,DA0100,active,500507680D46C99E,N/A,N/A,N/A,host,local_partner,1,1,500507680D02C99E
    1,2,2,fc,16Gb,1,node1,500507680D0AC99E,DA0100,active,500507680D4AC99E,N/A,N/A,N/A,host,local_partner,1,2,500507680D02C99E
    2,3,3,fc,N/A,1,node1,500507680D0EC99E,000000,inactive_unconfigured,0000000000000000,N/A,N/A,N/A,none,local_partner,1,3,0000000000000000
    3,4,4,fc,N/A,1,node1,500507680D12C99E,000000,inactive_unconfigured,0000000000000000,N/A,N/A,N/A,none,local_partner,1,4,0000000000000000
    16,1,1,fc,16Gb,2,node2,500507680D06C99F,DA0100,active,500507680D46C99F,N/A,N/A,N/A,host,local_partner,1,1,500507680D02C99F
    17,2,2,fc,16Gb,2,node2,500507680D0AC99F,DA0100,active,500507680D4AC99F,N/A,N/A,N/A,host,local_partner,1,2,500507680D02C99F
    18,3,3,fc,N/A,2,node2,500507680D0EC99F,000000,inactive_unconfigured,0000000000000000,N/A,N/A,N/A,none,local_partner,1,3,0000000000000000
    19,4,4,fc,N/A,2,node2,500507680D12C99F,000000,inactive_unconfigured,0000000000000000,N/A,N/A,N/A,none,local_partner,1,4,0000000000000000
    IBM_FlashSystem:IBMFS01:superuser>svcinfo lshost -delim : 0
    id:0
    name:HVOTN01
    port_count:2
    type:generic
    iogrp_count:2
    status:online
    site_id:
    site_name:
    host_cluster_id:0
    host_cluster_name:HVC-OT-CL01
    protocol:scsi
    status_policy:redundant
    status_site:all
    WWPN:2100XXXXXXXXXXA7
    node_logged_in_count:1
    state:degraded
    WWPN:2100XXXXXXXXXXA6
    node_logged_in_count:1
    state:degraded
    owner_id:
    owner_name:
    portset_id:64
    portset_name:portset64

    IBM_FlashSystem:IBMFS01:superuser>svcinfo lshost -delim : 1
    id:1
    name:HVOTN02
    port_count:2
    type:generic
    iogrp_count:2
    status:online
    site_id:
    site_name:
    host_cluster_id:0
    host_cluster_name:HVC-OT-CL01
    protocol:scsi
    status_policy:redundant
    status_site:all
    WWPN:2100XXXXXXXXXXC3
    node_logged_in_count:1
    state:degraded
    WWPN:2100XXXXXXXXXXC2
    node_logged_in_count:1
    state:degraded
    owner_id:
    owner_name:
    portset_id:64
    portset_name:portset64
    IBM_FlashSystem:IBMFS01:superuser>

    IBM_FlashSystem:KT-IBMFS01:superuser>svcinfo lshost
    id name      port_count iogrp_count status site_id site_name host_cluster_id host_cluster_name protocol owner_id owner_name portset_id portset_name
    0  HVOTN01 2          2           online                   0               HVC-OT-CL01    scsi                         64         portset64
    1  HVOTN02 2          2           online                   0               HVC-OT-CL01    scsi                         64         portset64



    ------------------------------
    Rajarajesh Nelakanti
    ------------------------------



  • 13.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 10 days ago
    Edited by Zaki Jääskeläinen 10 days ago

    Double-check your zoning. It looks like each host WWPN is only logged in on one FS5045 controller. Both host FC ports need to see both controllers. And both controllers need to be accessible on both fabrics (assuming each host FC port is connected to separate switches).



    ------------------------------
    Zaki Jääskeläinen
    ------------------------------



  • 14.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago

    @Zaki Jääskeläinen:

    The hosts are direct attached, so there is no possibility for a host WWPN to login to both FS5045 controllers.

    @Rajarajesh Nelakanti:

    What "status policy" is used for the attached hosts?

    I assume the status policy "complete" is configured. Changing this to "redundant" should solve that issue.

    -statuspolicy redundant | complete
    (Optional) Specifies if complete or redundant policy is used for online status calculations.
    • Complete connectivity is where online host status indicates complete connectivity, with every host port that is logged in to every node through which that host can access volumes, and degraded host status indicates one or more of these nodes does not have a login.
    • Redundant connectivity calculation is where online host status indicates redundant connectivity, where enough host ports are logged in to enough nodes such that the removal of a single node or a single host port allows that host to access all of its volumes. In addition, there must be the same total login count from the host to each node within an I/O group. Degraded host status indicates non-redundant connectivity, in which a single point of failure prevents a host from accessing at least some of its volumes, or there is no single point of failure but the total login count from the host to each node within an I/O group is not the same.


    ------------------------------
    Uwe Schreiber
    System Engineer / Solution Architect
    SVA System Vertrieb Alexander
    Wiesbaden
    ------------------------------



  • 15.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago

    Thanks Uwe. I missed the 'directly'. This is what happens when I try to be useful before my first cup.



    ------------------------------
    Zaki Jääskeläinen
    ------------------------------



  • 16.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago

    Hi @Rajarajesh Nelakanti,

    the overall status for the host will be shown as "online" (using command "lshost") because the host logged in into both nodecanisters at least once.

    If you display the detailed host informations using "lshost <hostname|id>" each direct connected WWPN of the host will get displayed with state "degraded", because each WWPN is only logged in into one nodecanister and not both.

    Regards



    ------------------------------
    Uwe Schreiber
    System Engineer / Solution Architect
    SVA System Vertrieb Alexander
    Wiesbaden
    ------------------------------



  • 17.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago

    Hello @Uwe Schreiber

    both servers are already Redundant.

    is it the reason the NPIV is still enabled ?

    if you are suspecting the cabling issue I can get it rechecked.

    but I am attaching the cabling diagram as well,

    Please check and let me know.

    Thank's in advance



    ------------------------------
    Rajarajesh Nelakanti
    ------------------------------



  • 18.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago
      |   view attached

    host is connected to both IOgroups.



    ------------------------------
    Rajarajesh Nelakanti
    ------------------------------



  • 19.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago

    The port status "degraded" is "works as designed", because that WWPN is not connected to both nodecanisters, because you are using a direct-attachment for each host FC port.

    If one of the direct attached FC ports will get disconnected, the overall host state will change vom "online" to "degraded" as well.



    ------------------------------
    Uwe Schreiber
    System Engineer / Solution Architect
    SVA System Vertrieb Alexander
    Wiesbaden
    ------------------------------



  • 20.  RE: Flashsystem 5045 Ports in DEGRADED mode

    Posted 9 days ago

    Hello Uwe Schreiber

    Thank you so much for responding.

    so, host WWPN status is degraded by design only and not any issue that need to be fixed.

    if that is as designed then I hope we can ignore that message.

    Thank you



    ------------------------------
    Rajarajesh Nelakanti
    ------------------------------