Primary Storage

 View Only
Expand all | Collapse all

SANNAV - No historical performance stats

  • 1.  SANNAV - No historical performance stats

    Posted Thu September 08, 2022 11:42 AM
    We've got SANnav working in our test environment, in that everything is connected, all in sync, no SNMP errors, switches/hosts/storage ports all reporting healthy, no errors/warnings in the event log. But we've hit a weird issue whereby we can't see any historical performance stats, but we can generate real-time stats.

    In the Investigate option on a SAN port, top right you have options of historical stats for 30 mins, 2 hours, etc..., or an option of Real-time. When we chose the default historical stats, we get nothing come up, no data points, "No data to display" on the graph area. However, as soon as we change from historical to real-time the stats appear, data points get written to the graph every 10 seconds, and we get a real-time graph. 

    So to me, logically, if the real-time performance monitoring is working, then the config of the switch and SANnav must be correct, but why is the historical graphs showing no data? We have a mix of switches/directors in Access Gateway mode, and normal switch mode. The normal switches show in the SANnav Inventory show the switches are bound successfully to the SANnav IP address. 

    We are stumped as to why real-time works, and historical does not. We are running SANnav 2.1.1.

    Anyone have any ideas as to the reason why, what we can do to allow SANnav to trap historical data?

    Just to add a curve ball in to this, we have in the same envionment Network Advisor running alongside Spectrum Control, which is trapping historical stats just fine, but the new implementation of SANnav won't trap historical stats. 

    Thanks, Andy

    ------------------------------
    Andy 91717
    ------------------------------

    #PrimaryStorage
    #StorageAreaNetworks
    #Storage


  • 2.  RE: SANNAV - No historical performance stats

    IBM Champion
    Posted Fri September 09, 2022 06:01 AM
    Andy-

    I don't have a solution for you-but I just wanted to let you know that I've seen that type of funky behavior as it relates to the performance data. I've see that if it doesn't have much data, it won't display anything-but if you tighten up the timeframe---it'll display it. Also, I'd suggest moving to 2.2. It has a lot of nice things in it. 

    Bob Mayotte

    ------------------------------
    Robert Mayotte
    ------------------------------



  • 3.  RE: SANNAV - No historical performance stats

    Posted Fri September 09, 2022 08:45 AM
    Small remark for the last part : If you are running a recent version of Spectrum Control, best is to redefine the SAN switches in Spectrum Control using Brocade RestAPI to collect the performance data. Advantage is that Spectrum Control still collects performance data, but without dependency on BNA.


    ------------------------------
    Hans Populaire
    ------------------------------



  • 4.  RE: SANNAV - No historical performance stats

    Posted Fri September 16, 2022 07:47 AM
    Unfortunately we can't integrate direct with Spectrum as our switches have a mixture of FOS levels due to hardware constraints, so we have FOS of 7.4, 8.2, and 9.0 all in the same fabric, and not all of these are able to be integrated with Spectrum direct. As we go through various hardware upgrades over the coming months we hope to get to the point where we can discard BNA altogether and just use Spectrum, with SANNAV as an additional tool also.

    Having historical data though is our preferred goal in SANNAV alongside Spectrum, we've paid for SANNAV, it'll be good to get it working also. The confusing thing is real-time port performance stats works, but historical does not, it's really confusing. The only error we are getting now in SANNAV Events log is:

    "Flow details cannot be processes and displayed because the clock on switch x is not synchronised with the SANnav server"

    Could this also be affecting capturing historical data. Also I have no idea why these flows are not working on our core directors as they use exactly the same NTP servers as SANNAV does, so they should be in sync just fine, even if one is GMT and one is BST, as they are from the same time source surely.

    ------------------------------
    Andy 91717
    ------------------------------



  • 5.  RE: SANNAV - No historical performance stats

    IBM Champion
    Posted Sat September 17, 2022 09:08 AM
    it looks like SANNAV collects flow data from switches but cant process due to time conflict. If remote fosexec is enabled on your fabrics check the time via fosexec and compare it with sannav's linux os.

    ------------------------------
    Nezih Boyacioglu
    ------------------------------



  • 6.  RE: SANNAV - No historical performance stats

    Posted Sat September 17, 2022 12:15 PM
    Looks like you are in a very difficult situation :
    - Spectrum Control via RestAPI does not support FOS 7.X
    - BNA (and as a result also Spectrum Control managed via BNA) does not support FOS 9.X
     
    So Spectrum Control is always missing a part (FOS 7 or FOS 9)

    Due to the cost model of SANNAV, we stay away from FOS 9 (and SANNAV) as long as we can.
    We continue to use BNA which is officially no longer supported, but which does what is expected from BNA. 
    We have not yet invested in the GEN7 64Gbit SAN because our current environment with 32Gbit components are still capable to handle all the workload.

    Most of our SAN performance monitoring is done via Spectrum Control, because advantage is that we can link easily the SAN switch performance data with the actual storage performance data (example performance per LUN) to find root causes of performance issues.


    ------------------------------
    Hans Populaire
    ------------------------------



  • 7.  RE: SANNAV - No historical performance stats

    Posted Tue September 20, 2022 08:32 AM
    Yeah, it's not ideal, although I can confirm BNA will recognise and connect to FOS 9 switches allowing Spectrum to record stats from them, even if not fully supported. Our installation of BNA 14.4.5 will connect through to IBM SAN256B-7 directors, and recognise it's at FOS 9.0.1, but it does not recognise the model number in BNA, and hence Spectrum Control shows it as model = "unavailable". But most importantly SC will record performance stats on the directors for monitoring and problem determination. We have other HPE Synergy back of chassis switches which are also FOS 9.0.1, but in access gateway mode, and the model number of them are recorded successfully, and again we get the performance stats no problem. 

    We don't use BNA for zoning or FOS upgrades, we will do that via CLI, and individual principle switch GUI where required, we're kinda old school for that stuff! But we hope to make full use of SANnav in the fullness of time when we get our FOS in sync.

    ------------------------------
    Andy 91717
    ------------------------------



  • 8.  RE: SANNAV - No historical performance stats

    Posted Tue September 20, 2022 10:47 AM
    OK,

    FYI, due to cost reasons, we decided several years ago to move away from directors and use Enterprise class switches instead :
    Price of a director blade is almost equivalent to enterprise switch with same amount of SAN ports.
    So total price of director (chassis + blades) turns out to be more expensive than same amount of SAN ports via enterprise switches.

    And for interconnects between enterprise switches, today we use QSFP connections.

    On top of that, there is a signifcant difference in total maintenance cost.

    With a decent SAN design, ISL traffic is not a bottleneck.

    Finally, lack of "online firmware" upgrades like you have on directors is not seen as a real disadvantage : in a dual fabric, we perform upgrades of all switches in 1 fabric during same maintenance window.
    We use the interruption during reboot of 1 SAN switch as part of fw upgrade as validation that host multipathing via dual fabric is correctly configured.


    ------------------------------
    Hans Populaire
    ------------------------------



  • 9.  RE: SANNAV - No historical performance stats

    Posted Wed September 21, 2022 11:05 AM
    Hello Andy,
    I don't know , if your prb. is already solved. I am just checking the prereqs for upgrading SANnav to release 2.2.1 and found this statement in the release notes from Brocade:

    When configuring the VM for SANnav installation, please make sure the MTU size of the network interface is set to 1500, otherwise SANnav will not receive Historical Performance data.

    Perhaps it may cause your problem.
    Regards, Martin

    ------------------------------
    Martin Hansen
    ------------------------------



  • 10.  RE: SANNAV - No historical performance stats

    IBM Champion
    Posted Wed September 21, 2022 12:05 PM
    it's a treasure Martin!

    ------------------------------
    Nezih Boyacioglu
    ------------------------------



  • 11.  RE: SANNAV - No historical performance stats

    Posted Fri September 23, 2022 08:07 AM
    We did check, our Linux box is set to MTU 1500, but good shout anyway.

    So we think we're getting closer to the solution, we've been checking the perfmon-be.log file on the Linux box, and it is showing errors:
    • SNMP get operation failed for port stats for switch: n.n.n.n port: etc....
    • Error in SNMP get. Device: n.n.n.n, ErrorIndex :0, ErrorStatus :16, Errorstring :Authorisation error
    So we assume there is some wrinkle in the SNMP settings on the switches which we need to update to get historical perf stats to be captured and downloaded to SANnav. We'll have a play and see how we get on

    ------------------------------
    Andy 91717
    ------------------------------



  • 12.  RE: SANNAV - No historical performance stats

    Posted Wed September 28, 2022 10:55 AM
    So we are all sorted now, SANNAV is working, trapping all performance stats. It was a wrinkle in the SMNPv3 settings on each of the switches which we needed to change. We ended up, whether this is right or wrong, simply going on to each switch and pre-configuring the SNMPv3 settings:

    - adding a new FOS user = sannav
    - adding SNMPv3 config on the switches for sannav user with MD5 Auth and noPriv Priv Protocol settings, and trap for the SANNAV server, with notify type of TRAP.

    Once all the switches were pre-configured, and SANNAV was set to go with no fabrics, i.e. from scratch, we added a switch per fabric in the SANNAV GUI with used the manual option to adding in the same details, i.e. MD5 and sannav user, and hey presto, fabric added, and performance stats logging worked ok. I'm sure if we did read more carefully the manual we may have done this right first time, or maybe used another option instead of pre-configuring our switches, but hey, it works now, so we're happy ;-)

    ------------------------------
    Andy 91717
    ------------------------------