Originally posted by: grukrz1

Hello,

it is really weird issue I never met before.
The are two VIO servers setup on small P6 520 model (without I/O drawer) each one has one physical dual-port 4Gbit FC card dedicated. There are no other FC cards in that small box.
All was working fine for months on these systems, until they were stopped an started again. First after startup I noticed that SAN storage is not available anymore (all EMC's plane hdisks and hdiskpower devices are only defined). When I run 'cfgmgr -l fcs0/fcs1" both completes quiet without any error. the whole cfgmgr complains only that VirtualTargetDevices cant be made available due to parrent device hdiskpower is in not available state). What EMC found in old emcgrab (done before VIOS re-activation) that WWNs are now different that these seen currently in the system. All HBAs are the same dual port models:

Part Number.................10N7255
Serial Number...............13B5747DLB
Manufacturer................001C
EC Level....................D76626
Customer Card ID Number.....5774
FRU Number.................. 10N7255

with the latest firmware installed:

Device Specific.(Z9)........ZS2.71X15

The firmware was installed a few mounts ago (online - adapter by adapter), and even it was successfully re-loaded (-f) on IBM's suggestion but still the same problem.

Anyone knows how it could happed that WWNs of physical cards are different after VIOS restart? All VSCSI devices are now defined because of SAN disks access problem due to WWNs change.

No NPIV (4Gbit HBA), physical assigned, no LPM, PowerVM standard, ioslevel 2.1.3.10-FP-23, HMC V7R7.1.0.1, EL350_071...

thx in advance for any hint,
regards, K.
#AIX-Forum

Originally posted by: The_Doctor

I suspect you have already done these checks, so my suggestions may be useless. Anyway,

1. I would not rely on old info from an emcgrab as the ONLY evidence to validate the assumption that the WWPNs have changed.... so

• what do your SAN switches say for all 4 ports.... before the change & now ?
• in other words, your SAN zoning will have the OLD WWPNs & your SANs should now be reporting new WWPNs as well.... how do they compare ? 100% match, completely new WWPN, only one digit off, or ?

2. Do the WWPNs, as shown on the SAN, match the output from "lscfg -vpl fcsX" commands ? Check all 4 fcsX ports.

• Again, how do they match up.... 100% match, completely new WWPN, only one digit off, or ?

3. Do the symptoms change again, following a complete POWER OFF / POWER ON cycle ?

• a.k.a. do the WWPNs change a 2nd time ?

I'm also surprised the problem hits BOTH FC adapters at the same time, which suggests to me.....

• flaky firmware (not too likely, since other customers should have reported this problem as well), or
• some unknown / yet to be discovered SAN changes.

If you ever figure it out, I'd be interested in hearing the resolution.
#AIX-Forum

Originally posted by: grukrz1

1) I'll need to ask SAN team about that.
From EMC I already got confirmation that Host Initiators configured in symmetrix are exactly the same as on my HBA labels and configuration was not changed since Aug 23,2010. The problem definitely occured after both VIO Servers were re-activating what happened in begin of this week - since when new WWNs are seen on HBAs. I can say more, before the reboots, on EMC disks on both VIOSes were avaialble and two client LPARs were booted from them (VSCSI disks)...

cfgmgr -l fcsX completes quiet and on "lsattr -El fscsiX" output I see "attach switch" so I guess SAN team will see on switch side new WWNs as well..

In each incorrect WWN last two bytes of the WWN number are changed:
10000000C97DXXXX

All EMC disks/hdiskpower devices are defined because of the changed WWNS...

2) I'll need to ask SAN team about that (but I guess on SAN switches they are also seen new/incorrect WWNs ...)

3) No, there are no more WWN changes doing another shutdown and again VIO lpar activation.

fist time when problem occured, the whole server was not powered off. only all LPARs were shutdown to modify minimum processing unit in each profile (increae). That's all. I only know from history, that since 2010/07/09 when "diag -d fcsX -T download" were run to upgrade HBAs firmware (online) no such restarts were done.
diag -d fcsX -T "download -f " run once again compete successfully and WWNs is not changing again after shutdown/activate VIO LPARs.

Right now I don;t know if I should ask SAN and Storage to change in configuration original WWNs (which are on the labels) with the fake/new ones...

really, really weird issue...
#AIX-Forum

Originally posted by: grukrz1

most probable HBAs have defect - that is why WWNs changed....(serial numbers are still the same:-)
IBM sent one to FC team for analyzing...
#AIX-Forum