Hi Power Community,

We encounter issue with SAS Raid disk replcament, where customer was able to unmirror & reduce rootvg with hdisk1 but while removing with smitty sisdam menu, the hdisk1 device shown busy, while checking with fuser -cu /dev/rhisk1, we found alots of root process associated with hdisk1 & similar with hdisk0.

The AIX version is 7100-05-09 (which i know out of support) configured with PowerHA as sencondary node.

Following are the fuser ourput details, if anyone face similar issue, please advise how to replace SAS0 Raid adapter hdisk1, which is default configuration.

root@horop2p02:/>fuser -cu /dev/rhdisk1

/dev/rhdisk1:        1c(root) 3670176c(root) 3735564c(root) 3997922c(root) 4849900r(root) 4980918c(root) 5046440r(root) 5374068c(root) 7143644r(root) 7274536r(root) 7340220c(root) 7405588c(root) 7536798r(root) 7733376c(root) 7864528c(root) 7995392r(root) 8192048c(root) 8257668c(root) 8388656c(root) 8454182r(root) 8585260c(root) 8716420c(root) 8782022c(root) 8978496c(root) 9044028r(sisips) 9109628c(root) 9371728r(root) 9437306c(root) 9502818c(root) 9568276r(root) 9634002c(root) 9699462c(root) 9896020r(root) 10027074c(daemon) 10158162c(root) 10223726r(root) 10289224c(root) 10354756r(root) 10420340c(root) 10485856c(root) 10551440r(root) 10682450c(root) 10813606c(root) 10879004c(root) 11075720r(root) 11337868r(root) 11534484c(root) 11665610r(appdmon) 11993230c(root) 12058788c(root) 12124326c(root) 12189826c(root) 12386430c(root) 12517544r(root) 12583064r(patrolag) 13041806c(root) 13107344r(root) 13238430r(root) 13303810r(root) 13369530c(root) 13697238r(root) 13828134r(root) 13893844c(root) 14024812r(root) 14090448r(root) 14155966r(root) 14221546c(patrolag) 14286854r(root) 14352510r(root) 14418118r(root) 14483664r(root) 14549088r(patrolag) 14614724c(root) 14680078r(root) 14745626r(root) 14942340c(root) 15073290r(root) 15138834r(patrolag) 15204478r(root) 15269910r(root) 15335672r(root) 15466532r(sisips) 15532098r(patrolag) 15663344r(root) 15794404r(root) 15859800r(root) 15990932r(root) 16121984r(patrolag) 16318520r(root) 16449544r(root) 17236040r(root)

root@horop2p02:/>

root@horop2p02:/>fuser -cu /dev/rhdisk0

/dev/rhdisk0:        1c(root) 3670176c(root) 3735564c(root) 3997922c(root) 4849900r(root) 4980918c(root) 5046440r(root) 5374068c(root) 7143644r(root) 7274536r(root) 7340220c(root) 7405588c(root) 7536798r(root) 7733376c(root) 7864528c(root) 7995392r(root) 8192048c(root) 8257668c(root) 8388656c(root) 8454182r(root) 8585260c(root) 8716420c(root) 8782022c(root) 8978496c(root) 9044028r(sisips) 9109628c(root) 9371728r(root) 9437306c(root) 9502818c(root) 9568276r(root) 9634002c(root) 9699462c(root) 9896020r(root) 10027074c(daemon) 10158162c(root) 10223726r(root) 10289224c(root) 10354756r(root) 10420340c(root) 10485856c(root) 10551440r(root) 10682450c(root) 10813606c(root) 10879004c(root) 11075720r(root) 11337868r(root) 11534484c(root) 11665610r(appdmon) 11993230c(root) 12058788c(root) 12124326c(root) 12189826c(root) 12386430c(root) 12517544r(root) 12583064r(patrolag) 13041806c(root) 13107344r(root) 13238430r(root) 13303810r(root) 13369530c(root) 13697238r(root) 13828134r(root) 13893844c(root) 14024872c(root) 14090448r(root) 14155966r(root) 14221546c(patrolag) 14286854r(root) 14352510r(root) 14418118r(root) 14483664r(root) 14549088r(patrolag) 14614724c(root) 14680078r(root) 14745626r(root) 14876874c(root) 14942340c(root) 15073290r(root) 15138834r(patrolag) 15204478r(root) 15269910r(root) 15335672r(root) 15466532r(sisips) 15532098r(patrolag) 15663344r(root) 15794404r(root) 15859800r(root) 15990932r(root) 16121984r(patrolag) 16318520r(root) 16449544r(root)

root@horop2p02:/>

If we are killing the process by kill -9 <PID> shown above, LPAR reboot abnormally and while booting hdisk1 shows again with all root & non-root related process with hdisk1, which is now shown as none. 

root@XXXX:/>lspv

hdisk0          00c27541c448ab90                    rootvg          active

hdisk1          00c2754162a2daed                    None

hdiskpower2     00c2754185a3b46d                    backupvg        active

hdisk26         none                                None

hdisk27         none                                None

hdisk28         none                                None

hdisk29         none                                None

hdisk30         none                                None

hdisk31         none                                None

hdisk32         none                                None

hdisk33         none                                None

hdisk34         none                                None

hdisk35         none                                None

hdisk36         none                                None

hdisk37         none                                None

hdisk38         none                                None

hdisk39         none                                None

hdisk40         none                                None

hdisk41         none                                None

hdisk42         none                                None

hdisk43         none                                None

hdisk44         none                                None

hdisk45         none                                None

hdisk46         none                                None

hdisk47         none                                None

hdisk48         none                                None

hdisk49         none                                None

hdiskpower3     00c2754185d80d16                    caavg_private   active

hdiskpower4     00c2754185d28166                    open2vg

hdiskpower0     none                                None

hdiskpower1     none                                None

root@XXXXX02:/>

BR / Zuberi  

------------------------------
Muhammad Faraz Zuberi
------------------------------

Hi Zuberi

Check out this technote:

https://www.ibm.com/support/pages/client-tasks-be-performed-order-prepare-ibm-power-system-concurrent-sas-or-ssd-disk-replacement-aix-or-vios

Regarding the fuser -cu output:

       -c
            Reports on any open files in the file system containing File.

i.e. fuser -cu /dev/rhdisk0 reports all processes with open files in the root file system.  To identify processes accessing a specific file, omit the -c flag.

HTH

------------------------------
Chris Wickremasinghe
IBM
------------------------------

Hi Faraz

I recommend using an available empty disk bay to install the new disk, then create a new RAID0 array on it and re-mirror rootvg to the new disk.

Once the mirror is completed and verified, and the old/failed disk is confirmed to be no longer part of rootvg, you can try removing the old device definition using:

rmdev -Rdl hdiskX

This approach avoids killing the processes reported by fuser, especially since similar processes are shown against both hdisk0 and hdisk1.

Thank you 

------------------------------
Anas AlSaleh
IBM Power Systems Software Specialist
Saudi Business Machines ( SBM )
Riyadh
------------------------------