AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
 View Only
Expand all | Collapse all

Cannot remove fcsX adapter from AIX

  • 1.  Cannot remove fcsX adapter from AIX

    Posted Fri June 10, 2011 04:56 AM

    Originally posted by: Pavel_Pokorny


    Hello to all,
    we are trying to remove Fibre-Channel HBA fcs1 from our AIX OS. When we run rmdev over this fcs1 adapter we are receiving this error:

    root@cpia /tmp/rman #rmdev -dl fcs1 -R
    Method error (/usr/lib/methods/ucfgdevice):
    0514-062 Cannot perform the requested function because the
    specified device is busy.

    Under this fcs1, there are no other devices:

    root@cpia /tmp/rman #lsdev -C|grep 03-01
    fcnet1 Defined 03-01-01 Fibre Channel Network Protocol Device
    fcs1 Available 03-01 4Gb FC PCI Express Adapter (df1000fe)
    fscsi1 Available 03-01-02 FC SCSI I/O Controller Protocol Device

    So we tried to run fuser to find the processes holding this devices:

    root@cpia /tmp/rman #fuser -Vxuc /dev/fscsi1
    /dev/fscsi1:
    1c(root)
    151726c(root)
    159968c(root)
    167986r(root)
    184486c(root)
    221410c(root)
    229428r(root)
    245796c(root)
    258252c(root)
    262302c(root)
    270352r(root)
    274588r(root)
    278714r(root)
    282782c(root)
    286788r(cpiadm)
    290994c(root)
    295164r(root)
    + .+
    + .+
    + .+
    + .+

    The question is: Why are all this processes holding our fcs1 devices? The device is not used anymore, all the tape drives that were zoned to this adapter were successfully removed.

    We also tried to run fuser on other similar machine and no processes were using fcs1 on that AIX:

    root@cwpa / #fuser -Vxuc /dev/fscsi1
    root@cwpa / #

    So, what is the correct procedure to remove such a device and also is there any explanation why are all these processes using our device?
    Thanks a lot in advance,
    Pavel
    #AIX-Forum


  • 2.  Re: Cannot remove fcsX adapter from AIX

    Posted Fri June 10, 2011 09:05 AM

    Originally posted by: dhunje


    What AIX version you are running?
    I have similar problem on 6100-06-03/04/05 and it has something with sfwcomm device.

    Did you try to run cfgmgr after (unsuccessful) removal. In my case next error appear:

    Method error (/usr/lib/methods/cfgstorfworkcom -l sfwcomm0 ):
    0514-040 Error initializing a device into the kernel.
    Method error (/usr/lib/methods/cfgstorfworkcom -l sfwcomm0 ):
    0514-040 Error initializing a device into the kernel.
    I have to reboot server to make error disappear.
    #AIX-Forum


  • 3.  Re: Cannot remove fcsX adapter from AIX

    Posted Fri June 10, 2011 09:10 AM

    Originally posted by: SystemAdmin


    Pavel , Can you do a : >lsdev -CH | grep fcs1

    fcs1 Available 17-08 FC Adapter

    Then do the same with the location code .. in this case 17-08 :
    lsdev -CH | grep 17-08

    That will show you any devices available or defined still linked to that adapter :

    In my case :
    >lsdev -CH | grep 17-08
    fcnet1 Defined 17-08-02 Fibre Channel Network Protocol Device
    fcs1 Available 17-08 FC Adapter
    fscsi1 Available 17-08-01 FC SCSI I/O Controller Protocol Device
    rmt.2b01.0 Available 17-08-01-PRI IBM 3580 Ultrium Tape Drive (FCP)
    rmt.2b02.0 Available 17-08-01-PRI IBM 3580 Ultrium Tape Drive (FCP)
    rmt.2b03.0 Available 17-08-01-PRI IBM 3580 Ultrium Tape Drive (FCP)
    rmt.2b04.0 Available 17-08-01-PRI IBM 3580 Ultrium Tape Drive (FCP)
    rmt.2b05.0 Available 17-08-01-PRI IBM 3580 Ultrium Tape Drive (FCP)
    smc0 Available 17-08-01-PRI IBM 3584 Library Medium Changer (FCP)
    smc1 Available 17-08-01-ALT IBM 3584 Library Medium Changer (FCP)

    That , hopefully, will show you what needs to be dropped to free your adapter.

    Good Luck ,,, Charlie
    #AIX-Forum


  • 4.  Re: Cannot remove fcsX adapter from AIX

    Posted Fri June 10, 2011 09:12 AM

    Originally posted by: SystemAdmin


    Pavel , My apologies , I see you already did that , It's a little early here ...

    Charlie
    #AIX-Forum


  • 5.  Re: Cannot remove fcsX adapter from AIX

    Posted Fri June 10, 2011 01:30 PM

    Originally posted by: shargus


    Try doing a lscfg, find the fcs1 device, then see if there are any other devices under it that belong to that path.

    Next, try doing lspath, and see if anything is using fscsi1. We have had problems in the past with disk arrays leaving behind a dac device from the array controller that wouldn't go away. We ended up having to reboot to get the device to let go.
    #AIX-Forum


  • 6.  Re: Cannot remove fcsX adapter from AIX

    Posted Mon June 13, 2011 05:08 AM

    Originally posted by: Pavel_Pokorny


    Hello,
    version of our AIX is 6100-04-02-1007.
    Under our FC HBa we already removed all children devices, as you can see here:

    root@cpia /tmp/rman #lsdev -C|grep 03-01
    fcnet1 Defined 03-01-01 Fibre Channel Network Protocol Device
    fcs1 Available 03-01 4Gb FC PCI Express Adapter (df1000fe)
    fscsi1 Available 03-01-02 FC SCSI I/O Controller Protocol Device

    I would appreciate any help,
    Pavel
    #AIX-Forum


  • 7.  Re: Cannot remove fcsX adapter from AIX

    Posted Tue June 14, 2011 07:55 AM

    Originally posted by: flodstrom


    Just curious, did you check what processes are blocking the removal of the HBA?

    Any backup software that is still running?

    Anything starting up automatically after killing it (inittab)?

    Were there only tape devices on that HBA?

    Does a "cfgmgr" take an abnormal time to finish or even stopping in the middle?

    Once we had a HBA device that just wouldn't allow itself to be unconfigured/removed properly and we could not just shutdown the machine either. We then went through the procedures for physically replacing a card on a live machine without actually removing the card. This involves cutting the power to the PCI or PCIe slot where the HBA is located. Doing this power switching on the slot apparently is more or less like a simulated reboot from the HBA's point of view. After this all was okay again for us.

    However this is really like a last resort kind of procedure. You need to do things in the correct order or you may face an emergency shutdown or a system panic. Needless to say you really need to know what you are doing! I did not do this on our system, it was an IBM technichian that did it for us :).

    Also, if you really can't reboot the system I would advice you to contact IBM support for this. They will be able to walk you through the correct procedures.
    #AIX-Forum


  • 8.  Re: Cannot remove fcsX adapter from AIX

    Posted Tue June 14, 2011 01:10 PM

    Originally posted by: j.gann


    your fuser command is not doing what you think it does (man fuser , option -c)

    lsdev -p fcsX ?
    lsdev -p <any child device>?
    #AIX-Forum


  • 9.  Re: Cannot remove fcsX adapter from AIX

    Posted Wed June 15, 2011 05:04 PM

    Originally posted by: Pavel_Pokorny


    Hello,
    we are sure that there are no child devices under fcs1, see:
    root@cpia /usr/local/cluster #lsdev -p fcs1
    fcnet1 Defined 03-01-01 Fibre Channel Network Protocol Device
    fscsi1 Available 03-01-02 FC SCSI I/O Controller Protocol Device

    Any backup software that is still running?
    • We are using TSM SA and BA client on this AIX, but I am almost sure that this is not the problem as we are using it the same way also on the other AIX machines.

    Were there only tape devices on that HBA?
    • Yes, this HBA was used only for tapes connecion. All the tape devices were correctly deleted using rmdev -R -dl fcs1.

    J.gann, do you now the right procedure how to find the processes or threads that are using our fcs1, fscsi1 devices?

    Thanks, Pavel
    #AIX-Forum


  • 10.  Re: Cannot remove fcsX adapter from AIX

    Posted Thu June 16, 2011 12:11 AM

    Originally posted by: j.gann


    > do you now the right procedure how to find the processes or threads that are using our fcs1, fscsi1 devices?

    just read my post above
    leave the -c out

    and inspect lsdev -p fscsi1
    #AIX-Forum


  • 11.  Re: Cannot remove fcsX adapter from AIX

    Posted Mon June 20, 2011 11:38 AM

    Originally posted by: Jebby


    This process seems to be apps related 286788 , try ps -ef |grep 286788 and if you find it not use full then kill that. you can check the same with other root process as well ,
    I cant C it can be rmdev till any single process is using it.
    HINT : Be care full
    #AIX-Forum


  • 12.  Re: Cannot remove fcsX adapter from AIX

    Posted Mon June 20, 2011 02:37 PM

    Originally posted by: christianIBM


    test
    #AIX-Forum