Originally posted by: sajid1

Hi All,

After failure of UPS AIX server went down and after starting server
some mount point are not showing.Find below for your information.

1. mount -a

mount: 0506-324 Cannot mount /dev/hd1 on /home: The requested resource is busy.
mount: 0506-324 Cannot mount /proc on /proc: The requested resource is busy.
mount: 0506-324 Cannot mount /dev/hd10opt on /opt: The requested resource is bus
y.
mount: 0506-324 Cannot mount /dev/fslv00 on /backup: There is a request to a dev
ice or address that does not exist.
mount: 0506-324 Cannot mount /dev/fslv01 on /archive: There is a request to a de
vice or address that does not exist.
mount: 0506-324 Cannot mount /dev/fslv02 on /app: There is a request to a device
or address that does not exist.
mount: 0506-324 Cannot mount /dev/fslv03 on /backup/indexes: A file or directory
in the path name does not exist.
mount: 0506-324 Cannot mount /dev/fslv04 on /app/indexes: A file or directory in
the path name does not exist.
mount: 0506-324 Cannot mount /dev/fslv05 on /app/data: A file or directory in th
e path name does not exist.
#

1. df -g

Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 1.00 0.49 52% 2012 2% /
/dev/hd2 3.00 0.53 83% 39531 24% /usr
/dev/hd9var 1.00 0.52 48% 637 1% /var
/dev/hd3 3.00 1.49 51% 4889 2% /tmp
/dev/hd1 1.00 0.57 43% 4293 4% /home
/proc - - - - - /proc
/dev/hd10opt 12.00 2.74 78% 1149 1% /opt
#

Regards,

Originally posted by: OTIT

I think that some filesystem super block corrupt or dirty when UPS failure. The first, use command "#fsck -y /dev/lv_name" for unmounted filesystems and mount again.

Originally posted by: sajid1

Hi,

Thanks for replying.
I performed as you said but it occured below error.

1. fsck -y /dev/fslv02

The current volume is: /dev/fslv02
Open volume exclusive read or write returned, rc = 6
fsck: 0507-289 Device unavailable or locked by another process.
Cannot continue.

waiting you.

Regards

Originally posted by: OTIT

Sorry, It not about dirty filesystems, The filesystems "/app" not mounted but LV(/dev/fslv02) used by someone.

1. Check the folder /app. Without /app create a new
by command . . . # mkdir /app

2. From the output of your >>
Device unavailable or locked by another process.Cannot continue.
As it has in some procress locking LV. User command "#fuser /app" or "#fuser /dev/fslv02" to check procress. If you see any procress and then check its by command
"#ps -p PID". If the procress is no significant kill its all.

3. Try to mount again. If you can mount, With the other filesystems to do the same and reboot when testing time.

Originally posted by: j.gann

are the volume groups that contain the respective lvs varied on?
compare output of "lsvg" and "lsvg -o"
maybe you have some volumegroups set to not varyon automatically at boot time.

Originally posted by: sajid1

Hi,

are the volume groups that contain the respective lvs varied on?
--Sorry i did not get you
Find below output for your information.

1. lsvg

rootvg
datavg

1. lsvg -o

rootvg
#

Thanks,

Originally posted by: j.gann

1. read manuals of commands mentioned to you
2. don't expect people spoon-feeding you the solution

Originally posted by: sajid1

I am not AIX admin I am Oracle DBA so not awared much command of AIX.
Thanks for patience.

Originally posted by: sajid1

Hi,

1. Check the folder /app. Without /app create a new
by command . . . # mkdir /app

• 1. mkdir /app

mkdir: 0653-358 Cannot create /app.
/app: Do not specify an existing file.

2. From the output of your >>
Device unavailable or locked by another process.Cannot continue.
As it has in some procress locking LV. User command "#fuser /app" or "#fuser /dev/fslv02" to check procress. If you see any procress and then check its by command

• 1. fuser /app

/app:

1. fuser /dev/fslv02

/dev/fslv02:
--No process

1. cd /app
2. pwd

/app
#ls
---Nothing output whereas all the Oracle Application files is located at this location(/app).

Many Thanks

Originally posted by: ddmurray

As j.gann eluded to above it looks like your datavg is not varied on.

try "varyonvg datavg" and then try to mount apps

lsvg - The lsvg command displays information about volume groups

lsvg -o - Lists only the active volume groups (those that are varied on). An active volume group is one that is available for use.

Good Luck

Originally posted by: sajid1

Hi,

Many Thanks.

I performed as per your action plan but it given below error.

1. varyonvg datavg

0516-013 varyonvg: The volume group cannot be varied on because
there are no good copies of the descriptor area.

Regards,

Originally posted by: sajid1

Hi,

I am waiting your answer.

Thanks,

Originally posted by: Kosala

Try to post the following outputs. Some one might be able to help you.

lspv
lsdev -Cc disk
errpt | head 10

Looks like your disks cannot be found. At this point errpt should be your best friend.

Kosala

Originally posted by: sajid1

Hi,

Find below output as required.

1. lspv

hdisk0 00c2c69e7770e021 rootvg active
hdisk1 00c2c69e656bded4 rootvg active
hdisk2 00c2c69e5534042d datavg

1. lsdev -Cc disk

hdisk0 Available 04-08-00-4,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 04-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk2 Available 08-08-02 1814 DS4700 Disk Array Device

1. errpt | head 10

10: A file or directory in the path name does not exist.

1. errpt

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
2E493F13 0223092510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092310 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092210 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092210 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092210 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092210 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092210 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0223092210 P H hdisk2 ARRAY OPERATION ERROR
BFE4C025 0212200810 P H sysplanar0 UNDETERMINED ERROR
A6DF45AA 0212200710 I O RMCdaemon The daemon is started.
2E493F13 0212200710 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200610 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200510 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
2E493F13 0212200410 P H hdisk2 ARRAY OPERATION ERROR
BFE4C025 0212200310 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0212200410 T O errdemon ERROR LOGGING TURNED ON
BFE4C025 0209161810 P H sysplanar0 UNDETERMINED ERROR
BFE4C025 0209161610 P H sysplanar0 UNDETERMINED ERROR
A6DF45AA 0209161510 I O RMCdaemon The daemon is started.
A6DF45AA 0209161410 I O RMCdaemon The daemon is started.
9DBCFDEE 0209161410 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0131123310 I O RMCdaemon The daemon is started.
C86ACB7E 0131123310 I H dac1 ARRAY CONFIGURATION CHANGED
A6DF45AA 0131123210 I O RMCdaemon The daemon is started.
9DBCFDEE 0131123210 T O errdemon ERROR LOGGING TURNED ON
7B82439B 0826204708 U U SYSJ2 FILE SYSTEM CORRUPTION
ABED1BA8 1231102907 U U SYSJ2 FILE SYSTEM CORRUPTION
ABED1BA8 1231102607 U U SYSJ2 FILE SYSTEM CORRUPTION
#

Thanks

Originally posted by: Kosala

Ok... it seems I made a mistake with the head command it should've been head -n 10. Anyways, looks like your array controller is failing. You need to talk to you IBM support guys... you may give a try with the "diag" command if you can.

From what I see, your hdisk2, which is part of the datavg, which is defined in the DS4700 storage box has a problem. Since lsdev command says it's available, you might need to check the PV STATE of this volume.

lspv hdisk2

and a detailed errpt would help as well.

errpt -aj 2E493F13

Kosala

Originally posted by: sajid1

I can not take support IBM guys now because contracts have finished there are discussing about it to renewal but obviously not now.
On test server that error is coming so just guide me to sort out it i will be thankful for this.

1. lspv hdisk2

0516-010 : Volume group must be varied on; use varyonvg command.
PHYSICAL VOLUME: hdisk2 VOLUME GROUP: datavg
PV IDENTIFIER: 00c2c69e5534042d VG IDENTIFIER 00c2c69e00004c000000011be
24b2fc2
PV STATE: ???????
STALE PARTITIONS: ??????? ALLOCATABLE: ???????
PP SIZE: ??????? LOGICAL VOLUMES: ???????
TOTAL PPs: ??????? VG DESCRIPTORS: ???????
FREE PPs: ??????? HOT SPARE: ???????
USED PPs: ??????? MAX REQUEST: ???????
FREE DISTRIBUTION: ???????
USED DISTRIBUTION: ???????

The second command is very big file in size i cannot copy paste here if possible provide me personal id so that i can attach it to you.

Many Thanks,

Originally posted by: SystemAdmin

chk DS4700,maybe the Rain have been failer.