Originally posted by: SystemAdmin

One of our R&D move the file /usr/lib/libcrypt.a to libcrypt.a__ in our AIX 5.3 server. After that, no one can access server at all. Here's the message:
> ls
Could not load program ls:
Could not load module /usr/lib/libc.a(shr.o).
Dependent module libcrypt.a(shr.o) could not be loaded.
Could not load module libcrypt.a(shr.o).
System error: No such file or directory
Could not load module ls.
Dependent module /usr/lib/libc.a(shr.o) could not be loaded.
Could not load module .

I have serial access to the server and try to boot from CD. But I don't know what else I can continue to move the file back.

Thanks,

David
#AIX-Forum

Originally posted by: j.gann

you might need a hard reset to stop the system (sync, shutdown, halt, telinit should be unusable now).
boot from cd/dvd media or nim spot, select "access a rootvg", this will give you a limited shell and allow you to move the file back.
#AIX-Forum

Originally posted by: SystemAdmin

Hi there,

Still didn't work:(

Yes, I did try to "access a rootvg" then it come out the following:

Type the number of your choice and press Enter. 1) Access this Volume Group and start a shell 2) Access this Volume Group and start a shell before mounting filesystems 99) Previous Menu Choice 99: 1 Importing Volume Group... 0516-1791 redefinevg: lvm_rec for PV hdisk1 has VGID 0000000000000000 (0004fc690000d6000000011542caea66 expected) PV Status: hdisk0 0004fc695ac772af PVACTIVE 0004fc6942caea0b NONAME varyonvg: Volume group rootvg is varied on. 0516-1281 synclvodm: WARNING, lv control block of hd5 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd6 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd8 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd4 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd2 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd9var has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd3 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd1 has been over written. 0516-1281 synclvodm: WARNING, lv control block of hd10opt has been over written. 0516-1281 synclvodm: WARNING, lv control block of fwdump has been over written. 0516-1281 synclvodm: WARNING, lv control block of lg_dumplv has been over written. 0516-1281 synclvodm: WARNING, lv control block of loglv00 has been over written. 0516-1281 synclvodm: WARNING, lv control block of DB has been over written. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. 0516-622 synclvodm: Warning, cannot write lv control block data. rootvg getlvodm: option requires an argument -- l Usage: getlvodm -a LVdescript -B LVdescript -b LVid -c LVid -C -d VGdescript -e LVid -F -g PVid -h -j PVdescript -k -L VGdescript -l LVdescript -m LVid -p PVdescript -r LVid -s VGdescript -t VGid -u VGdescript -v VGdescript -w VGid -y LVid -G LVdescript 0516-104 lquerylv: Incorrect LV identifier. The format must be VG_identifier.LV_minor_number Could not find "/" and/or "/usr" filesystems. Exiting to shell. # pwd / # cd /usr/lib # ls /usr/bin/ksh: ls: not found

I am not sure why the programmer move the file /usr/lib/libcrypt.a , then we lost all the connection and even we can use any shell command at all.
Please advise.

Thanks,

David
#AIX-Forum

Originally posted by: j.gann

never seen the like. time for a support call imo
#AIX-Forum

Originally posted by: SystemAdmin

Hi Joachim,

I did. There's no response from them yet...

Thanks,

David H.
#AIX-Forum

Originally posted by: SystemAdmin

Hi there,

Still didn't work:(

Yes, I did try to "access a rootvg" then it come out the following:

Type the number of your choice and press Enter.

1) Access this Volume Group and start a shell
2) Access this Volume Group and start a shell before mounting filesystems

99) Previous Menu

Choice 99: 1
Importing Volume Group...
0516-1791 redefinevg: lvm_rec for PV hdisk1 has VGID 0000000000000000 (0004fc690000d6000000011542caea66 expected)
PV Status: hdisk0 0004fc695ac772af PVACTIVE
0004fc6942caea0b NONAME
varyonvg: Volume group rootvg is varied on.
0516-1281 synclvodm: WARNING, lv control block of hd5
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd6
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd8
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd4
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd2
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd9var
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd3
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd1
has been over written.
0516-1281 synclvodm: WARNING, lv control block of hd10opt
has been over written.
0516-1281 synclvodm: WARNING, lv control block of fwdump
has been over written.
0516-1281 synclvodm: WARNING, lv control block of lg_dumplv
has been over written.
0516-1281 synclvodm: WARNING, lv control block of loglv00
has been over written.
0516-1281 synclvodm: WARNING, lv control block of DB
has been over written.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
0516-622 synclvodm: Warning, cannot write lv control block data.
rootvg
getlvodm: option requires an argument -- l
Usage: getlvodm -a LVdescript -B LVdescript -b LVid -c LVid
-C -d VGdescript -e LVid -F -g PVid -h -j PVdescript
-k -L VGdescript -l LVdescript -m LVid -p PVdescript
-r LVid -s VGdescript -t VGid -u VGdescript -v VGdescript
-w VGid -y LVid -G LVdescript
0516-104 lquerylv: Incorrect LV identifier. The format must be
VG_identifier.LV_minor_number
Could not find "/" and/or "/usr" filesystems.
Exiting to shell.

1. pwd

/

1. cd /usr/lib
2. ls

/usr/bin/ksh: ls: not found

I am not sure why the programmer move the file /usr/lib/libcrypt.a , then we lost all the connection and even we can not use any shell command at all.
Please advise.

Thanks,

David
#AIX-Forum

Originally posted by: shargus

That kinda looks like you are not importing the right disk.
Are you sure hdisk1 is your current rootvg?

Maybe it's an old rootvg?
#AIX-Forum

Originally posted by: shargus

Also, try "start a shell BEFORE mounting filesystems".

That will give you a chance to make sure the rootvg is good, and you can fsck the lvols before mounting them.

Be aware - once you mount /usr, it will start using the programs (like "ls") from THAT /usr, not the one from the CD. If there's a library mismatch or something, then these commands quit working. Even "reboot" quits working - you have to hard-reset (which can get really ugly if you do something like rename a file, then hard-reboot before the buffers get flushed out to disk).

I would recommend booting up a shell BEFORE mounting the filesystems, then mount /dev/hd2 as /mnt.
Then, rename the file (which is now /mnt/lib instead of /usr/lib).
Then, unmount /dev/hd2 and reboot.
#AIX-Forum

Originally posted by: SystemAdmin

Yes, I will try your way again.

Thanks,

David H.
#AIX-Forum

Originally posted by: broberts

If accessing specifically hdisk0 doesn't work using the "Access a Root VG BEFORE mounting file systems" method doesn't work, then you may have to go back to a previous mksysb if you have one. I believe that is what Support will come up with as well.

One additional step that they won't say, but that you should definitely do, is walk over to the developer's desk, slap him silly, and take away his root privileges!

Good luck!
--Blake R.
#AIX-Forum

Originally posted by: SystemAdmin

Hi Steve,

Actually, I was trying to move the first disk from AIX 5.3 box to another AIX 6.1 server to see if I can mount from there to move the file back which didn't complete.

So I really don't know where I can continue?

Thanks,

David H.
#AIX-Forum

Originally posted by: SystemAdmin

It's probably time to get out the backup media and reinstall. All of those warnings from synclvodm about overwriting the LV control blocks imply that you're screwed.

Next time, don't let the children play in the road.
#AIX-Forum

Originally posted by: shargus

I think the key is this:

0516-1791 redefinevg: lvm_rec for PV hdisk1 has VGID 0000000000000000 (0004fc690000d6000000011542caea66 expected)
PV Status: hdisk0 0004fc695ac772af PVACTIVE
0004fc6942caea0b NONAME

Looks like someone may have scrubbed hdisk1?
#AIX-Forum

Originally posted by: SystemAdmin

Will be possible because I move the first disk which should be hdisk0 to AIX 6.1 and do

extendvg hdiskx to rootvg
reducevg hdiskx to rootvg

which cause the AIX 5.3 can't recognize hdisk1?

Any idea how to fix it? Please advise.

Thanks,

David H.
#AIX-Forum

Originally posted by: SystemAdmin

Your extendvg and reducevg while running 6.1 have completed eliminated your chances of recovering rootvg on the 5.3 disk. Your only choice is to reinstall.
#AIX-Forum

Originally posted by: Kosala

I once tried this my own, just to learn that all the LVs will be imported under /dev as a device, and there cannot be two LVs with the same name. And importvg renamed all my LVs. This means I was pretty much screwed, luckily I had a mksysb, I hope you too :).

AFAIK, regular utilities are not linked against libcrypt. Are you using TCB or something? Generally sshd and ssh depends on libcrypt.

Kosala
#AIX-Forum

Originally posted by: JimPenny

I am also facing similar issue and do not know how to resolve it.
#AIX-Forum