Originally posted by: SystemAdmin

Hello All!

I am new to AIX (a storage area network guy by trade) and have been asked to move a number of AIX 5.3 servers on to a new storage platform (Data Direct 9550s w/80TB of space) and it was suggested that I use rsync to accomplish this task. I have trolled around the groups looking for some assistance in the posts and have pieced together the following steps below (without any real detail...that will come later) as a guideline to follow. The servers (7 in all) each have about 3TB of current SAN storage and my experience has been that rsync will probably take days to actually copy all of that data between file systems and the final sync could take hours (the systems are very busy with large image files) before we cut over to the new storage. I would like to follow the steps below and mirror the original logical volume to the new logical volume I will be creating, but had a few questions on the sync'ing of logical volumes as it pertains to striped volumes (this is how the new storage will be laid out):

1. Add new storage to existing server from the storage array and run cfgmgr to pull in the new storage
2. Create NEW volume group to hold the new hdisks
3. Create NEW striped logical volume with the new hdisks
4. Mirror the NEW striped logical volume to the existing logical volume which is NOT striped
5. Use syncvg to sync all the data over to the NEW logical volume
6. Break the mirror when it is in-sync and then mount up the new storage/logical volume in place of the old storage/logical volume

My questions are as follows (again, I am very new to the LVM way of doing things so forgive me if these questions are a bit routine):

1. Do I have to put the new logical volume (and therefore the new hdisks) in the original volume group or can I create a new volume group and still be able to sync the logical volumes with the syncvg command?

2. Can I mirror a striped logical volume to an existing NON-striped logical volume as long as my striped logical volume is the same size or larger than the original volume I want to copy?

3. I am pretty clear on using the syncvg command to get the sync'ing of the data going between the two logical volumes, but how can I tell when everything is actually "in-sync" and what LVM command can I use to break the mirror when it is in-sync so I can switch the logical volume that is being mounted up?

Again, any help is GREATLY appreciated as our current AIX guy has moved on to greener pastures and this was just dumped in my lap since we have no AIX admins at this point

Cheers,
Travis

Message was edited by: JackTheRipper

Originally posted by: unixgrl

Is there some reason you are wanting to put the new storage into new disk groups?

You can't use syncvg to copy data from one logical volume to another.

Easiest way I know of migrating data to new disks on same system:
1) Make new storage visible to system
2) Add new disks to existing disk groups using "extendvg"
3) Use "mirrorvg" to create another copy of the disk group on the new disk. You could also use "migratepv" if you don't want to keep the original copy intact
4) Use "unmirrorvg" & "reducevg" to remove the old disk out of the disk group.

This is an online change and you don't need to make any new disk groups or volumes. Be careful about having latest patches because there have been recent bugs with doing this type of work on busy systems.

Originally posted by: apple08

dear unixgrl and aix guru,
"Be careful about having latest patches because there have been recent bugs with doing this type of work on busy systems."
what do u mean by having latest patches? r u refering to aix technology level? if we perform during the system is not busy, will it mitigate the issue? how to check whether we have latest patches on our system.

hope to hear from you. thank you

Originally posted by: SystemAdmin

unixgrl:

Awesome and thank you! I think that this makes the most sense and appears to be a much more cut-and-dry approach to getting the data moved over. Thank you so much and I will now go ahead and put together my plan based on these steps...thank you! We will, as luck would have it, be able to take a system that is not being used to heavily to do first so we can work out any kinks...I will keep all posted and I will also provide my steps to the semi-colon once complete Thank you again all!!!

Cheers,
Travis

Originally posted by: SystemAdmin

unixgirl:

A coworker just suggested we take a look at cplv as a solution to this issue. This would allow us to create a new vg (to establish our new naming standard) and create the striped volumes we are after (for better performance) but my question after reading the man page on cplv is the following:

1. Is cplv incremental? In other words does anyone know if each time you run cplv is it just updating what has changed between the volumes are is it a complete copy (it looks like it might be a complete overwrite, but I just want to be sure)?

...and now that made me think of something else...

2. If I use cplv from a NON striped volume to copy to a striped volume (my goal) will it maintain my stripe layout, etc. after the copy...again, the man page is a bit fuzzy here and only talks about copying FROM a striped volume to a NON-striped new volume...

Again, any help is greatly appreciated!!!

Cheers,
Travis

Originally posted by: tony.evans

#1

cplv either creates a new logical volume or destroys the contents in the existing one when used to copy to an existing lv.

i.e. it's not incremental, and if it gets stopped during the process, your target lv is corrupt.

cplv doesn't understand filesystems, it's copying logical volume data below the filesystem structure level.

#2

No clue, I suggest testing on a development server.

Originally posted by: alethad

Go check the man page for cplv and see if it doesn't tell you what you need to know. If you're re-creating the LV's from scratch it may serve you to use it.

Otherwise if you're not re-creating the LV's look at the man page for migratepv which you can issue this command with users on the system. I've used this many times to save downtime to users.

You may need to compare the 2 commands to see which would serve you best.
Hope that helps.

Originally posted by: esv

striping is a good idea but you will be forced to do increments of N-disks at a time.

another idea is to leave some free space on every single disk, change your LVs to use max extents (I think it is option "-e x" for the chlv command) then run reorgvg , it will cause high I/O on your server for a couple of hours (depending on the size of the VG and your server's CPU and memory) but AIX will reorganize every LV spreading across all physical volumes on the VG.

best regards,
esv.

Originally posted by: SystemAdmin

Hello All!

As promised I am back to update you on the steps/procedures which we ended up using to meet our requirements. We initially pulled in 8 x 500GB Data Direct 9550 LUNs on to an IBM 9117-570 (running 5.3) system with 4 Emulex LP10000 fiber channel cards and an IBM 7039-651 (running 5.2) with 1 Emulex 9802 fiber channel card. If you remember, we needed to create stripe groups with a width of 4 (knowing that in the future we will have to add drives in groups of 4...we are fine with that). So, after I ran cfgmgr we went to pull in the first 4 disks into our new-and-improved volume group with the following command (oh, and a quick note - I tested this on both AIX 5.2 and 5.3 - we were trying to achieve the same results with both 5.2 and 5.3, with the exception of the stripe size (128k versus 2M due to the limitation in AIX 5.2):

1. mkvg -S -y 9550_vg00 hdisk7 hdisk8 hdisk9 hdisk10 (AIX 5.3)
2. mkvg -B -y 9550_vg00 hdisk7 hdisk8 hdisk9 hdisk10 (AIX 5.2)

This created our volume group and pulled in the initial 4 drives (remember, we needed to test our ability to successfully extend this guy out so that is why we didn't add in all 8 drives :-) ). Next, we created the logical volume from the 4 hdisks we added above:

1. mklv -C 4 -S2M -t jfs2 -x 32512 -y 9550_lv00 9550_vg00 7448 (AIX 5.3)
2. mklv -u4 -S128k -t jfs2 -x 32512 -y 9550_lv00 9550_vg00 7448 (AIX 5.2)

You will notice a distinct difference between the 5.2 and 5.2 commands here due to some of the limitations of AIX 5.2 (or at least the limitations the man page said were in effect...specifically the stripe size of 128k versus 2M which the 5.2 man pages say is the limit (128k that is...not the 2M). In fact, if you put in 256k or 512k you will get an error that says "0516-1332 mklv: Stripe size of a logical volume must be less than or equal to the LTG size of the volume group". You might be able to change the LTG I guess under 5.2 but I do not know how...sorry. Also, the 7448 at the end of the command there was found by running "lsvg 9550_vg00" in the FREE PPs: column and since we want to use all the space and use it all now, we put in the full amount shown of 7448.

Okay, so our volume group was created and our logical volume was set up as a stripe and now I created the file system on the logical volume with the following command:

1. crfs -v jfs2 -d /dev/9550_lv00 -m /WESTTEST -A y -p rw -a logname=INLINE (AIX 5.3)
2. crfs -v jfs2 -d /dev/9550_lv00 -m /WESTTEST -A y -p rw -a logname=INLINE (AIX 5.2)

Now, this took about 30 seconds to run (all the previous commands were pretty much instantaneous) and will echo out that the filesystem has been created and the size in kilobytes and then gives you another size but with no annotation as to the metric being displayed...it looks like 512k blocks but I didn't want to waste time doing the math :-) I now mounted up the new filesystem to the mount point listed after the -m option:

1. mount /WESTTEST

A df -g showed that this worked and I had my just-under-two-terrabyte filesystem mounted up and ready for use. Now it was time to practice/test the extension of the filesystem (which is critical with all the large satellite files we produce/keep and space gets used up pretty fast). I used the following series of commands to grow the filesystem on the fly:

1. extendvg 9550_vg00 hdisk11 hdisk12 hdisk13 hdisk14
2. chlv -u 8 9550_lv00
3. time extendlv 9550_lv00 7448 (command ran in 6 seconds)
4. time chfs -a size=+1906688M /TEMPIASI (command ran in 24 seconds)

The chlv command had to be run BEFORE I could extend out the logical volume because of an error I received the very first time I tried which said the following:

0516-1034 lquerypv: Not enough physical partitions in physical volume hdisk7.
0516-788 extendlv: Unable to extend logical volume.

I did some digging on Google (is there any better source :-) ) and found out that the problem was the fact that I had an "UPPER BOUND" of 4. I looked at the manpage for mkvg and mklv and didn't see anything in there (I could have missed it) on how to set that when the vg or lv are created - so I could set it to 32 or 64 or whatever. It looks like it just set it to the number of devices I added when I first created the vg - or it could be related, I guess, to the stripe width of 4...I still need to look into that. Anyway, in order to see what your UPPER BOUND is set to run the following command:

1. lslv 9550_lv00

So, the answer is to increase the UPPER BOUND so that it will allow me to extend the logical volume. I just extended it out to what I needed (in my case 8), but I did test setting it higher and was able to go all the way up to 1024 and when I tried to set it to 2048 I got the following error (which makes pretty obvious the limit to which you can set this parameter :-))):

0516-1203 chlv: The -u parameter for upperbound must be between 1 and 1024

I don't know if there is an adverse affect to setting it to 1024 if I only need 8...that, unfortunately, will require some more testing :-) I then was able to extend the logical volume and then I had to run the chfs command which I will explain:

1. time chfs -a size=+1906688M /WESTTEST (command ran in 24 seconds)

We timed it because we were curious how long it would take to add in ~2TB of storage to an existing filesystem which was mounted up and being used...24 seconds was not bad :-) The size= argument deserves some explanation and we got that number, 1906688M, by running the same command as before when we ran "lsvg 9550_vg00" and then looked at the FREE PPs column and grabbed all the free PPs we could :-) so it was nice not to have to break out a calculator. It was nice that we did this with the filesystem mounted, I/O running to it, and didn't have to unmount it or anything like that...all on the fly!

With the filesystem set up and mounted it was now time to migrate over the data (the fun part I guess :-) ). I decided to go with CPIO and actually found that as a recommended method on the IBM website of all places! Here is the link:
Link: http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/index.jsp?topic=/com.ibm.storage.ssic.help.doc/f2c_usinglvmdirectcopy_1cug94.html

Now, I immediately noticed that CPIO does not support files larger than 2GB and we actually tested that fact and found out that yes, it will throw an error message saying that it cannot read the file and continue on, but we didn't have too many files over 2GB in size and we manually moved them over with a script using the cp command (which we could not find any information that it could NOT handle files 2GB in size or larger and it certainly moved the ones over we needed moved and the data looks good...if I am wrong here, please let me know! :-) Okay, here are the commands I ran to find the files over 2GB in size and then the command (as seen on the IBM web site) to cpio over all the files UNDER 2GB in size :-)

1. cd /WEST
2. find . -size +2000000k -print
3. find . -size +2000000k -print | wc -l (to get a count to compare to later)
4. time find . -print | cpio -pdmuv /WESTTEST (this moved everything UNDER 2GB to the new filesystem)

Well...there you have it! I am not saying it is the most efficient way to get it done, but we are done...and if you are wondering, to move 2.1TB of data mounted under /WEST and stored on a Data Direct S2A8000-series array to the /WESTTEST mount point which had storage from a Data Direct S2A9550-series storage array going through Brocade 24000s took 11.7 hours! Not bad I guess and now it is on to other things...

...and I would like to again thank everyone who took time to respond to my post...it was greatly appreciated and got me going in the right direction. And if you are wondering why I would take so much time to post all of this up, it is in the hopes that some day, someone out there is able to use something in my post to help them out just like I have found so much help in so many posts here...so, your feedback is welcome (especially if something looks completely insane :-) above) and I hope this helps you one day...

Cheers,
Travis

Originally posted by: esv

first of all, I got to offer my sympathies and the best of luck.

I would not use rsync since that would imply that you have keep the copy updated as the data changes, then stop the service, umount all filesystems and modify the filesystem mount points.

I would use the following procedure:

Add the new storage LUNs to the server,
Add the luns to the volume group
Migrate each of the hdisks/vpaths to the new devices using migratepv.

it is a daunting task but I have successully migrated over 10 TB of data using that procedure, you need to use the similar size LUNs as the Volume Group can only create a specific number of physical partitions per drive, or change the volume group attributes.

good luck,
esv.

Originally posted by: SystemAdmin

Enrique:

Thank you for your feedback and I will be following the first two steps you outline and running with it! The reason for creating a new volume group is that I wanted to have a 4-column stripe instead of what we have now which is a system with a few very busy disks and some disks doing nothing at all. Thank you again for your help!

Cheers,
Travis

Originally posted by: unixgrl

re: being on latest patch levels.

First, "oslevel -s" will tell you which Operating system technology level and service pack you are on.

Example: 5300-06-04-0748 is AIX 5.3, TL6, SP4

There have been multiple problems reported with SAN attached disk on service packs that came out between last October and May 1. I've had a recent system crash that happens with mirrored LVM volumes and doesn't have a fix yet.

The latest service pack information can be found on IBM FixCentral at http://www-933.ibm.com/eserver/support/fixes/

For 5.3 TL6SP7, TL7SP4, or TL8SP2 are all considered to be the latest.

So, just wanted to warn you that if your system crashes during your disk migrations, you'll probably need to apply a service pack.

Originally posted by: esv

We're at 5.3 TL6 SP4, have a couple of boxes running 5.2 TL10 and 3 or 5 running at 5.1.

We migrate data left and right.... no problems so far...

best regards,
esv.

Originally posted by: SystemAdmin

Thanks for your help!