How to migrate a Xen Virtual Machine on RHEL5

[hannes_kuehnemund]

How to migrate a Xen Virtual Machine on RHEL5

Xen is a Virtual Machine Manager on Linux and part of the Linux Kernel. It was developed at the Cambridge University. RHEL5 is Red Hat Enterprise Linux 5, released on March 15th 2007. This blog basically handles the offline migration of an existing virtual machine from a RHEL5 host to another RHEL5 host. When the virtual machine is installed in a logical volume on local disks online migration is not possible, thus offline migration is the only option. This already makes the migration tricky, because you have to transfer a logical volume from the source host to the target host. To make the migration process a little bit harder, I use slightly different architectures. The source machine uses AMD Opteron processors the target machine uses Intel Xeon EM64T processors instead.

9 steps to success

I suggest to follow these steps to successfully finish the migration

1) Setup the target dom0 with the same software stack the source system has

2) Create a lvm layout which is similar to the one on the source machine

3) Copy over the configuration files

4) Prepare the virtual machine for offline migration

5) Shutdown the virtual machine

6) Copy the logical volume from source to target

7) Verify successful transfer using md5sums of logical volumes

😎 Start virtual machine

9) Finalize offline migration

Step 1: Setup target

To avoid problems during the migration it is strongly recommended that the dom0 of the source system has exactly the same Kernel and Xen version like the target system. When using RHEL5, you can easily use the Red Hat update tool "yum" to sync all packages on both servers. It is also recommended that you also update the virtual machine itself using yum, to ensure that all three involved machines (VM, source, target) use the same software stack.

Step 2: LVM layout

Before we can copy the content of the logical volume from source to the target, we have to create a similar logical volume on the target machine. A lvm layout contains of three important parts: the physical volumes, the volume group and the logical volumes:

physical volumes: The amount of physical volumes doesn't matter on the target host as long as they provide enough free space. You should consider configuring SAN devices as physical volumes to be able to perform live migrations in the future. Nevertheless, using internal drives as physical volumes will of course also work

volume group: When creating the volume group it is very important, that the extend size is the same as on the target host. It may be possible to use different extent sizes, but I never tested them. Using the same extend size will definitely work

logical volume: The logical volume on the target machine should exactly look like the logical volume on the source machine. It is very important that both logical volumes have exactly the same size. If the source lv (logical volume) has exactly 30GB of size the target lv also needs have 30GB of size. Nevertheless, you may want to stripe or mirror the new logical volume. This is possible as long as the lv size stays consistent

Step 3: Configuration files

After the lvm setup is done, copy the configuration file (e.g. /etc/xen/vm1) from source to target machine and change the logical volume path in case you used different names for the volume group or logical volume on the target.

Step 4: Prepare VM

In case your network infrastructure uses a network with different subnets and if source and target machine are in different subnets, then you should consider to disable the startup of the network and all network related daemons in the virtual machine, for example by calling "chkconfig network off" and so on. This will speed up the boot process when the vm is booted on the target system which has a different network subnet.

The most important daemon that you should disable inside the vm is microcode_ctl. When migrating from AMD to Intel microcode_ctl will cause the machine to crash during startup, thus disable it with "chkconfig microcode_ctl off". Furthermore, always disable this daemon inside all your virtual machine just in case you want to perform live migrations in the future or just in case you forget to disable it before an offline migration.

Step 5: Shutdown the virtual machine

Now it is time to shutdown the virtual machine by using "xm shutdown vm1 && vm delete vm1". The delete command ensures that the vm is released from xend management (xend forgets that there is a vm called vm1).

Step 6: Transfer logical volume

Having a network connection between both machines, just execute the following command as user root on the target host:

# ( ssh source.fqdn.net dd if=/dev/mapper/vg00-lv_v0 bs=32M ) | dd of=/dev/mapper/virt-lv_v0 bs=32M

Please adept source.fqdn.net with your source machines hostname. The paths to the lv's and the block size (which is the extend size!)might also differ so please also adept them. After executing this command, and entering the ssh password the shell seems to hang. There is nothing to worry about because "dd" does not have a progress bar or any output. Just wait and after some time (depending on the throughput of your network) the logical volume is copied to its new location.

In case you have configured several logical volumes for your vm, just create and copy them in the same way. In case you have added a file as additional virtual device you can simply copy the file to the target machine using "scp"

Step 7: Verify transfer

To verify the successful transfer you have to enter the md5sum command on both lv's on both machines. The md5sum results should be equal. If not, something went wrong and you may have a network problem or the target lv is not setup correctly

on source: md5sum /dev/mapper/vg00-lv_v0 c36f10fd0ff59c3bcce088d7a7a6c410 /dev/mapper/vg00-lv_v0 on target: md5sum /dev/mapper/virt-lv_v0 c36f10fd0ff59c3bcce088d7a7a6c410 /dev/mapper/virt-lv_v0

The above example shows a successful finished transfer. In case you have copied a file, just use md5sum on the file, it will also print out a hash value which must be the same on both machines

Step 8: start virtual machine

Having everything in place, you can simply start your vm with

# xm create vm1

Check with "xm list" if the machine is running. Check for errors using "xm dmesg vm1".

Step 9: Finalize offline migration

In case you haven't shutdown any network daemons the vm is already usable but in case of different subnets you have to use virt-manager to connect to the console of the just migrated vm. Log on to the machine and change the network settings using "system-config-network". After changing these settings and maybe adapting /etc/hosts and /etc/resolv.conf by hand, restart the network using "/etc/init.d/network restart". Your machine should now have a working network. Don't forget to enable the network and the network daemons once the network is working with "chkconfig network on" and so on.

Thats basically all to perform a Xen offline migration on RHEL5. The two tricky parts are the transfer of the logical volumes and the deactivation of microcode_ctl on startup. If you have any questions or comments, just let me know.