So, after trying out a few off the shelf implementations, we decided to write our own. After all, how hard could it be....
Our current virtual hypervisors are 6 VMware ESXi servers and a vCenter server to manage them. Downloading the SDK for the vSphere API and developing to those API's is for another story, and I'll focus on the part that took the longest, and most frustrating; vixDiskLib (VDDK).
Ubuntu is our Linux distribution of choice here, and I wanted to stay on one distro as much as possible. Downloading and installing the libraries was fairly strait forward. However, because it uses out of date libraries and is closed source, you have to do a little tweaking (see Re: How to solve "Fail to load libvixDiskLibVim.so" and Repository site).
The fun part began when I actually tried to use it. There are assumptions built into the library that only work for older distributions with kernel versions less then 2.6.18. It expects certain files to be in certain places, and doesn't do much as far as checks and recovery. This is the problem with closed source, you can't change it to work for you, and you can't get any help from a company that only deals in dollars. Basically, after asking for help (which would cost $5000 just to email someone), and begging on the forums (see Re: You do not have access rights to this file, Re: VDDK and SAN on Ubuntu, Re: No path to device LVID) I had to take matters into my own hands, and fake out the library.
This setup assumes that you have multipath running, and your devices are in /dev/mapper. Pick a place on your filesystem where you would like this scratch area. In my case, I chose /vmware.
First, we needed to change where vixDiskLib looks for these files. This is a one time change, and shouldn't be repeated in a setup script, for example. Execute the following in a terminal (change "64" to "32" if on a 32 bit machine):
# cd /usr/lib/vmware-vix-disklib/plugins64/
# mv libdiskLibPlugin.so libdiskLibPlugin.so.org
# sed -e 's@%s/class/scsi_disk@/////vmware/mapper@' libdiskLibPlugin.so.org libdiskLibPlugin.so
The commands above assume your on a 64 bit machine and you're using /vmware as the location for the "scratch area".
Next, you can run the following script, which will make symbolic links to the actual location of files in the system. I put this in /etc/init.d and made sure it ran on boot. However, an upstart script would be much better, making sure that multipath has come up and had time to do it's thing.
#!/bin/bash
shopt -s extglobfor f in /dev/mapper/mpath+([0-9]) ; do
d=$(basename $f)
mkdir -p /vmware/mapper/$d/device/$d
ln -sf /vmware/mapper/$d/device/$d /vmware/mapper/$d/device/block:$d
ln -sf /dev/mapper/$d /dev/$d
done
This script assumes that your mapped drives are named mpathX (where X is a number). That is the names of the devices defaulted to when I used:
user_friendly_names yes
}
In the /etc/multipath.conf file.
I hope this helps with anyone out there struggling with this as well. Please let me know if this is a Bad Idea(tm), or you have improvements that I could use.
