Monday, June 6, 2011

VMware's closed source VDDK

Well, in an effort to get a backup solution in place at my current place of employment, I needed a way to backup our growing list of virtual machines.  I needed a way to, not only backup files (for that I use backuppc), but to backup at the image level.  We basically needed a disaster recovery scenario. 

So, after trying out a few off the shelf implementations, we decided to write our own.  After all, how hard could it be....

Our current virtual hypervisors are 6 VMware ESXi servers and a vCenter server to manage them.  Downloading the SDK for the vSphere API and developing to those API's is for another story, and I'll focus on the part that took the longest, and most frustrating; vixDiskLib (VDDK).


Ubuntu is our Linux distribution of choice here, and I wanted to stay on one distro as much as possible.  Downloading and installing the libraries was fairly strait forward.  However, because it uses out of date libraries and is closed source, you have to do a little tweaking (see Re: How to solve "Fail to load libvixDiskLibVim.so" and Repository site).

The fun part began when I actually tried to use it.  There are assumptions built into the library that only work for older distributions with kernel versions less then 2.6.18.  It expects certain files to be in certain places, and doesn't do much as far as checks and recovery.  This is the problem with closed source, you can't change it to work for you, and you can't get any help from a company that only deals in dollars.  Basically, after asking for help (which would cost $5000 just to email someone), and begging on the forums (see Re: You do not have access rights to this file, Re: VDDK and SAN on Ubuntu, Re: No path to device LVID) I had to take matters into my own hands, and fake out the library.

This setup assumes that you have multipath running, and your devices are in /dev/mapper.  Pick a place on your filesystem where you would like this scratch area.  In my case, I chose /vmware.


First, we needed to change where vixDiskLib looks for these files.  This is a one time change, and shouldn't be repeated in a setup script, for example.  Execute the following in a terminal (change "64" to "32" if on a 32 bit machine):

# cd  /usr/lib/vmware-vix-disklib/plugins64/
# mv libdiskLibPlugin.so libdiskLibPlugin.so.org
# sed -e 's@%s/class/scsi_disk@/////vmware/mapper@' libdiskLibPlugin.so.org libdiskLibPlugin.so

The commands above assume your on a 64 bit machine and you're using /vmware as the location for the "scratch area".

Next, you can run the following script, which will make symbolic links to the actual location of files in the system.  I put this in /etc/init.d and made sure it ran on boot.  However, an upstart script would be much better, making sure that multipath has come up and had time to do it's thing.

#!/bin/bash
shopt -s extglob

for f in /dev/mapper/mpath+([0-9]) ; do
    d=$(basename $f)
    mkdir -p /vmware/mapper/$d/device/$d
    ln -sf /vmware/mapper/$d/device/$d /vmware/mapper/$d/device/block:$d
    ln -sf /dev/mapper/$d /dev/$d
done


This script assumes that your mapped drives are named mpathX (where X is a number).  That is the names of the devices defaulted to when I used:

defaults {
        user_friendly_names yes
}


In the /etc/multipath.conf file.

I hope this helps with anyone out there struggling with this as well.  Please let me know if this is a Bad Idea(tm), or you have improvements that I could use.

Thursday, April 28, 2011

Upgrading puppet 0.25.4 to puppet 2.6.6 in Ubuntu 10.04 LTS

If you are administering Ubuntu servers with puppet, getting the latest puppet to work can be a painful process.  So, this is what I ended up doing.  It may not be the best way, and probably still has some kinks to work out, but this will get you started.

I have a module called puppet.  In the /path/to/modules/puppet/manifests/init.pp I have the usual:

import "classes/*.pp"

in classes/puppet_client.pp, I have:

class puppet::client {
    include apt
    include monit::puppet

    package { "puppet":
        ensure => latest,
        notify => Service["puppet"],
    }

    package { "facter":
        ensure => latest,
        notify => Service["puppet"],
     }

    file { "/etc/puppet/auth.conf":
        source  => "puppet:///modules/puppet/auth.conf",
        owner   => "puppet",
        group   => "puppet",
        mode    => "640",
        notify  => Service["puppet"],
        require => [Package["puppet"], Package["facter"]],
    }

    file { "/etc/puppet/namespaceauth.conf":
        source  => "puppet:///modules/puppet/namespaceauth.conf",
        owner   => "puppet",
        group   => "puppet",
        mode    => "640",
        notify  => Service["puppet"],
        require => [Package["puppet"], Package["facter"]],
    }

    file { "/etc/puppet/puppet.conf":
        source  => "puppet:///modules/puppet/puppet.conf",
        owner   => "puppet",
        group   => "puppet",
        mode    => "640",
        notify  => Service["puppet"],
        require => [Package["puppet"], Package["facter"]],
    }

    file { "/etc/default/puppet":
        source  => "puppet:///modules/puppet/puppet",
        owner   => "puppet",
        group   => "puppet",
        mode    => "640",
        notify  => Service["puppet"],
        require => [Package["puppet"], Package["facter"]],
    }
    service { "puppet":
        ensure     => running,
        enable     => true,
        hasstatus  => true,
        hasrestart => true,
    }

    # puppet PPA
    case $operatingsystem {
        Ubuntu :  {
            apt::key { "AFF68B78":
                ensure => present,
            }
            apt::sources_list { "aroth-puppet.list":
                ensure  => present,
                content => "deb http://ppa.launchpad.net/aroth/ppa/ubuntu ${lsbdistcodename} main",
                notify  => Exec["apt-get_update"],
            }
        }
    }
}


The key part of this is installing Andreas Roth's PPA.  It's a great PPA and usually up to date.

So this is how it works.  Once the Ubuntu repo is added, it should trigger an apt-get update.  The next time puppet checks for updates, it should notice there is a higher version of puppet and facter, and upgrade those.  The upgrade should trigger puppet to be restarted, and it should be on the new version.

You should have the different configuration files in your /path/to/modules/puppet/files directory and edited to taste.

I also use three other modules:
Well, I hope this helps some people out there.  Sometimes I find that it can't update without some intervention, but it seems to work for the most part.  Let me know if you have any recommendations!

Tuesday, March 22, 2011

vSphere SDK for Python

I have created a git-hub project called pyvisdk.  It's an implementation of the vSphere API written in python.  My hope is that it will be useful to python developers and help fuel more development.

To complement this module, I created another project called vixDiskLib.  It's basically a python wrapper around the vixDiskLib C library from VMware.  I used Cython to create the bindings, and provide an easier and more pythonic interface to this library.

I hope everyone enjoys it, and feel free to post any issue you might have.  Keep in mind that it's still relatively new and I would be grateful for any feedback you could give.