Gearman 0.41 with MySQL Persistent Storage and PHP PECL 0.8.3 with GearmanManager on Ubuntu 12.04

The Problem

If you wish to use Brian Moon’s GearmanManager daemon to manage your gearman worker scripts, the most recent PHP PECL gearman release (v1.1.0) will not work due to a known bug (which caused me many hours of pain).

TL;DR (The Short Version)

To use GearmanManager we need to use the previous version of PECL Gearman (currently v0.8.3) and thus the previous version of libgearman (Gearmand currently v0.41). Using Gearmand v0.41 we have native MySQL support with libmysqlclient.

Installation instructions are very much as per my how to install gearman with mysql persistent storage on ubuntu post, plus installation of PHP PECL Gearman and GearmanManager.

From a shell prompt install Gearman 0.41:

apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev -y
wget https://launchpad.net/gearmand/1.0/0.41/+download/gearmand-0.41.tar.gz
tar xzf gearmand-0.41.tar.gz
cd gearmand-0.41
./configure
make
make install
ldconfig

Create the upstart process:

nano /etc/init/gearmand.conf

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Start the daemon:

service gearmand start

Install PHP CLI, along with any extensions you might need, and PECL Gearman 0.8.3:

apt-get install php5-cli php5-dev php-pear -y
pecl install gearman-0.8.3

Enable PECL Gearman extension in PHP:

nano /etc/php5/cli/conf.d/gearman.ini

extension=gearman.so

Install Gearman Manager:

apt-get install git-core
git clone https://github.com/brianlmoon/GearmanManager.git /var/gearman/GearmanManager
/var/gearman/GearmanManager/install/install.sh

Configure Gearman Manager:

nano /etc/gearman-manager/config.ini

Set your gearman job servers e.g. host=172.29.17.56,172.29.17.57

Reset the GearmanManager process:

/etc/init.d/gearman-manager restart

The Long Version

The PHP PECL Gearman 1.x branch was written to work with with later gearmand versions (libgearman v0.21 and above), and which appears to have some fairly fundamental changes. As such this entire branch seems to be unstable and unusable with GearmanManager.

The previous beta version however does work, and looks to be updated in parallel with the 1.x version (great for us!), with the latest edition being PECL-Gearman 0.8.3.

PECL-Gearman 1.1.0 works with the most recent Gearmand release (1.1.3), which uses the updated libgearman, and is incompatible with the PECL-Gearman 0.8.3 release we want to use, so, we need to use the previous version of gearman which does work – Gearmand v0.41.

Once-upon-a-time if we wanted a persistent queue in MySQL we’d have to go through the pain of trying to get libdrizzle installed, however as of Gearman v0.38 libmysqlclient is supported, meaning we have native MySQL support. Hurrah!

The key point here to install Gearmand v0.41 with PHP PECL-Gearman v0.8.3.

Installation instructions for Gearman are very much as per my how to install gearman with mysql persistent storage on ubuntu post, plus installation of the PHP PECL Gearman extension and GearmanManager itself.

Run the following from a shell prompt to install Gearman 0.41 with MySQL:

apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev
wget https://launchpad.net/gearmand/1.0/0.41/+download/gearmand-0.41.tar.gz
tar xzf gearmand-0.41.tar.gz
cd gearmand-0.41
./configure
make
make install
ldconfig

Now create the upstart process:

nano /etc/init/gearmand.conf

And paste the following:

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Start the job server daemon:

service gearmand start

Now install PHP, along with any extensions you might need, and the php-pear extension so that we can install PECL-Gearman afterwards:

apt-get install php5-cli php5-dev php-pear

Install the PECL Gearman 0.8.3 extension:

pecl install gearman-0.8.3

Enable PECL Gearman extension in PHP:

nano /etc/php5/cli/conf.d/gearman.ini

And paste the following:

extension=gearman.so

Now install Gearman Manager:

apt-get install git-core
git clone https://github.com/brianlmoon/GearmanManager.git /var/gearman/GearmanManager
/var/gearman/GearmanManager/install/install.sh

Make sure you select the PECL installation and then we can configure Gearman Manager:

nano /etc/gearman-manager/config.ini

Set your gearman job servers e.g. host=172.29.17.56,172.29.17.57 and restart the GearmanManager process:

/etc/init.d/gearman-manager restart

You can see a full list of the GearmanManager config options on the GearmanManager github project page.

Compiling Gearman 1.1.3 with MySQL Persistent Storage on Ubuntu 12.04 LTS Precise

If you wish to use Brian Moon’s GearmanManager daemon to manage your gearman worker scripts, the most recent PHP PECL gearman release (v1.1.0) will not work due to a known bug (which caused me many hours of pain).

Your best bet is to follow my How to install Gearman with MySQL Persistent Storage and PHP PECL Gearman 0.8.3 with Brian Moon’s GearmanManager on Ubuntu 12.04 post.

I had some problems getting Gearman working on Ubuntu and figured others might have the same issues.

I’d previously written instructions for getting gearman working on 10.04 with standard ubuntu packages and with libdrizzle for persistent mysql storage of the jobs, but this didn’t work when attempting to use the latest version of gearman from launchpad and ubuntu 12.04 precise.

I opted to compile gearman from source on a clean installation of Ubuntu 12.04 Precise and after a bit of trouble getting libdrizzle to work found that gearman will compile with libmysql support for direct MySQL integration. This saved a lot of hassle, and here’s how i did it -

TL;DR (The Short Version)

Instructions for compiling and installing Gearman 1.1.3 with MySQL persistent storage on Ubuntu 12.04 LTS Precise:

apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev -y
wget https://launchpad.net/gearmand/1.2/1.1.3/+download/gearmand-1.1.3.tar.gz
tar xzf gearmand-1.1.3.tar.gz
cd gearmand-1.1.3
./configure
make
make install
ldconfig

Create the upstart process:

nano /etc/init/gearmand.conf

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Start the daemon:

service gearmand start

Good to go!

The Long Version

First things first, let’s download the gearman source files and configure, so in an ssh window -

root@gearman-01:/# wget https://launchpad.net/gearmand/1.2/1.1.3/+download/gearmand-1.1.3.tar.gz
...
2012-12-11 15:56:53 (2.25 MB/s) - `gearmand-1.1.3.tar.gz' saved [857122/857122]

root@gearman-01:/# tar xzf gearmand-1.1.3.tar.gz
root@gearman-01:/# cd gearmand-1.1.3
root@gearman-01:/gearmand-1.1.3# ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/gearmand-1.1.3':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

So… we have a problem. We need to get a compiler.

Luckily Ubuntu have a lovely package called build-essential which gets us the required compilers we need. This also saved on lots of hassle trying to install them individually.

root@gearman-01:/gearmand-1.1.3# apt-get install build-essential
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  binutils cpp cpp-4.6 dpkg-dev fakeroot g++ g++-4.6 gcc gcc-4.6 libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libc-bin libc-dev-bin libc6 libc6-dev libdpkg-perl libgomp1 libmpc2 libmpfr4 libquadmath0
  libstdc++6-4.6-dev linux-libc-dev make manpages-dev
...
2 upgraded, 24 newly installed, 0 to remove and 79 not upgraded.
Need to get 36.4 MB of archives.
After this operation, 83.9 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
...

So let’s try and configure again:

root@gearman-01:/gearmand-1.1.3# ./configure
...
checking for Boost headers version >= 1.39.0... no
configure: error: cannot find Boost headers version >= 1.39.0

OK so we need to get some updated boost headers. No problem:

root@gearman-01:/gearmand-1.1.3# apt-get install libboost-thread-dev libboost-program-options-dev
The following extra packages will be installed:
  libboost-date-time1.46-dev libboost-date-time1.46.1 libboost-program-options1.46-dev libboost-program-options1.46.1
  libboost-serialization1.46-dev libboost-serialization1.46.1 libboost-thread1.46-dev libboost-thread1.46.1 libboost1.46-dev
...
0 upgraded, 11 newly installed, 0 to remove and 79 not upgraded.
Need to get 358 kB/8,619 kB of archives.
After this operation, 63.6 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
...

root@gearman-01:/gearmand-1.1.3# ./configure
...
checking event.h usability... no
checking event.h presence... no
checking for event.h... no
configure: error: Unable to find libevent

Let’s get libevent then:

root@gearman-01:/gearmand-1.1.3# apt-get install libevent-dev
The following extra packages will be installed:
  libevent-core-2.0-5 libevent-extra-2.0-5 libevent-openssl-2.0-5 libevent-pthreads-2.0-5
...
0 upgraded, 5 newly installed, 0 to remove and 79 not upgraded.
Need to get 426 kB of archives.
After this operation, 1,832 kB of additional disk space will be used.
Do you want to continue [Y/n]? y

root@gearman-01:/gearmand-1.1.3# ./configure
...
Configuration summary for gearmand version 1.1.3

   * Installation prefix:       /usr/local
   * System type:               unknown-linux-gnu
   * Host CPU:                  x86_64
   * C Compiler:                gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
   * C Flags:                   -g -O2  -Wno-pragmas -Wall -Wextra -Wunknown-pragmas -std=c99 --param=ssp-buffer-size=1 -Wno-attri                butes -Waddress -Warray-bounds -Wbad-function-cast -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat-security -Wformat=2 -Wlogica                l-op -Wmissing-field-initializers -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wnormalized=id -Woverride-init -Wpoint                er-arith -Wpointer-sign -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-overflow=1 -Wswitch-enum -Wundef -Wunused -Wunused-resu                lt -Wunused-variable -Wwrite-strings -floop-parallelize-all -fwrapv -pipe
   * C++ Compiler:              g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
   * C++ Flags:                 -g -O2 -Wno-pragmas -Wall -Wextra -Wunknown-pragmas --param=ssp-buffer-size=1 -Wno-attributes -Wad                dress -Warray-bounds -Wchar-subscripts -Wcomment -Wctor-dtor-privacy -Wfloat-equal -Wformat=2 -Wmissing-field-initializers -Wlogic                al-op -Wnon-virtual-dtor -Wnormalized=id -Woverloaded-virtual -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-o                verflow=1 -Wswitch-enum -Wundef -Wunused -Wunused-result -Wunused-variable -Wwrite-strings -Wformat-security -floop-parallelize-al                l -fwrapv
   * CPP Flags:                  -fvisibility=hidden
   * LIB Flags:                 -pie
   * Assertions enabled:        no
   * Debug enabled:             no
   * Warnings as failure:       no
   * Building with libsqlite3   no
   * Building with libdrizzle   no
   * Building with libmemcached not found
   * Building with libpq        no
   * Building with tokyocabinet no
   * Building with libmysql     no
   * make -j:                   2
   * VCS checkout:              no

Great, so gearman is configured, but we need to get libmysql:

root@gearman-01:/gearmand-1.1.3# apt-get install libmysqld-dev
...
The following extra packages will be installed:
  libmysqlclient-dev libmysqlclient18 mysql-common zlib1g-dev

root@gearman-01:/gearmand-1.1.3# ./configure
...
Configuration summary for gearmand version 1.1.3
   * Building with libmysql     yes

And now proceed with compiling:

root@gearman-01:/gearmand-1.1.3# make
make -j2  all-am
make[1]: Entering directory `/gearmand-1.1.3'
  CXX    libgearman/libgearman_libgearman_la-gearman.lo
  CXX    libgearman/libgearman_libgearman_la-job.lo
  CXX    libgearman/libgearman_libgearman_la-kill.lo
libgearman/job.cc: In function 'gearman_return_t _job_send(gearman_job_st*)':
libgearman/job.cc:606:25: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed
libgearman/job.cc: In function 'gearman_return_t gearman_job_send_complete_fin(gearman_job_st*, const void*, size_t)':
libgearman/job.cc:355:18: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed
libgearman/job.cc: In function 'bool gearman_job_build_reducer(gearman_job_st*, gearman_return_t (*)(gearman_aggregator_st*, gearman_task_st*, gearman_result_st*))':
libgearman/job.cc:209:6: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed
make[1]: *** [libgearman/libgearman_libgearman_la-job.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/gearmand-1.1.3'
make: *** [all] Error 2

Looks like we need another package called libcloog-ppl0 to get this working:

root@gearman-01:/gearmand-1.1.3# apt-get install libcloog-ppl0
...
The following extra packages will be installed:
  libgmpxx4ldbl libppl-c4 libppl9 libpwl5

root@gearman-01:/gearmand-1.1.3# make

I didn’t get any errors so all good to proceed installing the program:

root@gearman-01:/gearmand-1.1.3# make install
...
Libraries have been installed in:
   /usr/local/lib
...
libtool: install: /usr/bin/install -c bin/.libs/gearman /usr/local/bin/gearman
libtool: install: /usr/bin/install -c bin/gearadmin /usr/local/bin/gearadmin
make[2]: Leaving directory `/gearmand-1.1.3'
make[1]: Leaving directory `/gearmand-1.1.3'

All done! Gearman is now compiled with libmysql for MySQL support and installed on the server. Hurrah!

You’ll find the installation at /usr/local/sbin/gearmand and you can run gearmand to start a server.

To use MySQL persistent storage with gearman you will want to pass some flags in the command e.g.

root@gearman-01:/# gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Set the MySQL server host, the port (default 3306), MySQL user (in my case named gearman), the user password, the database name for the gearman queues (i called mine gearman) and a table name to use for this server (i called mine gearman_01).

This will run the gearman program and store any jobs it received in the gearman_01 table. You will notice if you run the command above the gearman_01 table is created in the gearman database.

A better idea might be to put this code into an upstart job to create a daemon, so that gearman is started along with the server:

root@gearman-01:/# nano /etc/init/gearmand.conf

And paste in the following, substituting the flags as per above:

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

This will start the gearman daemon whenever the server boots up and try to ensure it is always running.

To start the service manually just type:

root@gearman-01:/# service gearmand start
gearmand start/running, process 12803

To test the gearman job server just run the following command:

root@gearman-01:/# gearman -f testq -b payload
gearman: error while loading shared libraries: libgearman.so.8: cannot open shared object file: No such file or directory

If you get the error above as i did you’ll need to run ldconfig to clean a few things up:

root@gearman-01:/# ldconfig
root@gearman-01:/# gearman -f testq -b payload

Great, the job seems to have been entered into gearman.

If you look at the gearman_01 table in the gearman database you’ll see there is a row with “testq” function_name and a data blob with “payload” as per the job we just added.

It does seem odd however there there is no unique_key. Let’s try to insert another job and see what happens:

root@gearman-01:/# gearman -f testq -b payload
gearman: gearman_client_run_tasks : _client_run_tasks(GEARMAN_SERVER_ERROR) queue_insert_error:QUEUE_ERROR -> libgearman/client.cc:1504

So, clearly an error there!

We need to install the uuid-dev package to generate unique IDs.

root@gearman-01:/# apt-get install uuid-dev
The following NEW packages will be installed
  uuid-dev

What a PITA. We’ll need to re-compile, then restart the gearmand daemon.

root@gearman-01:/# cd /gearmand-1.1.3
root@gearman-01:/gearmand-1.1.3# ./configure
root@gearman-01:/gearmand-1.1.3# make && make install && ldconfig
root@gearman-01:/gearmand-1.1.3# service gearmand restart

And we can now run the test command as much as we like, which will create lots of entries we can see the gearman table, so all good to go.

Job jobbed!

Updated Citrix XenServer 5.5 Automatic VM Backup Scripts

I have re-written the popular Citrix XenServer 5.5 Automated Live VM Backup scripts from scratch, with some big improvements:

  • Separate backup configuration script – So you don’t need to touch the key execution and function scripts to configure backups.
  • VDI removal – Run in addition to the standard vm-uninstall command to stop snapshotted VMs allocating all your disk space
  • Logging – Enable/disable functionality with more advanced function logging and file path customisation.
  • Backup VM selection – The facility to select between all, running, none and specifically set (by VM uuid) virtual machines.
  • Quiesce snapshots – To take advantage of the quiesce snapshot functionality if the VM supports it.
  • HA disabled – High availability is disabled on exported virtual machines to stop configuration issues when re-importing.

This is a complete re-write from scratch and as such is relatively untested other than on my systems – please provide feedback regarding any successful or problematic installations.

The latest release of the updated Citrix XenServer Backup Scripts can be downloaded from http://www.andy-burton.co.uk/files/xenserver_backup/xenserver_backup.tar.gz as a tar gzip archive.

Installation instructions:


[root@xs /]# mkdir /home/backup
[root@xs /]# cd /home/backup
[root@xs backup]# wget http://www.andy-burton.co.uk/files/xenserver_backup/xenserver_backup.tar.gz
...
23:45:25 (12.9 MB/s) - `xenserver_backup.tar.gz' saved [3181/3181]
[root@xs backup]# tar -xzf xenserver_backup_3.0.tar.gz
[root@xs backup]# chmod 700 vm_backup.*
[root@xs backup]# ./vm_backup.sh
[root@xs backup]# nano vm_backup.log

Open and edit vm_backup.cfg to configure the backup.

Run vm_backup.sh to run the backup process, this script does not need to be modified.

The virtual machine backup functions are located in vm_backup.lib and do not need to be modified.

As usual, all comments, negative, neutral or positive are all very welcome.

For those that require it, I will write additional information and a more detailed installation procedure in due course.

Update Log:

Citrix XenServer 5.5 Automated Live VM Backup to Windows CIFS Share

Up until now reliably backing up virtual machines from any of the free virtualisation hypervisors has been either expensive or impossible.

With the launch of Citrix XenServer 5.5 you can now not only take VM snapshots of live servers without shutting them down and taking them offline, you can also export these snapshots including the hard drive disk data to a template file. Continue reading