If you wish to use Brian Moon’s GearmanManager daemon to manage your gearman worker scripts, the most recent PHP PECL gearman release (v1.1.0) will not work due to a known bug (which caused me many hours of pain).

Your best bet is to follow my How to install Gearman with MySQL Persistent Storage and PHP PECL Gearman 0.8.3 with Brian Moon’s GearmanManager on Ubuntu 12.04 post.

I had some problems getting Gearman working on Ubuntu and figured others might have the same issues.

I’d previously written instructions for getting gearman working on 10.04 with standard ubuntu packages and with libdrizzle for persistent mysql storage of the jobs, but this didn’t work when attempting to use the latest version of gearman from launchpad and ubuntu 12.04 precise.

I opted to compile gearman from source on a clean installation of Ubuntu 12.04 Precise and after a bit of trouble getting libdrizzle to work found that gearman will compile with libmysql support for direct MySQL integration. This saved a lot of hassle, and here’s how i did it -

TL;DR (The Short Version)

Instructions for compiling and installing Gearman 1.1.3 with MySQL persistent storage on Ubuntu 12.04 LTS Precise:

apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev -y
wget https://launchpad.net/gearmand/1.2/1.1.3/+download/gearmand-1.1.3.tar.gz
tar xzf gearmand-1.1.3.tar.gz
cd gearmand-1.1.3
./configure
make
make install
ldconfig

Create the upstart process:

nano /etc/init/gearmand.conf

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Start the daemon:

service gearmand start

Good to go!

The Long Version

First things first, let’s download the gearman source files and configure, so in an ssh window -

root@gearman-01:/# wget https://launchpad.net/gearmand/1.2/1.1.3/+download/gearmand-1.1.3.tar.gz
...
2012-12-11 15:56:53 (2.25 MB/s) - `gearmand-1.1.3.tar.gz' saved [857122/857122]

root@gearman-01:/# tar xzf gearmand-1.1.3.tar.gz
root@gearman-01:/# cd gearmand-1.1.3
root@gearman-01:/gearmand-1.1.3# ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/gearmand-1.1.3':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

So… we have a problem. We need to get a compiler.

Luckily Ubuntu have a lovely package called build-essential which gets us the required compilers we need. This also saved on lots of hassle trying to install them individually.

root@gearman-01:/gearmand-1.1.3# apt-get install build-essential
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  binutils cpp cpp-4.6 dpkg-dev fakeroot g++ g++-4.6 gcc gcc-4.6 libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libc-bin libc-dev-bin libc6 libc6-dev libdpkg-perl libgomp1 libmpc2 libmpfr4 libquadmath0
  libstdc++6-4.6-dev linux-libc-dev make manpages-dev
...
2 upgraded, 24 newly installed, 0 to remove and 79 not upgraded.
Need to get 36.4 MB of archives.
After this operation, 83.9 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
...

So let’s try and configure again:

root@gearman-01:/gearmand-1.1.3# ./configure
...
checking for Boost headers version >= 1.39.0... no
configure: error: cannot find Boost headers version >= 1.39.0

OK so we need to get some updated boost headers. No problem:

root@gearman-01:/gearmand-1.1.3# apt-get install libboost-thread-dev libboost-program-options-dev
The following extra packages will be installed:
  libboost-date-time1.46-dev libboost-date-time1.46.1 libboost-program-options1.46-dev libboost-program-options1.46.1
  libboost-serialization1.46-dev libboost-serialization1.46.1 libboost-thread1.46-dev libboost-thread1.46.1 libboost1.46-dev
...
0 upgraded, 11 newly installed, 0 to remove and 79 not upgraded.
Need to get 358 kB/8,619 kB of archives.
After this operation, 63.6 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
...

root@gearman-01:/gearmand-1.1.3# ./configure
...
checking event.h usability... no
checking event.h presence... no
checking for event.h... no
configure: error: Unable to find libevent

Let’s get libevent then:

root@gearman-01:/gearmand-1.1.3# apt-get install libevent-dev
The following extra packages will be installed:
  libevent-core-2.0-5 libevent-extra-2.0-5 libevent-openssl-2.0-5 libevent-pthreads-2.0-5
...
0 upgraded, 5 newly installed, 0 to remove and 79 not upgraded.
Need to get 426 kB of archives.
After this operation, 1,832 kB of additional disk space will be used.
Do you want to continue [Y/n]? y

root@gearman-01:/gearmand-1.1.3# ./configure
...
Configuration summary for gearmand version 1.1.3

   * Installation prefix:       /usr/local
   * System type:               unknown-linux-gnu
   * Host CPU:                  x86_64
   * C Compiler:                gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
   * C Flags:                   -g -O2  -Wno-pragmas -Wall -Wextra -Wunknown-pragmas -std=c99 --param=ssp-buffer-size=1 -Wno-attri                butes -Waddress -Warray-bounds -Wbad-function-cast -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat-security -Wformat=2 -Wlogica                l-op -Wmissing-field-initializers -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wnormalized=id -Woverride-init -Wpoint                er-arith -Wpointer-sign -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-overflow=1 -Wswitch-enum -Wundef -Wunused -Wunused-resu                lt -Wunused-variable -Wwrite-strings -floop-parallelize-all -fwrapv -pipe
   * C++ Compiler:              g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
   * C++ Flags:                 -g -O2 -Wno-pragmas -Wall -Wextra -Wunknown-pragmas --param=ssp-buffer-size=1 -Wno-attributes -Wad                dress -Warray-bounds -Wchar-subscripts -Wcomment -Wctor-dtor-privacy -Wfloat-equal -Wformat=2 -Wmissing-field-initializers -Wlogic                al-op -Wnon-virtual-dtor -Wnormalized=id -Woverloaded-virtual -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-o                verflow=1 -Wswitch-enum -Wundef -Wunused -Wunused-result -Wunused-variable -Wwrite-strings -Wformat-security -floop-parallelize-al                l -fwrapv
   * CPP Flags:                  -fvisibility=hidden
   * LIB Flags:                 -pie
   * Assertions enabled:        no
   * Debug enabled:             no
   * Warnings as failure:       no
   * Building with libsqlite3   no
   * Building with libdrizzle   no
   * Building with libmemcached not found
   * Building with libpq        no
   * Building with tokyocabinet no
   * Building with libmysql     no
   * make -j:                   2
   * VCS checkout:              no

Great, so gearman is configured, but we need to get libmysql:

root@gearman-01:/gearmand-1.1.3# apt-get install libmysqld-dev
...
The following extra packages will be installed:
  libmysqlclient-dev libmysqlclient18 mysql-common zlib1g-dev

root@gearman-01:/gearmand-1.1.3# ./configure
...
Configuration summary for gearmand version 1.1.3
   * Building with libmysql     yes

And now proceed with compiling:

root@gearman-01:/gearmand-1.1.3# make
make -j2  all-am
make[1]: Entering directory `/gearmand-1.1.3'
  CXX    libgearman/libgearman_libgearman_la-gearman.lo
  CXX    libgearman/libgearman_libgearman_la-job.lo
  CXX    libgearman/libgearman_libgearman_la-kill.lo
libgearman/job.cc: In function 'gearman_return_t _job_send(gearman_job_st*)':
libgearman/job.cc:606:25: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed
libgearman/job.cc: In function 'gearman_return_t gearman_job_send_complete_fin(gearman_job_st*, const void*, size_t)':
libgearman/job.cc:355:18: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed
libgearman/job.cc: In function 'bool gearman_job_build_reducer(gearman_job_st*, gearman_return_t (*)(gearman_aggregator_st*, gearman_task_st*, gearman_result_st*))':
libgearman/job.cc:209:6: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed
make[1]: *** [libgearman/libgearman_libgearman_la-job.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/gearmand-1.1.3'
make: *** [all] Error 2

Looks like we need another package called libcloog-ppl0 to get this working:

root@gearman-01:/gearmand-1.1.3# apt-get install libcloog-ppl0
...
The following extra packages will be installed:
  libgmpxx4ldbl libppl-c4 libppl9 libpwl5

root@gearman-01:/gearmand-1.1.3# make

I didn’t get any errors so all good to proceed installing the program:

root@gearman-01:/gearmand-1.1.3# make install
...
Libraries have been installed in:
   /usr/local/lib
...
libtool: install: /usr/bin/install -c bin/.libs/gearman /usr/local/bin/gearman
libtool: install: /usr/bin/install -c bin/gearadmin /usr/local/bin/gearadmin
make[2]: Leaving directory `/gearmand-1.1.3'
make[1]: Leaving directory `/gearmand-1.1.3'

All done! Gearman is now compiled with libmysql for MySQL support and installed on the server. Hurrah!

You’ll find the installation at /usr/local/sbin/gearmand and you can run gearmand to start a server.

To use MySQL persistent storage with gearman you will want to pass some flags in the command e.g.

root@gearman-01:/# gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Set the MySQL server host, the port (default 3306), MySQL user (in my case named gearman), the user password, the database name for the gearman queues (i called mine gearman) and a table name to use for this server (i called mine gearman_01).

This will run the gearman program and store any jobs it received in the gearman_01 table. You will notice if you run the command above the gearman_01 table is created in the gearman database.

A better idea might be to put this code into an upstart job to create a daemon, so that gearman is started along with the server:

root@gearman-01:/# nano /etc/init/gearmand.conf

And paste in the following, substituting the flags as per above:

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

This will start the gearman daemon whenever the server boots up and try to ensure it is always running.

To start the service manually just type:

root@gearman-01:/# service gearmand start
gearmand start/running, process 12803

To test the gearman job server just run the following command:

root@gearman-01:/# gearman -f testq -b payload
gearman: error while loading shared libraries: libgearman.so.8: cannot open shared object file: No such file or directory

If you get the error above as i did you’ll need to run ldconfig to clean a few things up:

root@gearman-01:/# ldconfig
root@gearman-01:/# gearman -f testq -b payload

Great, the job seems to have been entered into gearman.

If you look at the gearman_01 table in the gearman database you’ll see there is a row with “testq” function_name and a data blob with “payload” as per the job we just added.

It does seem odd however there there is no unique_key. Let’s try to insert another job and see what happens:

root@gearman-01:/# gearman -f testq -b payload
gearman: gearman_client_run_tasks : _client_run_tasks(GEARMAN_SERVER_ERROR) queue_insert_error:QUEUE_ERROR -> libgearman/client.cc:1504

So, clearly an error there!

We need to install the uuid-dev package to generate unique IDs.

root@gearman-01:/# apt-get install uuid-dev
The following NEW packages will be installed
  uuid-dev

What a PITA. We’ll need to re-compile, then restart the gearmand daemon.

root@gearman-01:/# cd /gearmand-1.1.3
root@gearman-01:/gearmand-1.1.3# ./configure
root@gearman-01:/gearmand-1.1.3# make && make install && ldconfig
root@gearman-01:/gearmand-1.1.3# service gearmand restart

And we can now run the test command as much as we like, which will create lots of entries we can see the gearman table, so all good to go.

Job jobbed!