Gearman 0.41 with MySQL Persistent Storage and PHP PECL 0.8.3 with GearmanManager on Ubuntu 12.04

The Problem

If you wish to use Brian Moon’s GearmanManager daemon to manage your gearman worker scripts, the most recent PHP PECL gearman release (v1.1.0) will not work due to a known bug (which caused me many hours of pain).

TL;DR (The Short Version)

To use GearmanManager we need to use the previous version of PECL Gearman (currently v0.8.3) and thus the previous version of libgearman (Gearmand currently v0.41). Using Gearmand v0.41 we have native MySQL support with libmysqlclient.

Installation instructions are very much as per my how to install gearman with mysql persistent storage on ubuntu post, plus installation of PHP PECL Gearman and GearmanManager.

From a shell prompt install Gearman 0.41:

apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev -y
wget https://launchpad.net/gearmand/1.0/0.41/+download/gearmand-0.41.tar.gz
tar xzf gearmand-0.41.tar.gz
cd gearmand-0.41
./configure
make
make install
ldconfig

Create the upstart process:

nano /etc/init/gearmand.conf

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Start the daemon:

service gearmand start

Install PHP CLI, along with any extensions you might need, and PECL Gearman 0.8.3:

apt-get install php5-cli php5-dev php-pear -y
pecl install gearman-0.8.3

Enable PECL Gearman extension in PHP:

nano /etc/php5/cli/conf.d/gearman.ini

extension=gearman.so

Install Gearman Manager:

apt-get install git-core
git clone https://github.com/brianlmoon/GearmanManager.git /var/gearman/GearmanManager
/var/gearman/GearmanManager/install/install.sh

Configure Gearman Manager:

nano /etc/gearman-manager/config.ini

Set your gearman job servers e.g. host=172.29.17.56,172.29.17.57

Reset the GearmanManager process:

/etc/init.d/gearman-manager restart

The Long Version

The PHP PECL Gearman 1.x branch was written to work with with later gearmand versions (libgearman v0.21 and above), and which appears to have some fairly fundamental changes. As such this entire branch seems to be unstable and unusable with GearmanManager.

The previous beta version however does work, and looks to be updated in parallel with the 1.x version (great for us!), with the latest edition being PECL-Gearman 0.8.3.

PECL-Gearman 1.1.0 works with the most recent Gearmand release (1.1.3), which uses the updated libgearman, and is incompatible with the PECL-Gearman 0.8.3 release we want to use, so, we need to use the previous version of gearman which does work – Gearmand v0.41.

Once-upon-a-time if we wanted a persistent queue in MySQL we’d have to go through the pain of trying to get libdrizzle installed, however as of Gearman v0.38 libmysqlclient is supported, meaning we have native MySQL support. Hurrah!

The key point here to install Gearmand v0.41 with PHP PECL-Gearman v0.8.3.

Installation instructions for Gearman are very much as per my how to install gearman with mysql persistent storage on ubuntu post, plus installation of the PHP PECL Gearman extension and GearmanManager itself.

Run the following from a shell prompt to install Gearman 0.41 with MySQL:

apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev
wget https://launchpad.net/gearmand/1.0/0.41/+download/gearmand-0.41.tar.gz
tar xzf gearmand-0.41.tar.gz
cd gearmand-0.41
./configure
make
make install
ldconfig

Now create the upstart process:

nano /etc/init/gearmand.conf

And paste the following:

description "Gearmand Server"

start on startup
stop on runlevel [016]

kill timeout 3

respawn

exec gearmand \
--log-file=/var/log/gearmand.log \
--queue-type=MySQL \
--mysql-host=192.168.1.10 \
--mysql-port=3306 \
--mysql-user=gearman \
--mysql-password=your_password \
--mysql-db=gearman \
--mysql-table=gearman_01

Start the job server daemon:

service gearmand start

Now install PHP, along with any extensions you might need, and the php-pear extension so that we can install PECL-Gearman afterwards:

apt-get install php5-cli php5-dev php-pear

Install the PECL Gearman 0.8.3 extension:

pecl install gearman-0.8.3

Enable PECL Gearman extension in PHP:

nano /etc/php5/cli/conf.d/gearman.ini

And paste the following:

extension=gearman.so

Now install Gearman Manager:

apt-get install git-core
git clone https://github.com/brianlmoon/GearmanManager.git /var/gearman/GearmanManager
/var/gearman/GearmanManager/install/install.sh

Make sure you select the PECL installation and then we can configure Gearman Manager:

nano /etc/gearman-manager/config.ini

Set your gearman job servers e.g. host=172.29.17.56,172.29.17.57 and restart the GearmanManager process:

/etc/init.d/gearman-manager restart

You can see a full list of the GearmanManager config options on the GearmanManager github project page.

6 thoughts on “Gearman 0.41 with MySQL Persistent Storage and PHP PECL 0.8.3 with GearmanManager on Ubuntu 12.04

  1. Pingback: Compiling Gearman 1.1.3 with MySQL Persistent Storage on Ubuntu 12.04 LTS Precise | Andy Burton

  2. Andy,

    Great post! I had to change the upstart script to:

    start on runlevel [2345]

    To get everything going on startup. Hopefully this will save someone some time if they cannot figure out whey gearmand is not coming up when they restart the machine.

  3. Andy

    I have compiled gearman 0.38 on CentOS 6 x86_64. I have specified the queue to be MySQL but I am not sure when does gearmand persists in DB. After submitting few jobs the count in gearman_queue table is 0.

    gearmand log confirms that it has loaded the MySQL queue

    INFO 2013-00-23 22:26:01.557691 [ main ] Initializing MySQL module
    INFO 2013-00-23 22:26:01.561853 [ main ] Initializing Gear
    INFO 2013-00-23 22:26:01.562111 [ main ] Starting up, verbose set to DEBUG

    Any clue?

    Rohit Jain

    • Hi Rohit,

      Gearman uses MySQL to provide persistent storage for the job queue.

      Whenever a job is added to the queue it is stored in MySQL, providing you have configured it correct and given it MySQL details that it can use.

      When the job is processed the row is removed from MySQL.

      Note that gearman uses MySQL as persistent storage, it does not read jobs from MySQL unless gearman is restarted e.g. if you add a row into the gearman MySQL table it will not be processed by gearman unless you restart the gearman process, which causes gearman to load any jobs stored in MySQL.

      Hopefully this answers your questions :)

      Andy

  4. Hi, I’m currently trying/evaluating gearman for job server solution. Would you mind answering a few questions?

    Are you saying that GearmanManager cannot run with gearman v 1.x?

    And totally unrelated – did you figure out the way to create a background job from client and poll for status later (or better – use some kind of notification at the end of the job) instead of doing it in the same script (as doc examples suggest)?
    I mean – the whole point of job server is to have a responsive script that puts the job in the queue, rather than wait until the job is done.
    Obviously, we would need to return the job id, save it somewhere and use it to query the server for job status later.

    Sorry for bothering, but I’m a bit discouraged by the fact that docs/examples don’t suggest that this is possible, but I just don’t see the other way it could all work together, so I’d appreciate a positive signal that things just can work as expected.

    Cheers
    v.

    • Hi,

      The current version of GearmanManager will not run with Gearman v1.x no. You may well be able to edit GearmanManager to stop it form erroring, but i haven’t attempted this or seen anybody else.

      The problem is with the PHP PECL Gearman extension though, not GearmanManager as i understand it.

      Gearman does not log a record of jobs it’s processed, so there is no built-in functionality to poll Gearman to check for processed jobs after the event.

      You would have to have the worker process log a record when it successfully completes it’s job, and then use this to work out what’s happened. This is how i’ve done it before, which is quite easy as you just create a generic worker class which logs activity and inherit all worker processes from it.

      The whole point of Gearman is to provide a scaleable parallel processing system, which it does. You then have the flexibility to run it in blocking or non-blocking mode (i.e background job or not) to keep the client waiting.

      Hopefully this helps,

      Andy

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>