The Problem
If you wish to use Brian Moon’s GearmanManager daemon to manage your gearman worker scripts, the most recent PHP PECL gearman release (v1.1.0) will not work due to a known bug (which caused me many hours of pain).
TL;DR (The Short Version)
To use GearmanManager we need to use the previous version of PECL Gearman (currently v0.8.3) and thus the previous version of libgearman (Gearmand currently v0.41). Using Gearmand v0.41 we have native MySQL support with libmysqlclient.
Installation instructions are very much as per my how to install gearman with mysql persistent storage on ubuntu post, plus installation of PHP PECL Gearman and GearmanManager.
From a shell prompt install Gearman 0.41:
apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev -y wget https://launchpad.net/gearmand/1.0/0.41/+download/gearmand-0.41.tar.gz tar xzf gearmand-0.41.tar.gz cd gearmand-0.41 ./configure make make install ldconfig
Create the upstart process:
nano /etc/init/gearmand.conf description "Gearmand Server" start on startup stop on runlevel [016] kill timeout 3 respawn exec gearmand \ --log-file=/var/log/gearmand.log \ --queue-type=MySQL \ --mysql-host=192.168.1.10 \ --mysql-port=3306 \ --mysql-user=gearman \ --mysql-password=your_password \ --mysql-db=gearman \ --mysql-table=gearman_01
Start the daemon:
service gearmand start
Install PHP CLI, along with any extensions you might need, and PECL Gearman 0.8.3:
apt-get install php5-cli php5-dev php-pear -y pecl install gearman-0.8.3
Enable PECL Gearman extension in PHP:
nano /etc/php5/cli/conf.d/gearman.ini extension=gearman.so
Install Gearman Manager:
apt-get install git-core git clone https://github.com/brianlmoon/GearmanManager.git /var/gearman/GearmanManager /var/gearman/GearmanManager/install/install.sh
Configure Gearman Manager:
nano /etc/gearman-manager/config.ini
Set your gearman job servers e.g. host=172.29.17.56,172.29.17.57
Reset the GearmanManager process:
/etc/init.d/gearman-manager restart
The Long Version
The PHP PECL Gearman 1.x branch was written to work with with later gearmand versions (libgearman v0.21 and above), and which appears to have some fairly fundamental changes. As such this entire branch seems to be unstable and unusable with GearmanManager.
The previous beta version however does work, and looks to be updated in parallel with the 1.x version (great for us!), with the latest edition being PECL-Gearman 0.8.3.
PECL-Gearman 1.1.0 works with the most recent Gearmand release (1.1.3), which uses the updated libgearman, and is incompatible with the PECL-Gearman 0.8.3 release we want to use, so, we need to use the previous version of gearman which does work – Gearmand v0.41.
Once-upon-a-time if we wanted a persistent queue in MySQL we’d have to go through the pain of trying to get libdrizzle installed, however as of Gearman v0.38 libmysqlclient is supported, meaning we have native MySQL support. Hurrah!
The key point here to install Gearmand v0.41 with PHP PECL-Gearman v0.8.3.
Installation instructions for Gearman are very much as per my how to install gearman with mysql persistent storage on ubuntu post, plus installation of the PHP PECL Gearman extension and GearmanManager itself.
Run the following from a shell prompt to install Gearman 0.41 with MySQL:
apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev wget https://launchpad.net/gearmand/1.0/0.41/+download/gearmand-0.41.tar.gz tar xzf gearmand-0.41.tar.gz cd gearmand-0.41 ./configure make make install ldconfig
Now create the upstart process:
nano /etc/init/gearmand.conf
And paste the following:
description "Gearmand Server" start on startup stop on runlevel [016] kill timeout 3 respawn exec gearmand \ --log-file=/var/log/gearmand.log \ --queue-type=MySQL \ --mysql-host=192.168.1.10 \ --mysql-port=3306 \ --mysql-user=gearman \ --mysql-password=your_password \ --mysql-db=gearman \ --mysql-table=gearman_01
Start the job server daemon:
service gearmand start
Now install PHP, along with any extensions you might need, and the php-pear extension so that we can install PECL-Gearman afterwards:
apt-get install php5-cli php5-dev php-pear
Install the PECL Gearman 0.8.3 extension:
pecl install gearman-0.8.3
Enable PECL Gearman extension in PHP:
nano /etc/php5/cli/conf.d/gearman.ini
And paste the following:
extension=gearman.so
Now install Gearman Manager:
apt-get install git-core git clone https://github.com/brianlmoon/GearmanManager.git /var/gearman/GearmanManager /var/gearman/GearmanManager/install/install.sh
Make sure you select the PECL installation and then we can configure Gearman Manager:
nano /etc/gearman-manager/config.ini
Set your gearman job servers e.g. host=172.29.17.56,172.29.17.57 and restart the GearmanManager process:
/etc/init.d/gearman-manager restart
You can see a full list of the GearmanManager config options on the GearmanManager github project page.
Pingback: Compiling Gearman 1.1.3 with MySQL Persistent Storage on Ubuntu 12.04 LTS Precise | Andy Burton
Andy,
Great post! I had to change the upstart script to:
start on runlevel [2345]
To get everything going on startup. Hopefully this will save someone some time if they cannot figure out whey gearmand is not coming up when they restart the machine.
Andy
I have compiled gearman 0.38 on CentOS 6 x86_64. I have specified the queue to be MySQL but I am not sure when does gearmand persists in DB. After submitting few jobs the count in gearman_queue table is 0.
gearmand log confirms that it has loaded the MySQL queue
INFO 2013-00-23 22:26:01.557691 [ main ] Initializing MySQL module
INFO 2013-00-23 22:26:01.561853 [ main ] Initializing Gear
INFO 2013-00-23 22:26:01.562111 [ main ] Starting up, verbose set to DEBUG
Any clue?
Rohit Jain
Hi Rohit,
Gearman uses MySQL to provide persistent storage for the job queue.
Whenever a job is added to the queue it is stored in MySQL, providing you have configured it correct and given it MySQL details that it can use.
When the job is processed the row is removed from MySQL.
Note that gearman uses MySQL as persistent storage, it does not read jobs from MySQL unless gearman is restarted e.g. if you add a row into the gearman MySQL table it will not be processed by gearman unless you restart the gearman process, which causes gearman to load any jobs stored in MySQL.
Hopefully this answers your questions
Andy
Hi, I’m currently trying/evaluating gearman for job server solution. Would you mind answering a few questions?
Are you saying that GearmanManager cannot run with gearman v 1.x?
And totally unrelated – did you figure out the way to create a background job from client and poll for status later (or better – use some kind of notification at the end of the job) instead of doing it in the same script (as doc examples suggest)?
I mean – the whole point of job server is to have a responsive script that puts the job in the queue, rather than wait until the job is done.
Obviously, we would need to return the job id, save it somewhere and use it to query the server for job status later.
Sorry for bothering, but I’m a bit discouraged by the fact that docs/examples don’t suggest that this is possible, but I just don’t see the other way it could all work together, so I’d appreciate a positive signal that things just can work as expected.
Cheers
v.
Hi,
The current version of GearmanManager will not run with Gearman v1.x no. You may well be able to edit GearmanManager to stop it form erroring, but i haven’t attempted this or seen anybody else.
The problem is with the PHP PECL Gearman extension though, not GearmanManager as i understand it.
Gearman does not log a record of jobs it’s processed, so there is no built-in functionality to poll Gearman to check for processed jobs after the event.
You would have to have the worker process log a record when it successfully completes it’s job, and then use this to work out what’s happened. This is how i’ve done it before, which is quite easy as you just create a generic worker class which logs activity and inherit all worker processes from it.
The whole point of Gearman is to provide a scaleable parallel processing system, which it does. You then have the flexibility to run it in blocking or non-blocking mode (i.e background job or not) to keep the client waiting.
Hopefully this helps,
Andy