If you wish to use Brian Moon’s GearmanManager daemon to manage your gearman worker scripts, the most recent PHP PECL gearman release (v1.1.0) will not work due to a known bug (which caused me many hours of pain).
Your best bet is to follow my How to install Gearman with MySQL Persistent Storage and PHP PECL Gearman 0.8.3 with Brian Moon’s GearmanManager on Ubuntu 12.04 post.
I had some problems getting Gearman working on Ubuntu and figured others might have the same issues.
I’d previously written instructions for getting gearman working on 10.04 with standard ubuntu packages and with libdrizzle for persistent mysql storage of the jobs, but this didn’t work when attempting to use the latest version of gearman from launchpad and ubuntu 12.04 precise.
I opted to compile gearman from source on a clean installation of Ubuntu 12.04 Precise and after a bit of trouble getting libdrizzle to work found that gearman will compile with libmysql support for direct MySQL integration. This saved a lot of hassle, and here’s how i did it -
TL;DR (The Short Version)
Instructions for compiling and installing Gearman 1.1.3 with MySQL persistent storage on Ubuntu 12.04 LTS Precise:
apt-get install build-essential libboost-thread-dev libboost-program-options-dev libevent-dev libcloog-ppl0 libmysqld-dev uuid-dev -y wget https://launchpad.net/gearmand/1.2/1.1.3/+download/gearmand-1.1.3.tar.gz tar xzf gearmand-1.1.3.tar.gz cd gearmand-1.1.3 ./configure make make install ldconfig
Create the upstart process:
nano /etc/init/gearmand.conf description "Gearmand Server" start on startup stop on runlevel [016] kill timeout 3 respawn exec gearmand \ --log-file=/var/log/gearmand.log \ --queue-type=MySQL \ --mysql-host=192.168.1.10 \ --mysql-port=3306 \ --mysql-user=gearman \ --mysql-password=your_password \ --mysql-db=gearman \ --mysql-table=gearman_01
Start the daemon:
service gearmand start
Good to go!
The Long Version
First things first, let’s download the gearman source files and configure, so in an ssh window -
root@gearman-01:/# wget https://launchpad.net/gearmand/1.2/1.1.3/+download/gearmand-1.1.3.tar.gz ... 2012-12-11 15:56:53 (2.25 MB/s) - `gearmand-1.1.3.tar.gz' saved [857122/857122] root@gearman-01:/# tar xzf gearmand-1.1.3.tar.gz root@gearman-01:/# cd gearmand-1.1.3 root@gearman-01:/gearmand-1.1.3# ./configure checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking for gcc... no checking for cc... no checking for cl.exe... no configure: error: in `/gearmand-1.1.3': configure: error: no acceptable C compiler found in $PATH See `config.log' for more details
So… we have a problem. We need to get a compiler.
Luckily Ubuntu have a lovely package called build-essential which gets us the required compilers we need. This also saved on lots of hassle trying to install them individually.
root@gearman-01:/gearmand-1.1.3# apt-get install build-essential Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: binutils cpp cpp-4.6 dpkg-dev fakeroot g++ g++-4.6 gcc gcc-4.6 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libc-bin libc-dev-bin libc6 libc6-dev libdpkg-perl libgomp1 libmpc2 libmpfr4 libquadmath0 libstdc++6-4.6-dev linux-libc-dev make manpages-dev ... 2 upgraded, 24 newly installed, 0 to remove and 79 not upgraded. Need to get 36.4 MB of archives. After this operation, 83.9 MB of additional disk space will be used. Do you want to continue [Y/n]? y ...
So let’s try and configure again:
root@gearman-01:/gearmand-1.1.3# ./configure ... checking for Boost headers version >= 1.39.0... no configure: error: cannot find Boost headers version >= 1.39.0
OK so we need to get some updated boost headers. No problem:
root@gearman-01:/gearmand-1.1.3# apt-get install libboost-thread-dev libboost-program-options-dev The following extra packages will be installed: libboost-date-time1.46-dev libboost-date-time1.46.1 libboost-program-options1.46-dev libboost-program-options1.46.1 libboost-serialization1.46-dev libboost-serialization1.46.1 libboost-thread1.46-dev libboost-thread1.46.1 libboost1.46-dev ... 0 upgraded, 11 newly installed, 0 to remove and 79 not upgraded. Need to get 358 kB/8,619 kB of archives. After this operation, 63.6 MB of additional disk space will be used. Do you want to continue [Y/n]? y ... root@gearman-01:/gearmand-1.1.3# ./configure ... checking event.h usability... no checking event.h presence... no checking for event.h... no configure: error: Unable to find libevent
Let’s get libevent then:
root@gearman-01:/gearmand-1.1.3# apt-get install libevent-dev The following extra packages will be installed: libevent-core-2.0-5 libevent-extra-2.0-5 libevent-openssl-2.0-5 libevent-pthreads-2.0-5 ... 0 upgraded, 5 newly installed, 0 to remove and 79 not upgraded. Need to get 426 kB of archives. After this operation, 1,832 kB of additional disk space will be used. Do you want to continue [Y/n]? y root@gearman-01:/gearmand-1.1.3# ./configure ... Configuration summary for gearmand version 1.1.3 * Installation prefix: /usr/local * System type: unknown-linux-gnu * Host CPU: x86_64 * C Compiler: gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 * C Flags: -g -O2 -Wno-pragmas -Wall -Wextra -Wunknown-pragmas -std=c99 --param=ssp-buffer-size=1 -Wno-attri butes -Waddress -Warray-bounds -Wbad-function-cast -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat-security -Wformat=2 -Wlogica l-op -Wmissing-field-initializers -Wmissing-noreturn -Wmissing-prototypes -Wnested-externs -Wnormalized=id -Woverride-init -Wpoint er-arith -Wpointer-sign -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-overflow=1 -Wswitch-enum -Wundef -Wunused -Wunused-resu lt -Wunused-variable -Wwrite-strings -floop-parallelize-all -fwrapv -pipe * C++ Compiler: g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3 * C++ Flags: -g -O2 -Wno-pragmas -Wall -Wextra -Wunknown-pragmas --param=ssp-buffer-size=1 -Wno-attributes -Wad dress -Warray-bounds -Wchar-subscripts -Wcomment -Wctor-dtor-privacy -Wfloat-equal -Wformat=2 -Wmissing-field-initializers -Wlogic al-op -Wnon-virtual-dtor -Wnormalized=id -Woverloaded-virtual -Wpointer-arith -Wredundant-decls -Wshadow -Wsign-compare -Wstrict-o verflow=1 -Wswitch-enum -Wundef -Wunused -Wunused-result -Wunused-variable -Wwrite-strings -Wformat-security -floop-parallelize-al l -fwrapv * CPP Flags: -fvisibility=hidden * LIB Flags: -pie * Assertions enabled: no * Debug enabled: no * Warnings as failure: no * Building with libsqlite3 no * Building with libdrizzle no * Building with libmemcached not found * Building with libpq no * Building with tokyocabinet no * Building with libmysql no * make -j: 2 * VCS checkout: no
Great, so gearman is configured, but we need to get libmysql:
root@gearman-01:/gearmand-1.1.3# apt-get install libmysqld-dev ... The following extra packages will be installed: libmysqlclient-dev libmysqlclient18 mysql-common zlib1g-dev root@gearman-01:/gearmand-1.1.3# ./configure ... Configuration summary for gearmand version 1.1.3 * Building with libmysql yes
And now proceed with compiling:
root@gearman-01:/gearmand-1.1.3# make make -j2 all-am make[1]: Entering directory `/gearmand-1.1.3' CXX libgearman/libgearman_libgearman_la-gearman.lo CXX libgearman/libgearman_libgearman_la-job.lo CXX libgearman/libgearman_libgearman_la-kill.lo libgearman/job.cc: In function 'gearman_return_t _job_send(gearman_job_st*)': libgearman/job.cc:606:25: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed libgearman/job.cc: In function 'gearman_return_t gearman_job_send_complete_fin(gearman_job_st*, const void*, size_t)': libgearman/job.cc:355:18: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed libgearman/job.cc: In function 'bool gearman_job_build_reducer(gearman_job_st*, gearman_return_t (*)(gearman_aggregator_st*, gearman_task_st*, gearman_result_st*))': libgearman/job.cc:209:6: sorry, unimplemented: Graphite loop optimizations can only be used if the libcloog-ppl0 package is installed make[1]: *** [libgearman/libgearman_libgearman_la-job.lo] Error 1 make[1]: *** Waiting for unfinished jobs.... make[1]: Leaving directory `/gearmand-1.1.3' make: *** [all] Error 2
Looks like we need another package called libcloog-ppl0 to get this working:
root@gearman-01:/gearmand-1.1.3# apt-get install libcloog-ppl0 ... The following extra packages will be installed: libgmpxx4ldbl libppl-c4 libppl9 libpwl5 root@gearman-01:/gearmand-1.1.3# make
I didn’t get any errors so all good to proceed installing the program:
root@gearman-01:/gearmand-1.1.3# make install ... Libraries have been installed in: /usr/local/lib ... libtool: install: /usr/bin/install -c bin/.libs/gearman /usr/local/bin/gearman libtool: install: /usr/bin/install -c bin/gearadmin /usr/local/bin/gearadmin make[2]: Leaving directory `/gearmand-1.1.3' make[1]: Leaving directory `/gearmand-1.1.3'
All done! Gearman is now compiled with libmysql for MySQL support and installed on the server. Hurrah!
You’ll find the installation at /usr/local/sbin/gearmand and you can run gearmand to start a server.
To use MySQL persistent storage with gearman you will want to pass some flags in the command e.g.
root@gearman-01:/# gearmand \ --log-file=/var/log/gearmand.log \ --queue-type=MySQL \ --mysql-host=192.168.1.10 \ --mysql-port=3306 \ --mysql-user=gearman \ --mysql-password=your_password \ --mysql-db=gearman \ --mysql-table=gearman_01
Set the MySQL server host, the port (default 3306), MySQL user (in my case named gearman), the user password, the database name for the gearman queues (i called mine gearman) and a table name to use for this server (i called mine gearman_01).
This will run the gearman program and store any jobs it received in the gearman_01 table. You will notice if you run the command above the gearman_01 table is created in the gearman database.
A better idea might be to put this code into an upstart job to create a daemon, so that gearman is started along with the server:
root@gearman-01:/# nano /etc/init/gearmand.conf
And paste in the following, substituting the flags as per above:
description "Gearmand Server" start on startup stop on runlevel [016] kill timeout 3 respawn exec gearmand \ --log-file=/var/log/gearmand.log \ --queue-type=MySQL \ --mysql-host=192.168.1.10 \ --mysql-port=3306 \ --mysql-user=gearman \ --mysql-password=your_password \ --mysql-db=gearman \ --mysql-table=gearman_01
This will start the gearman daemon whenever the server boots up and try to ensure it is always running.
To start the service manually just type:
root@gearman-01:/# service gearmand start gearmand start/running, process 12803
To test the gearman job server just run the following command:
root@gearman-01:/# gearman -f testq -b payload gearman: error while loading shared libraries: libgearman.so.8: cannot open shared object file: No such file or directory
If you get the error above as i did you’ll need to run ldconfig to clean a few things up:
root@gearman-01:/# ldconfig root@gearman-01:/# gearman -f testq -b payload
Great, the job seems to have been entered into gearman.
If you look at the gearman_01 table in the gearman database you’ll see there is a row with “testq” function_name and a data blob with “payload” as per the job we just added.
It does seem odd however there there is no unique_key. Let’s try to insert another job and see what happens:
root@gearman-01:/# gearman -f testq -b payload gearman: gearman_client_run_tasks : _client_run_tasks(GEARMAN_SERVER_ERROR) queue_insert_error:QUEUE_ERROR -> libgearman/client.cc:1504
So, clearly an error there!
We need to install the uuid-dev package to generate unique IDs.
root@gearman-01:/# apt-get install uuid-dev The following NEW packages will be installed uuid-dev
What a PITA. We’ll need to re-compile, then restart the gearmand daemon.
root@gearman-01:/# cd /gearmand-1.1.3 root@gearman-01:/gearmand-1.1.3# ./configure root@gearman-01:/gearmand-1.1.3# make && make install && ldconfig root@gearman-01:/gearmand-1.1.3# service gearmand restart
And we can now run the test command as much as we like, which will create lots of entries we can see the gearman table, so all good to go.
Job jobbed!
Pingback: Gearman 0.41 with MySQL Persistent Storage and PHP PECL 0.8.3 with GearmanManager | Andy Burton
Andy this is fantastic I was getting a random error from my client written in C.
_client_run_tasks(GEARMAN_SERVER_ERROR) queue_insert_error:QUEUE_ERROR -> libgearman/client.cc:1504
So I followed the steps and installed the uuid-dev package, recompiled and it works. (steps below)
apt-get install uuid-dev
make && make install && ldconfig
rebuilt my application and the errors went away!!!
Well done! I wish I had the patience to record like this whenever I setup something by trial and error!
So great!
I met the most of same problems with Any
root@gearman-01:/gearmand-1.1.6# make
I met the new error:-std=c99
So I edit the Makefile,add “-std=c99″ on the 2053 line.
Thanks for Andy’s share!
Pingback: Gearman 安裝心得 (Ubuntu) | Zack's Note
i got error: gearmand : Error while initializing the queue : MySQL
do i need to create the database in advance?
You will need to create the database yes but not the tables.
thx for ur input, i figure that part out..btw the upstart script is not working when machine reboot..i need to manually start the gearmand each time i boot the machine..can u help with that?