Gearman是一个分发任务的程序框架,可以用在各种场合,与相比,Gearman更偏向于任务分发功能。它的 任务分布非常 简单,简单得可以只需要用脚本即可完成。Gearman最初用于LiveJournal的图片resize功能,由于图片resize需要消耗大量计算资 源,因此需要调度到后端多台服务器执行,完成任务之后返回前端再呈现到界面。
Gearman 分布式任务实现原理上只用到2个字段,function name和data。function name即任务名称,由client传给job server, job server根据function name选择合适的worker节点来执行。data通常为执行任务所需的自定义的内容,比如简单的做法可以把需要执行的脚本当成data即可(当然要注 意其中的安全防范)。如果有多个worker可以处理同一个function name, 则job server会自动分配一个。当用于远程监控场景时,我们可以让每个worker注册成不同的业务名称,以达到方便控制每台worker节点的目的。
gearmand安装
安装依赖
yum install -y gcc-c++ boost-devel gperf libevent-devel libuuid-devel
boost也可以编译安装(从网上找的,未测试)
//安装ICU4C wget http://downloads.sourceforge.net/project/icu/ICU4C/4.0/icu4c-4_0-src.tgz?use_mirror=cdnetworks-kr-2tar zxvf icu4c-4_0-src.tgzcd icu/source./configure –prefix=/usrmakemake installldconfig//安装Boost wget http://sourceforge.net/projects/boost/files/boost/1.43.0/boost_1_43_0.tar.gz/downloadtar zxvf boost_1_43_0.tar.gzcd boost_1_43_0rm -rf /usr/include/boost/rm -rf /usr/lib/libboost*./bootstrap.sh -prefix=/usr/local/boost./b2编译大概半小时,完成后:./b2 install此时编译安装gearman需要制定boost目录./configure --with-boost=/usr/local/boost --with-boost-libdir=/usr/local/boost/lib
编译安装gearman
下载地址https://github.com/gearman/gearmand/releases|https://launchpad.net/gearmand/
# tar xf gearmand-1.1.15.tar.gz# cd gearmand-1.1.15# ./configure --prefix=/app/gearman# make && make install
运行
/app/gearman/sbin/gearmand -d
php扩展安装
下载地址http://pecl.php.net/package/gearman
图简单,直接yum安装php环境# yum install httpd php php-devel php-mysql php-mhash php-cli php-gd php-common php-mbstring php-ldap php-pdo mariadb-server -y# tar xf gearman-1.1.2.tgz# cd gearman-1.1.2# ./configure --with-php-config=/usr/bin/php-config --with-gearman=/app/gearman# phpize# make# make install# vim /etc/php.d/gearman.iniextension=/usr/lib64/php/modules/gearman.so
重启PHP服务
查看状态:/www/wdlinux/php/bin/php --info |grep gearman
PHP测试
Work端
addServer();$worker->addFunction("reverse", "my_reverse_function");while ($worker->work());function my_reverse_function($job){ return strrev($job->workload());}?>
运行
php worker.php
客户端
addServer();print $client->do("reverse", "Hello World!");?>
运行
[root@localhost app]# php client.php!dlroW olleH
可以看到已经可以使用。
JAVA测试
代码引入gearman包
org.gearman gearman-java 0.6
编写Work代码
具体执行work的工作类
import org.gearman.client.GearmanJobResult;import org.gearman.client.GearmanJobResultImpl;import org.gearman.util.ByteUtils;import org.gearman.worker.AbstractGearmanFunction;public class ReverseFunction extends AbstractGearmanFunction { public GearmanJobResult executeFunction() { StringBuffer sb = new StringBuffer(ByteUtils.fromUTF8Bytes((byte[]) this.data)); // 封装结果 GearmanJobResult gjr = new GearmanJobResultImpl( this.jobHandle, true, sb.reverse().toString().getBytes(), new byte[0], new byte[0], 0, 0); return gjr; }}
Work端
import java.util.ArrayList;import java.util.List;import org.gearman.common.Constants;import org.gearman.common.GearmanNIOJobServerConnection;import org.gearman.worker.GearmanFunction;import org.gearman.worker.GearmanWorker;import org.gearman.worker.GearmanWorkerImpl;public class WorkerRunner { GearmanNIOJobServerConnection conn; List> functions; public WorkerRunner(String host, int port, List > funs) { conn = new GearmanNIOJobServerConnection(host, port); functions = new ArrayList >(); functions.addAll(funs); } public void start() { GearmanWorker worker = new GearmanWorkerImpl(); worker.addServer(conn); for (Class fun : functions) { worker.registerFunction(fun); } worker.work(); } @SuppressWarnings(value = { "unchecked", "rawtypes" }) public static void main(String[] args) throws Exception { String host = Constants.GEARMAN_DEFAULT_TCP_HOST; int port = Constants.GEARMAN_DEFAULT_TCP_PORT; Class c = Class.forName(ReverseFunction.class.getCanonicalName()); if (!GearmanFunction.class.isAssignableFrom(ReverseFunction.class)) { System.out.println("Class is not an instance of " + GearmanFunction.class.getCanonicalName()); return; } List > functions = new ArrayList >(); functions.add(c); new WorkerRunner(host, port,functions).start(); }}
客户端
import org.gearman.client.GearmanClient;import org.gearman.client.GearmanClientImpl;import org.gearman.client.GearmanJob;import org.gearman.client.GearmanJobImpl;import org.gearman.client.GearmanJobResult;import org.gearman.common.Constants;import org.gearman.common.GearmanJobServerConnection;import org.gearman.common.GearmanNIOJobServerConnection;import org.gearman.util.ByteUtils;public class ReverseClient { private GearmanClient client; private String function = ReverseFunction.class.getCanonicalName(); public ReverseClient(String host, int port) { GearmanJobServerConnection conn = new GearmanNIOJobServerConnection(host, port); client = new GearmanClientImpl(); client.addJobServer(conn); } public String reverse(String input) { String uniqueId = null; byte[] data = ByteUtils.toUTF8Bytes(input); GearmanJobResult res = null; GearmanJob job = GearmanJobImpl.createJob(function, data, uniqueId); String value = ""; client.submit(job); try { res = job.get(); value = ByteUtils.fromUTF8Bytes(res.getResults()); } catch (Exception e) { e.printStackTrace(); } return value; } public void shutdown() throws IllegalStateException { if (client == null) { throw new IllegalStateException("No client to shutdown"); } client.shutdown(); } public static void main(String[] args) { String host = Constants.GEARMAN_DEFAULT_TCP_HOST; int port = Constants.GEARMAN_DEFAULT_TCP_PORT; String payload = "Hello World"; ReverseClient rc = new ReverseClient(host, port); System.out.println(rc.reverse(payload)); rc.shutdown(); }}
先运行Work端,在运行客户端即可
JAVA和PHP互通测试
1)PHP作为Work端,JAVA作为客户端
Work.php代码不变,ReverseClient.java的function改为reverse即可(Work.php注册的函数名为reverse),即可做测试。
2)JAVA作为Work端,PHP作为客户端
WorkerRunner.java代码不变,client.php的do("reverse", "Hello World!")修改为do("org.inull.util.gearman.ReverseFunction", "Hello World!"),即可做测试。
测试第三方的gearman服务器
https://github.com/mywiki/gearman-java,测试,不兼容官方协议吗,不过提供了消息的持久化和web界面功能,也可以试试。