Gearman

Gearman & anyworker

Gearman is an Open Source application framework that distributes processes to on or more machines to be executed there. There are many use cases for Gearman, with the most common to be load balancing, parallel work, sharing processes between different programming languages and executing processes in the background of the main process, with a pre-defined execution priority.

How Gearman works

There are three main components in an application that uses of Gearman:
1) The Gearman Client
2) The Job Server
3) And the Gearman worker

1) The Gearman Client runs on your application side and it is responsible for creating jobs (tasks), which forwards to the Job Server.

2) The Job Server is responsible for finding one or more available worker(s) for executing the task(s).

3) The worker (the anyworker in the case of Anymeta) is responsible for executing the task (concerning its priority) and returns a rensponse back to the client through the Job Server (see the image "Gearman-stack").

Gearman and Anymeta

Gearman was introduced in Anymeta 4.12 and since then it is used in all the Anymeta websites.
Anymeta has a client implementation in PHP (GearmanClient.php) and a worker implementation in python.
The GearmanClient adds prioritized tasks in the task queue, in "low", "normal" and "high" priority, which will be executed asynchronously by the anyworker. Anymeta has also, a number of helper functions (any_gearman.php) builded on top of the GearmanClient in order to perform checks in the tasks passed to the GearmanClient.
The single worker implementation, called anyworker, is based on Twisted (a python networking engine) and it is responsible for executing tasks from the task queue and pivot actions. The pivot actions transform Anymeta Things in a structure readable by sphinx search engine, in order to perform efficient searches (full-text, geo, etc.) on Anymeta data (see anymeta-sphinx).

Links

Usefull external links

Links to trac