Resque Queue Priority

Resque allows each worker process to work a prioritized list of work queues. When jobs are added, they end up in one particular queue. Each worker scans the queues in that priority order to determine the next job to process. This allows you to ensure that higher priority work is processed before lower priority work.

PatientsLikeMe

At PatientsLikeMe we leverage two different priority lists to allocate 80% of the workers to a “Responsive” strategy, where time-sensitive or business critical jobs get processed before necessary, but deferrable scheduled or maintenance work. The other 20% reverses this list to process scheduled work before responsive work.

Every worker can process every type of job, so if there is a large amount of just one job type – the entire cluster is devoted to those jobs. However, when there is a large volume of jobs and the cluster is at capacity, 80% of the workers will do “Responsive” jobs and 20% will do scheduled jobs.

Confusion about Priority Queues

I presented this as part of a larger talk at Boston Devops last night and several members of the audience were confused about how a Resque worker selects it’s next job. Looking at the Resque Worker#reserve source code (which implements job selection) you can see the queue list is walked in priority order and the first queue with a job to process is reserved for execution.

However, since 2-3 people seemed strongly convinced this was not the case, I decided to do a quick experiment and see for myself

Priority Experiment

I created two Worker types (Alpha/Beta) and two workers running QUEUE=alpha,beta and QUEUE=beta,alpha. I then created a large workload (100 Beta jobs) and let both workers pick-up work from the beta queue. Then I introduced an Alpha job in the Alpha queue. The first worker (QUEUE=alpha,beta) then picked up that job and executed it, shortly after it was enqueued.

Full code/results are available in this gist.

$> rake resque:work QUEUE=alpha,beta
Beta
Beta
Beta
Alpha
Beta
Beta

$> rake resque:work QUEUE=beta,alpha
Beta
Beta
Beta
Beta
Beta
Beta

Conclusion (TL;DR)

Resque workers process work in the priority order specified. Higher priority work will not interrupt the currently running job, but will be the next job processed by a worker (even with a high volume of lower priority work).