Huey - a nice celery alternative



I am going to show a simple example of huey usage in django project.
The problem that huey should solve in this example - handle some hard tasks that should be done when user navigate to particular url.
The 'hard task' in example would be time.sleep(10) :).
To make sure that task was successfully done, we will calculate + 1 day and save result to file.

Installation and configuration

Huey is available on PyPI:

pip install huey

Also we'll need redis:

pip install redis




HUEY = {
    'backend': 'huey.backends.redis_backend',  # required.
    'name': 'hueytest',
    'connection': {'host': 'localhost', 'port': 6379},
    'always_eager': False, # Defaults to False when running via run_huey

    # Options to pass into the consumer when running `` run_huey``
    'consumer_options': {'workers': 4},

See more about huey settings in it's documentation.


import datetime
import os
import time

from huey.djhuey import crontab, periodic_task, task

def hard_task():
    now =
    path = os.path.join(
        'time_%s.txt' % datetime.datetime.strftime(now, '%Y%d%m_%H%M%S'))
    with open(path, 'w') as f:
        f.write('Task done.')


from django.shortcuts import render_to_response, redirect

from .tasks import hard_task

def home(request):
    return render_to_response('home.html')

def hardview(request):
    return redirect('home')

Running consumers:

python run_huey

I got an error: No handlers could be found for logger "huey.consumer":, but this can be easily fixed:

# myproject/
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        'syslog': {
            'address': '/dev/log',
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
       'huey.consumer': {
            'handlers': ['syslog'],
            'level': 'INFO',
            'propagate': True,

Logs (see /var/logs/syslog):

Jul 28 22:15:26 nanvel-ws Setting signal handler
Jul 28 22:15:26 nanvel-ws Huey consumer initialized with following commands#012+ hard_task
Jul 28 22:15:26 nanvel-ws 4 worker threads
Jul 28 22:15:26 nanvel-ws Starting scheduler thread
Jul 28 22:15:26 nanvel-ws Starting worker threads
Jul 28 22:15:26 nanvel-ws Starting periodic task scheduler thread
Jul 28 22:15:43 nanvel-ws Executing <hueytest.testapp.tasks.queuecmd_hard_task object at 0x1c96250>

If you'll open reverse('hardview'), huew task should be created and file similar to myapp/time_20132807_142719.txt should be created.

Running huey consumers on production

Author advised us to use supervisor:

A well-behaved Unix daemon process is tricky to get right

For code clarity, this is not provided as a part of 'huey'. Adding this would require a dependency outside the standard lib, which I'm not > really interested in adding. If you want, though, you should be able to very easily use:
I run all my web apps and consumers using supervisord.

So, let's install and configure it.

sudo pip install supervisor
echo_supervisord_conf > supervisord.conf
vim supervisord.conf

Add this to supervisord.conf:

command=/home/deploy/envs/hueytest/.env/bin/python run_huey
sudo ln -s /usr/local/bin/supervisord /usr/bin/supervisord
sudo ln -s /usr/local/bin/supervisorctl /usr/bin/supervisorctl
sudo mv debian-norrgard /etc/init.d/supervisord
sudo chmod +x /etc/init.d/supervisord
sudo update-rc.d supervisord defaults
sudo service supervisord start

Check is all ok:

sudo supervisorctl
huey                             RUNNING    pid 31875, uptime 0:00:24
supervisor> exit
ps aux | grep huey
deploy   30451  2.1  0.5 520764 22836 ?        Sl   22:23   0:00 /home/deploy/envs/hueytest/.env/bin/python run_huey
deploy   31248  0.0  0.0  13584   920 pts/4    S+   22:24   0:00 grep --color=auto huey

Stopping process:

supervisor> stop huey
huey: stopped
supervisor> exit

more commands:

supervisor> help

default commands (type help <topic>):
add    clear  fg        open  quit    remove  restart   start   stop  update
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

Huey has next useful features:

Read more in documentation.


Licensed under CC BY-SA 3.0