Alex Raichev

Adding Celery with RabbitMQ to a Django project on Webfaction

I recently added Celery with RabbitMQ to one of my Django projects on Webfaction. Along the way, i read lots of documentation, encountered difficulties, and got advice from the helpful staff at Webfaction. What worked for me in the end are the instructions below. May they might work for you too and save you some pain!

I'll assume that you've already added Celery with RabbitMQ to your Django project in your development environment (encapsulated within a virtual environment) and have that working as desired. If not, follow the Celery docs. By the way, the latest version of Celery, version 3.1.13, works with Django out of the box and no longer requires installing the additional django-celery Django app.

Now, let's add Celery with RabbitMQ to your production environment (encapsulated within a virtual environment) on Webfaction.

Preparation

  • Ssh into your Webfaction server. All the command line instructions below take place in that session.
  • Also log in to your Webfaction control panel on the web
  • When following the instructions below, replace text of the form <stuff> with text appropriate to your situation. For example, replace <username> by your Webfaction username.

Install and configure Erlang

  1. From your Webfaction control panel, create a new application with App category: Custom, App type: Custom app (listening on port). Note the port number that gets assigned to your app.
  2. Go to Erlang's download webpage and copy the link to the latest .tar.gz version of Erlang
  3. In your ssh session, download the latest version of Erlang (currently OTP 17.1): wget <the Erlang link you copied>
  4. Unzip it: gunzip -c otp_src_17.1.tar.gz | tar xf -
  5. Change to the directory created: cd otp_src_17.1
  6. Configure the build: ./configure --prefix=/home/<username>/
  7. Make it: make
  8. Install it: make install

Test Erlang

  1. Run Erlang on the port assigned to your Erlang app (noted in the first step): epmd -port <your Erlang port> -daemon
  2. Check that it is running by looking for an epmd process: ps aux | grep epmd

Install and configure RabbitMQ

  1. From your Webfaction control panel, create a new application with App category: Custom, App type: Custom app (listening on port). Note the port number that gets assigned to your app.

  2. Go to RabbitMQ's download webpage and copy the link to the latest binary .tar.gz version of RabbitMQ (currently 3.3.4)

  3. In your ssh session, download the latest binary version of RabbitMQ: wget <the RabbitMQ link you copied>

  4. Unzip it: gunzip -c rabbitmq-server-generic-unix-3.3.4.tar.gz | tar xf -

  5. Simlink RabbitMQ to the Erlang lib directory: cd ~/lib/erlang/lib/; ln -s ../src/rabbitmq_server-3.3.4 rabbitmq_server-3.3.4

  6. Edit $HOME/rabbitmq_server-3.3.4/sbin/rabbitmq-defaults:

    ...
    # comment these lines:
    #CONFIG_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq
    #LOG_BASE=${SYS_PREFIX}/var/log/rabbitmq
    #MNESIA_BASE=${SYS_PREFIX}/var/lib/rabbitmq/mnesia
    # add these lines:
    CONFIG_FILE=/home/<username>/rabbitmq_server-3.1.0/sbin/
    LOG_BASE=/home/<username>/logs/user/rabbitmq # create this directory!
    MNESIA_BASE=/home/<username>/rabbitmq_server-3.1.0/sbin/
    ...
    
  7. Create the RabbitMQ log directory: mkdir /home/<username>/logs/user/rabbitmq

  8. Edit $HOME/rabbitmq_server-3.3.4/sbin/rabbitmq-env:

    ...
    # add to the end
    export ERL_EPMD_PORT=<your Erlang port>
    export RABBITMQ_NODE_PORT=<your RabbitMQ port>
    export ERL_INETRC=$HOME/.erl_inetrc
    
  9. Create $HOME/hosts, replacing dweb125 with your Webfaction hostname prefix:

    127.0.0.1 developmenthost.developmentdomain developmenthost
    ::1      developmenthost6.developmentdomain6 developmenthost6
    127.0.0.1 dweb125 dweb125.webfaction.com
    
  10. Create $HOME/.erl_inetrc:

    {hosts_file, "/home/<username>/hosts"}.
    {lookup, [file,native]}.
    

Test RabbitMQ

  1. Make sure you have Erlang running. Start RabbitMQ via ./rabbitmq_server-3.3.4/sbin/rabbitmq-server -detached, then check on its status via ./rabbitmq_server-3.3.4/sbin/rabbitmqctl status, and look for a line with {amqp,<your RabbitMQ port>,"::"}.

Install and configure Celery

  1. Within your ssh session, activate the virtual environment for your Django project and install Celery via pip install celery.

  2. Assuming you've already configured Celery to work in your development environment, all that's left to do is add the following line to the production version of your Django settings file:

    BROKER_URL = 'amqp://guest:guest@localhost:<your RabbitMQ port>//'
    

Test Celery

  1. Within your ssh session and Django project directory, run celery -A <your Django project> worker -l info

That's it. Fyew! You now have Erlang, RabbitMQ, and Celery installed and running on Webfaction.

Todo

I've heard that controlling Erlang + RabbitMQ + Celery with Supervisor is a good idea. Will look into that.

Author: Alex Raichev
Date: 2014-08-02
Comment