Celery is an asynchronous task queue based on distributed message passing. Task queues are used as a strategy to distribute the workload between threads/machines. In this tutorial I will explain how to install and setup Celery + RabbitMQ to execute asynchronous in a Django application.
To work with Celery, we also need to install RabbitMQ because Celery requires an external solution to send and receive messages. Those solutions are called message brokers. Currently, Celery supports RabbitMQ, Redis, and Amazon SQS as message broker solutions.
Table of Contents
- Why Should I Use Celery?
- Celery Basic Setup
- Creating Our First Celery Task
- Starting The Worker Process
- Managing The Worker Process in Production with Supervisord
- Further Reading
Why Should I Use Celery?
Web applications works with request and response cycles. When the user access a certain URL of your application the Web browser send a request to your server. Django receive this request and do something with it. Usually it involves executing queries in the database, processing data. While Django does his thing and process the request, the user have to wait. When Django finalize its job processing the request, it sends back a response to the user who finally will see something.
Ideally this request and response cycle should be fast, otherwise we would leave the user waiting for way too long. And even worse, our Web server can only serve a certain number of users at a time. So, if this process is slow, it can limit the amount of pages your application can serve at a time.
For the most part we can work around this issue using cache, optimizing database queries, and so on. But there are some cases that theres no other option: the heavy work have to be done. A report page, export of big amount of data, video/image processing are a few examples of cases where you may want to use Celery.
We don’t use Celery through the whole project, but only for specific tasks that are time-consuming. The idea here is to respond to the user as quick as possible, and pass the time-consuming tasks to the queue so to be executed in the background, and always keep the server ready to respond to new requests.
The easiest way to install Celery is using pip:
Now we have to install RabbitMQ.
Installing RabbitMQ on Ubuntu 16.04
To install it on a newer Ubuntu version is very straightforward:
Then enable and start the RabbitMQ service:
Check the status to make sure everything is running smooth:
Installing RabbitMQ on Mac
Homebrew is the most straightforward option:
The RabbitMQ scripts are installed into
/usr/local/sbin. You can add it to your
Then add it to the bottom of the file:
Restart the terminal to make sure the changes are in effect.
Now you can start the RabbitMQ server using the following command:
Celery Basic Setup
First, consider the following Django project named mysite with an app named core:
CELERY_BROKER_URL configuration to the settings.py file:
Alongside with the settings.py and urls.py files, let’s create a new file named celery.py.
Now edit the __init__.py file in the project root:
This will make sure our Celery app is important every time Django starts.
Creating Our First Celery Task
We can create a file named tasks.py inside a Django app and put all our Celery tasks into this file. The Celery app
we created in the project root will collect all tasks defined across all Django apps listed in the
Just for testing purpose, let’s create a Celery task that generates a number of random User accounts.
The important bits here are:
Then I defined a form and a view to process my Celery task:
This form expects a positive integer field between 50 and 500. It looks like this:
Then my view:
The important bit is here:
Instead of calling the
create_random_user_accounts directly, I’m calling
way we are instructing Celery to execute this function in the background.
Then Django keep processing my view
GenerateRandomUserView and returns smoothly to the user.
But before you try it, check the next section to learn how to start the Celery worker process.
Starting The Worker Process
Open a new terminal tab, and run the following command:
Change mysite to the name of your project. The result is something like this:
Now we can test it. I submitted 500 in my form to create 500 random users.
The response is immediate:
Meanwhile, checking the Celery Worker Process:
Then after a few seconds, if we refresh the page, the users are there:
If we check the Celery Worker Process again, we can see it completed the execution:
Managing The Worker Process in Production with Supervisord
If you are deploying your application to a VPS like DigitalOcean you will want to run the worker process in the background. In my tutorials I like to use Supervisord to manage the Gunicorn workers, so it’s usually a nice fit with Celery.
First install it (on Ubuntu):
Then create a file named mysite-celery.conf in the folder: /etc/supervisor/conf.d/mysite-celery.conf:
In the example below, I’m considering my Django project is inside a virtual environment. The path to my virtual environment is /home/mysite/.
Now reread the configuration and add the new process:
sudo supervisorctl reread sudo supervisorctl update