Celery Integration with Flask and Redis

Flask is one of the most popular micro web framework written in Python. It is classified as a microframework because it does not require particular tools or libraries. It is designed to make getting started quick and easy, with the ability to scale up to complex applications. In this article, I will show a very basic flask set up with celery to create async tasks or schedules.

Installing

Create virtualenv, activate it and install flask and celery inside it using pip.

mkdir flaskdemo
cd flaskdemo
virtualenv venv --python=python3.6
source venv/bin/activate
pip install flask celery

Simple Example

Create a python file with an appropriate name. I will name this file “myflaskapp.py”. Now write the following code inside this file:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def home():
    return 'Welcome to my app!'

To run this application, type in the terminal:

FLASK_APP=myflaskapp.py flask run

Configure Celery

This is all the changes that are necessary to properly integrate Celery with Flask:

from celery import Celery

def make_celery(app):
    celery = Celery(
        app.import_name,
        backend=app.config['CELERY_RESULT_BACKEND'],
        broker=app.config['CELERY_BROKER_URL']
    )
    celery.conf.update(app.config)

    class ContextTask(celery.Task):
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)

    celery.Task = ContextTask
    return celery

The function, “make_celery” creates a new Celery object, configures it with the broker from the application config, updates the rest of the Celery config from the Flask config and then creates a subclass of the task that wraps the task execution in an application context.

Create Task

Create a task that adds two numbers together and returns the result. We have to configure Celery’s broker and backend to use Redis, create a celery application using the factor from above, and then use it to define the task. Check this Redis installation guide to install Redis in Ubuntu.

app = Flask(__name__)
app.config.update(
    CELERY_BROKER_URL='redis://localhost:6379',
    CELERY_RESULT_BACKEND='redis://localhost:6379'
)
celery = make_celery(app)

@celery.task()
def add_together(a, b):
    return a + b

The task, “add_together” can now be called in the background. I will call this task in the background when someone hits the homepage.

@app.route('/')
def home():
    result = add_together.delay(10, 20)
    print(result.wait())
    return 'Welcome to my app!'

Now, we need to run a Celery worker to receive and execute the task. Here, wait will return the result once the task is finished.

Run Celery Worker

Fire up the Flask server in one terminal and then start the celery worker in another terminal by typing:

FLASK_APP=myflaskapp.py flask run
celery -A myflaskapp.celery worker -l info

Open browser, visit the home page and we can see the task result in the console. Here, replace “myflaskapp” with your file name.

Create a Scheduler

Update the app config to define a scheduled task, named “run-every-1-minute”:

from datetime import timedelta

CELERYBEAT_SCHEDULE = {
        'run-every-1-minute': {
            'task': 'myflaskapp.print_hello',
            'schedule': timedelta(seconds=60)
        },
    }

Create the scheduled task which is similar to the task that we created earlier.

@celery.task()
def print_hello():
    print('Hello World!')

So, the scheduler will print “Hello World!” in the console for every 1 minute.

Run Celery Beat

To run this scheduler, we need to run Celery beat alongside with the Flask server and Celery worker. So, start the Flask server, Celery worker and then start Celery beat using this command in another terminal:

celery -A myflaskapp.celery beat -l info

If everything goes well, then we can see that the scheduler is running in every 5 minutes.

The full source code can be found in the Github repo for better understanding.

Visit Flask Documentation for detailed Flask documentation.
Visit Celery documentation for detailed configuration and optimization.