Django Chatbot with Celery, WebSockets, and Redis.

Sergey Lyapustin

Published in

ITNEXT

5 min readJul 9, 2019

Let’s create a Chatbot which can execute tasks asynchronously using Celery and communicate via WebSockets.

TL;DR.

Here a Github repo, just click Deploy to Heroku.

Django

We will use Django as a base for our application:

HTTP requests handler
App configuration
Template engine

Redis

We need to use Redis in a couple of places:

Cache Storage
Celery Message Broker and Result Backend
Channel layer for the WebSocket communication

We will use Heroku for deployment, so we don’t need to install and configure Redis manually. Redis DSN will be available as an environment variable REDIS_URL so we can use it everywhere we need it.

Redis as a Cache Storage

Django does not support Redis internally, so we need to use the extra package. We are going to usedjango-redis. It’s full-featured Redis cache backend for Django.

pip install django-redis

This will install a couple more dependencies, including redis-py — Python interface to the Redis.

We only need to update our Django project configuration with the CACHES settings.

Celery with Redis as a Message Broker

We will have some tasks which may take a while. For example, getting a response from the remote server. So we need to do them in the background and send the result back to the client when it will be available.

Configuration for Celery is pretty simple, we are going to reuse our REDIS_URL for the CeleryBROKER_URL and RESULT_BACKEND.

Let’s define our Celery instance inside project/celery.py:

And we need to import Celery instance in our project, to ensure the app is loaded when Django starts.

So now our Celery is ready and we can add background tasks to our project. For example this one:

WebSockets with Django Channels

We want to send a result from to the client immediately as it is available. Without querying it every couple of seconds via scheduled AJAX calls.

WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection. So user can send data to the backend and backend can send data to the user.

Django Channels are the way to go if you need to have WebSockets. There two versions available now 1.x and 2.x. They have major changes and upgrade from 1.x to 2.x can be a big project if you have a fairly big application. Version 1.x is no longer supported, so for new projects, you should use the 2.x branch.

Django-Channels has really good documentation available on the official website, so I encourage you to check.

Key moments are — we need to change couple of things in our application to support WebSockets.

Daphne

In order to use WebSockets we need to have a web server, which supports it. So we need to replace Gunicorn which we usually use to serve Django-powered websites with the Daphne.

Daphne is an HTTP, HTTP2 and WebSocket protocol server for ASGI and ASGI-HTTP, developed to power Django Channels.

There is nothing to configure for our application, we can just run it the same way as we run WSGI application, but instead of WSGI we need to point it to the ASGI application:

daphne project.asgi:application

ASGI configuration looks the same as WSGI but points to the Django Channels application instead:

Django-Channels and Redis

Django channels are some kind of communication system, which allow multiple consumer instances to talk with each other, and with other parts of Django.

A channel layer provides the following abstractions:

A channel is a mailbox there messages can be sent to. Each channel has a name. Anyone who has the name of a channel can send a message to the channel.
A group is a group of related channels. A group has a name. Anyone who has the name of a group can add/remove a channel to the group by name and send a message to all channels in the group.

Every consumer instance has an automatically generated unique channel name, and so can be communicated via a channel layer.

So let’s define CHANNEL_LAYERS for that project. We are going to use channels_redis for that. So we need to define the backend and provide Redis DSN:

WebSocket Consumers and Routing

Now we need to define how we will interact with our clients — frontend JS calls. So we will receive every message (as a JSON string) from the client and do something with it.

Here is how it may look:

And as our application can have multiple consumers (same way as we may have multiple Django views) we need to have URL routing similar to the Django urls.py file:

So when the client sent a message to the WebSocket /ws/chat/ path — it will be processed by our ChatConsumer.

And finally, we need to include our application URLs to the main router — project/routing.py:

# project/routing.py
from channels.routing import ProtocolTypeRouter, URLRouter
import chat.routing application = ProtocolTypeRouter({    
    # (http->Django views is added by default)    
    'websocket':    
        URLRouter(
            chat.routing.websocket_urlpatterns
        ),
})

And point to it from our project/settings.py file:

ASGI_APPLICATION = "project.routing.application"

Deployment to Heroku

I will use Heroku for deployment as it’s the quickest way to deploy everything in one step. And also — it’s free (they required an account verification though).

Procfile

That file contains processes types definition:

Here we define what we need to do during the application release phase and configure two application instances:

web — Daphne server which will listen on $PORT and handle our https:// and wss:// requests.
worker — Celery worker instance to process our asynchronous task queue.

Heroku app.json manifest

This is the file that will be used to create a new Heroku website instance from scratch.

Environment variables we need to be set for our app (like Django SECRET_KEY value)
Which services do we need to have (PostgreSQL, Redis)
Which type of the virtual machine do we want to have heroku/python.

That file can also be used to create a nice Deploy to Heroku button. So it can be deployed in one click next time.