Build a Django one-click-app on DigitalOcean and scale it with Memcache

This tutorial will walk you through the steps of creating a simple Django One-Click application on DigitalOcean and then add Memcache to prevent or alleviate a performance bottleneck.

We’ll walk you through creating the application from start to finish, but you can view the finished product source code here.

Adding caching to your web applications can drastically improve performance. The results of complex database queries, expensive calculations, or slow calls to external resources can be stored in Memcache that can be accessed via fast O(1) lookups. Even for small sites, Memcache can make page loads snappy and help future-proof your app.

Prerequisites

Before you complete the steps in this guide, make sure you have all of the following:

  • Familiarity with Python (and ideally Django)
  • A DigitalOcean account.
  • If you like managing DigitalOcean resource via the CLI, you need the doctl installed and configured.

Create a Django One-Click application

To build an app we first need a droplet. Either go to your DigitalOcean dashboard and create one or launch one via the CLI:

Give the droplet a minute to come up and then look up its IP via the dashboard or by typing

Now you can login to your droplet via

and visit the page at http://<DROPLET_IP>/.

Configure environment

Before we start adding functionality, let’s create a file for environment variables for our application.

Login to your droplet and change to the django user:

[root]$ sudo -i -u django

Note: Ideally you open two terminals to log into your droplet. One stays in root for system related configurations and actions, and one uses the django user to develop the app.

The application is located in the django_project directory:

If you want, you can change the name of the folder but then you will need to also change the paths referenced within /etc/systemd/system/gunicorn.service and /etc/nginx/sites-enabled/django.

Let’s create a .env file to host our environment variables that is currently empty:

In order for our application to pick up the environment variables in our .env file, we need to reference it from the systemd unit file that starts the django app. As root edit /etc/systemd/system/gunicorn.service as follows:

[Unit]
# ...

[Service]
# ...
EnvironmentFile=/home/django/django_project/.env

[Install]
# ...

Note: your droplet comes with nano or vim preinstalled. If you have never used an editor in a terminal before, I recommend you use nano for now.

To take effect, have systemd reload the unit files (as root):

Add task list functionality

The Django application we are building is a task list. In addition to displaying the list, it will have actions to add new tasks and to remove them. To accomplish this, we need to:

  1. Create a task list app
  2. Create a Task model
  3. Create the route, view, and controller logic

Note that the Django One-click-app comes with PostgreSQL pre-configured so you don’t need to worry about setting up a database.

Create a task list app

Django has the concept of apps and we need to create one in order to add any functionality. We will create a mc_tasklist app:

Add mc_tasklist to the list of installed apps in django_project/settings.py:

Create the Task model

To create and store tasks, we need to do two things:

  1. Create a simple Task model in mc_tasklist/models.py:

  2. Use makemigrations and migrate to create a migration for the mc_tasklist app as well as create the mc_tasklist_task table, along with all other default Django tables:

Create the task list application

The actual application consists of a view that is displayed in the front end and a controller that implements the functionality in the back end. You also need to tell Django which controller corresponds to which URL.

  1. Setup the routes for add, remove, and index methods in django_project/urls.py:

  2. Add corresponding view controllers in mc_tasklist/views.py:

  3. Create a template with display code in mc_tasklist/templates/index.html:

    <!DOCTYPE html>
    <head>
      <meta charset="utf-8">
      <title>MemCachier Django tutorial</title>
      <!-- Fonts -->
      <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.4.0/css/font-awesome.min.css"
            rel='stylesheet' type='text/css' />
      <!-- Bootstrap CSS -->
      <link href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css"
            rel="stylesheet" />
    </head>
    
    <body>
      <div class="container">
        <!-- New Task Card -->
        <div class="card">
          <div class="card-body">
            <h5 class="card-title">New Task</h5>
    
            <form action="add" method="POST">
              {% csrf_token %}
              <div class="form-group">
                <input type="text" class="form-control" placeholder="Task Name"
                       name="name" required>
              </div>
              <button type="submit" class="btn btn-default">
                <i class="fa fa-plus"></i> Add Task
              </button>
            </form>
          </div>
        </div>
    
        <!-- Current Tasks -->
        {% if tasks %}
        <div class="card">
          <div class="card-body">
            <h5 class="card-title">Current Tasks</h5>
    
            <table class="table table-striped">
              {% for task in tasks %}
              <tr>
                <!-- Task Name -->
                <td class="table-text">{{ task.name }}</td>
                <!-- Delete Button -->
                <td>
                  <form action="remove" method="POST">
                    {% csrf_token %}
                    <input type="hidden" name="id" value="{{ task.id }}">
                    <button type="submit" class="btn btn-danger">
                      <i class="fa fa-trash"></i> Delete
                    </button>
                  </form>
                </td>
              </tr>
              {% endfor %}
            </table>
          </div>
        </div>
        {% endif %}
      </div>
    
      <!-- Bootstrap related JavaScript -->
      <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js"></script>
      <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js"></script>
      <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"></script>
    </body>
    </html>

    The view consists of two cards: one that contains a form to create new tasks, and another that contains a table with existing tasks and a delete button associated with each task.

    Note that Django will automatically check each apps templates folder for templates.

Our task list is now functional. Restart your app (as root):

We now have a functioning task list running on DigitalOcean. Visit it at http://<DROPLET_IP>/ and add a few tasks. With this complete, we can learn how to improve its performance with Memcache.

Add caching to Django

Memcache is an in-memory, distributed cache. Its primary API consists of two operations: SET(key, value) and GET(key). Memcache is like a hashmap (or dictionary) that is spread across multiple servers, where operations are still performed in constant time.

The most common use for Memcache is to cache the results of expensive database queries and HTML renders so that these expensive operations don’t need to happen over and over again.

Provision a Memcache

To use Memcache in Django, you first need to provision an actual Memcached cache. You can easily get one for free from MemCachier. This allows you to just use a cache without having to setup and maintain actual Memcached servers yourself. Make sure to create the cache in the same region as your droplet is in.

There are three config variables you’ll need for your application to be able to connect to your cache: MEMCACHIER_SERVERS, MEMCACHIER_USERNAME, and MEMCACHIER_PASSWORD. Get them from your MemCachier dashboard and add them to your /home/django/django_project/.env file:

MEMCACHIER_USERNAME=<YOUR_USERNAME>
MEMCACHIER_PASSWORD=<YOUR_PASSWORD>
MEMCACHIER_SERVERS=<YOUR_SERVERS>

Configure Django with MemCachier

Django requires pylibmc in order to connect the Memcache server:

As of Django 1.11 we can use its native pylibmc backend. For older versions of Django you will need to install django-pylibmc.

Configure your cache by adding the following to the end of django_project/settings.py:

This configures the cache for both development and production. If the MEMCACHIER_* environment variables exist, the cache will be setup with pylibmc, connecting to MemCachier. Whereas, if the MEMCACHIER_* environment variables don’t exist – hence development mode – Django’s simple in-memory cache is used instead.

Cache expensive database queries

Memcache is often used to cache expensive database queries. This simple example doesn’t include any expensive queries, but for the sake of learning, let’s assume that getting all tasks from the database is an expensive operation.

The task list database query code in mc_tasklist/views.py can be modified to check the cache first like so:

The above code first checks the cache to see if the tasks.all key exists in the cache. If it does not, a database query is executed and the cache is updated. Subsequent pageloads will not need to perform the database query. The time.sleep(2) only exists to simulate a slow query.

Reload the app and test the new functionality.

If you get the error FAILED TO SEND AUTHENTICATION TO SERVER, no mechanism available, check your pylibmc version. Version 1.6.0 comes with a buggy libmemcached. To circumvent the problem do the following:

To see what’s going on in your cache, open the MemCachier dashboard for your cache. The first time you loaded your task list, you should have gotten an increase for the get miss and set commands. Every subsequent reload of the task list should increase get hits (refresh the stats in the dashboard).

Our cache is working, but there is still a major problem. Add a new task and see what happens. No new task appears on the current tasks list! The new task was created in the database, but the app is serving the stale task list from the cache.

Clear stale data

There are many techniques for dealing with an out-of-date cache.

  1. Expiration: The easiest way to make sure the cache does not get stale is by setting an expiration time. The cache.set method can take an optional third argument, which is the time in seconds that the cache key should stay in the cache. If this option is not specified, the default TIMEOUT value in settings.py will be used instead.

    You could modify the cache.set method to look like this:

    But this functionality only works when it is known for how long the cached value is valid. In our case however, the cache gets stale upon user interaction (add, remove a task).

  2. Delete cached value: A straight forward strategy is to invalidate the tasks.all key when you know the cache is out of date – namely, to modify the add and remove views to delete the tasks.all key:

  3. Key based expiration: Another technique to invalidate stale data is to change the key:

    The upside of key based expiration is that you do not have to interact with the cache to expire the value. The LRU eviction of Memcache will clean out the old keys eventually.

  4. Update cache: Instead of invalidating the key, the value can also be updated to reflect the new task list:

    Updating the value instead of deleting it will allow the first pageload to avoid having to go to the database

You can use option 2, 3, or 4 to make sure the cache will not ever be out-of-date. As usual, reload the app afterwards.

Now when you add a new task, all the tasks you’ve added since implementing caching will appear.

Use Django’s integrated caching

Django also has a few built in ways to use your Memcache to improve performance. These mainly target the rendering of HTML which is an expensive operation that is taxing for the CPU.

Caching and CSRF

You cannot cache any views or fragments that contain forms with CSRF tokens because the token changes with each request. For the sake of learning how to use Django’s integrated caching we will disable Django’s CSRF middleware. Since this task list is public, this is not a big deal but do not do this in any serious production application.

Comment CsrfViewMiddleware in django_project/settings.py:

Cache template fragments

Django allows you to cache rendered template fragments. This is similar to snippet caching in Flask, or caching rendered partials in Laravel. To enable fragment caching add {% load cache %} to the top of your template.

Do not cache fragments that include forms with CSRF tokens.

To cache a rendered set of task entries, we use a {% cache timeout key %} statement in mc_tasklist/templates/index.html:

Here the timeout is None and the key is a list of strings that will be concatenated. As long as task IDs are never reused, this is all there is to caching rendered snippets. The PostgreSQL database does not reuse IDs, so we’re all set.

If you use a database that does reuse IDs, you need to delete the fragment when its respective task is deleted. You can do this by adding the following code to the task deletion logic:

Let’s see the effect of caching the fragments in our application. You should now observe an additional get hit for each task in your list whenever you reload the page (except the first reload).

Cache entire views

We can go one step further and cache entire views instead of fragments. This should be done with care, because it can result in unintended side effects if a view frequently changes or contains forms for user input. In our task list example, both of these conditions are true because the task list changes each time a task is added or deleted, and the view contains forms to add and delete a task.

Do not cache views that include forms with CSRF tokens.

You can cache the task list view with the @cache_page(timeout) decorator in mc_tasklist/views.py:

Because the view changes whenever we add or remove a task, we need to delete the cached view whenever this happens. This is not straight forward. We need to learn the key when the view is cached in order to be then able to delete it:

To see the effect of view caching, reload your application. On the first refresh, you should see the get hit counter increase according to the number of tasks you have, as well as an additional get miss and set, which correspond to the view that is now cached. Any subsequent reload will increase the get hit counter by just two, because the entire view is retrieved with two get commands.

Note that view caching does not obsolete the caching of expensive operations or template fragments. It is good practice to cache smaller operations within cached larger operations, or smaller fragments within larger fragments. This technique (called Russian doll caching) helps with performance if a larger operation, fragment, or view is removed from the cache, because the building blocks do not have to be recreated from scratch.

Using Memcache for session storage

Memcache works well for storing information for short-lived sessions that time out. However, because Memcache is a cache and therefore not persistent, long-lived sessions are better suited to permanent storage options, such as your database.

For short-lived sessions configure SESSION_ENGINE to use the cache backend in django_project/settings.py:

For long-lived sessions, Django allows you to use a write-through cache, backed by a database. This is the best option for performance while guaranteeing persistence. To use the write-through cache, configure the SESSION_ENGINE in django_project/settings.py like so:

For more information on how to use sessions in Django, please see the Django Session Documentation

Further reading and resources