Deploy Django and Memcache on DigitalOcean App Platform

Paddy ~ June 22, 2023

Tags: DigitalOcean, App Platform, Django, Tutorial

Django is a popular, high-level Python web framework that encourages rapid development and clean, pragmatic design. DigitalOcean’s App Platform is a Platform as a Service (PaaS) product to configure and deploy applications from a code repository. It offers a fast and effective way to deploy your Django app. In this tutorial, you’ll deploy a Django application to DigitalOcean App Platform and then make it more scalable by adding caching with the DigitalOcean Marketplace Add-On for MemCachier. MemCachier is compliant with the memcached object caching system but has several advantages, such as never touching servers, daemons, or configuration files.

You’ll first build a Django task list app. Then, you’ll push your app’s code to GitHub and deploy it on App Platform. Finally, you’ll implement three object caching techniques to make your app faster and more scalable. By the end of this tutorial, you’ll be able to deploy a Django application to App Platform and implement techniques for caching resource-intensive database queries, template fragments, views, and session storage.

Outline

• Initialize a Django project
• Create a Django task list app
• Prepare your app for App Platform
• Deploy Django to App Platform
• Set up caching in Django
• Implement caching strategies in Django

Prerequisites

• Familiarity with Python and, ideally, Django.
• A DigitalOcean account for deploying to App Platform. Running this app on App Platform will incur a charge. See App Platform Pricing for details.
• A GitHub account and Git installed on your local machine. To deploy to App Platform, you must push code to a remote code repository. App Platform supports GitHub, GitLab, Docker, and more.
• Python (version >= 3.8) installed on your computer. I’ll use 3.11.1.

Initialize a Django project

The following commands will create an isolated Python virtual environment and bootstrap a starter Django project.

Create a new folder for the project and change into that directory:

mkdir django-memcache && cd django-memcache

Make sure your terminal window is using at least Python 3.8. 3.8 is the minimum required version for Django 4.2, which you’ll use in this tutorial. Then, create a new Python virtual environment.

python -m venv venv

When you run this command, Python creates a new directory called venv. venv allows you to install additional packages and dependencies into this virtual environment without affecting your system’s global Python installation. -m venv tells Python to use the venv module to create a new virtual environment. The final venv is the directory’s name where the virtual environment will be created. You can choose any name, but venv is a common convention.

Next, activate the Python virtual environment:

source venv/bin/activate

After running that command, you should see the name of the virtual environment displayed in your terminal prompt, e.g., (venv) ➜ django-memcache.

Install the latest version of Django (4.2 at the time of writing) using the pip package installer:

(venv) $ python -m pip install Django

Next, create a new Django project:

(venv) $ django-admin startproject myproject .

This command creates a new Django project named myproject in the current directory (.). django-admin is Django’s command-line utility for administrative tasks.

Next, start the Django development server:

(venv) $ python manage.py runserver

manage.py does the same thing as django-admin but also sets the DJANGO_SETTINGS_MODULE environment variable so that it points to your project’s settings.py file. That tells Django which settings to use for your project.

# Output
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
June 21, 2023 - 07:38:46
Django version 4.2.2, using settings 'myproject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Visit http://127.0.0.1:8000/ in your browser. You’ll see a Congratulations! page, with a rocket taking off.

Create a Django task list app

The Django application we are building is a task manager. In addition to displaying a list of tasks, it will have actions to add new tasks and to remove them. To accomplish this, we need to:

1. Create a Django app.
2. Create a Task model.
3. Run database migrations locally.
4. Add the route, view, and controller logic for adding, removing, viewing a task, and viewing all tasks.

Create a Django app

Django has the concept of apps, and we must create one to add functionality. Create an app named myapp:

(venv) $ python manage.py startapp myapp

Add myapp to the list of installed apps in myproject/settings.py:

# myproject/settings.py
# ...

INSTALLED_APPS = [
    'myapp.apps.MyappConfig',
    # ...
]

# ...

See the Configuring applications Django documentation for a detailed explanation of this configuration.

Create a Task model

Create a Task model in myapp/models.py:

# myapp/models.py
from django.db import models

class Task(models.Model):
    name = models.TextField()
    notes = models.TextField()

A Task has a name, and can have some notes, which we’ll make optional.

Next, create a migration for the myapp app:

(venv) $ python manage.py makemigrations myapp

# Output
Migrations for 'myapp':
  myapp/migrations/0001_initial.py
    - Create model Task

Run database migrations locally

Run database migrations locally to create the myapp_task table, along with all other default Django tables:

(venv) $ python manage.py migrate

# Output
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, myapp, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying myapp.0001_initial... OK
  Applying sessions.0001_initial... OK

With the local database setup complete, you’re ready to add some functionality to your app.

Add views

Next, you’ll create the following four views:

• index: display all tasks.
• detail: view a single task.
• add: add a task.
• remove: remove a task.

In myapp/views.py, replace the contents of the file with the following:

# myapp/views.py
from django.template.context_processors import csrf
from django.shortcuts import render, redirect
from myapp.models import Task

def index(request):
    tasks = Task.objects.order_by('id')
    context = {'tasks': tasks}
    return render(request, 'index.html', context)

def detail(request, task_id):
    task = Task.objects.get(id=task_id)
    context = {'task': task}
    return render(request, 'detail.html', context)

def add(request):
    if 'name' in request.POST:
        task = Task(name=request.POST['name'], notes=request.POST['notes'])
        task.save()
        return redirect('/')
    return render(request, 'add.html')

def remove(request):
    task = Task.objects.get(id=request.POST['id'])
    if task:
        task.delete()
    return redirect('/')

• The index view gets all tasks from the database and passes them as the template’s context.
• The detail view gets a task from the database and passes it as the template’s context.
• The add view checks if a POST value named name is in the request. If so, a task is created and saved to the database, after which a redirect to the index view happens. Otherwise, the add view is rendered.
• The remove view gets a task from the database by ID. If the item exists, it is deleted. Then, a redirect to the index view happens.

The render() function takes the request object as its first argument, a template name as its second argument, and a dictionary as its optional third argument. You will create the corresponding templates next.

Create the index.html template

Now, create the template to display all tasks when the index view is requested. Create the file myapp/templates/index.html and add the following markup:

<!-- myapp/templates/index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>MemCachier Django tutorial</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>
  <body>
    <a href="/add">Add a Task</a>

    <h1>Tasks</h1>
    {% if tasks %}
    <ul>
      {% for task in tasks %}
      <li>
        <a href="/detail/{{ task.id }}">{{ task.name }}</a>
      </li>
      {% endfor %}
    </ul>
    {% else %}
      <a href="/add">Add your first task</a>
    {% endif %}
  </body>
</html>

The template contains a link to the add task view and lists existing tasks. A link to Add your first task is displayed if no tasks exist.

Create the detail.html template

Next, add a template for the detail view. Create the file myapp/templates/detail.html and add the following markup:

<!-- myapp/templates/detail.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>MemCachier Django tutorial | Task View</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>
  <body>
    <a href="/">Back to Tasks</a>
    <h1>{{ task.name }}</h1>
    <p>{{ task.notes }}</p>
    <form action="/remove" method="POST">
      {% csrf_token %}
      <input type="hidden" name="id" value="{{ task.id }}" />
      <button>Delete Task</button>
    </form>
  </body>
</html>

This template displays the name and notes for a task and a form with a delete button for removing a task.

Create the add.html template

Finally, create the add view. Create the file myapp/templates/add.html and add the following markup:

<!-- myapp/templates/add.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>MemCachier Django tutorial | Add a Task</title>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
  </head>
  <body>
    <a href="/">Back to Tasks</a>
    <h1>New Task</h1>
    <form action="/add" method="POST">
      {% csrf_token %}
      <div>
        <label>
          Task Name:
          <input
            type="text"
            name="name"
            placeholder="e.g. Wash the car"
            required
          />
        </label>
      </div>
      <div>
        <label>
          Task Notes:
          <textarea name="notes"></textarea>
        </label>
      </div>
      <button>Add Task</button>
    </form>
  </body>
</html>

This template contains a task creation form, with input fields for task name and optional notes.

Route views to URLs

To call these views, we need to map them to URLs by adding the following in myproject/urls.py:

# myproject/urls.py
# ...
from myapp import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', views.index),
    path('detail/<int:task_id>', views.detail),
    path('add', views.add),
    path('remove', views.remove),
]

Our task list is now functional. With the Django development server running, python manage.py runserver, visit http://127.0.0.1:8000/ in your browser and try it out. You should be able to add and delete tasks, to view all tasks, and to view a single task.

Prepare your app for App Platform

To prepare your Django app for production, you’ll do the following:

• Install gunicorn
• Update project settings to use environment variables
• Configure PostgreSQL for App Platform
• Create a remote code repository for your app

Install gunicorn

Gunicorn is a Python WSGI HTTP Server that will serve your app. Install the gunicorn package:

(venv) $ pip install gunicorn

Later, you’ll specify a gunicorn run command on App Platform.

Update project settings to use environment variables

In this step, you’ll update the following project settings to enable your Django application to run on App Platform: SECRET_KEY, ALLOWED_HOSTS, and DEBUG.

Secret key

Django’s SECRET_KEY plays a vital role in the security of your app. It is used for generating cryptographic signatures and should be kept confidential.

By default, a new secret key is automatically generated for you in your project’s settings.py file. However, this default configuration is not suitable for production for a few reasons:

Risk of Exposure: If the key is accidentally committed to a public repository, it can be a significant security issue.

Lack of Flexibility: The secret key should ideally vary between different environments (development, production, staging, etc.).

Open up myproject/settings.py in your code editor, import get_random_secret_key, and update the SECRET_KEY value as follows:

# myproject/settings.py
# ...
import os
from django.core.management.utils import get_random_secret_key
# ...
SECRET_KEY = os.getenv("DJANGO_SECRET_KEY", get_random_secret_key())
# ...

This change uses an environment variable to specify the secret key. If the environment variable is not found, the get_random_secret_key() function generates a new, random secret key, adding an extra layer of security.

Debug

Next, update the DEBUG setting to be controllable with an environment variable.

# myproject/settings.py
# ...
DEBUG = os.getenv("DEBUG", "False") == "True"
# ...

This change defaults DEBUG to False for safety.

Allowed hosts

The ALLOWED_HOSTS setting is a security measure to prevent HTTP Host header attacks. You’ll set ALLOWED_HOSTS to an environment variable, allowing you to use the custom URL that App Platform supplies.

# myproject/settings.py
# ...
ALLOWED_HOSTS = os.getenv("DJANGO_ALLOWED_HOSTS", "127.0.0.1,localhost").split(",")
# ...

This change sets 127.0.0.1,localhost as the ALLOWED_HOSTS default for local development if the DJANGO_ALLOWED_HOSTS environment isn’t present.

Static files

The Django development server automatically serves static files. In production, though, you must define a STATIC_ROOT directory where collectstatic will copy them.

Even though no static files are used in this tutorial, App Platform will fail to build your app if STATIC_ROOT is missing.

In myproject/settings.py, update the STATIC_ROOT and STATIC_URL settings as follows:

# myproject/settings.py
# ...
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
# ...

STATIC_URL tells Django at what URL the static files should be accessible on the web server. STATIC_ROOT is the absolute path to the directory where you want Django to collect static files for deployment using collectstatic.

Configure PostgreSQL for App Platform

App Platform databases provide connection strings. You’ll use the DJ-Database-URL package to enable Django to use connection strings.

(venv) $ pip install dj-database-url

You also need to install the psycopg2 package. psycopg2 is a PostgreSQL adaptor for Python. It’s needed for the Django ORM to communicate with PostgreSQL.

(venv) $ pip install psycopg2-binary

Next, use pip freeze to write your dependencies to a file named requirements.txt:

(venv) $ pip freeze > requirements.txt

This file is required to tell App Platform what Python dependencies to install.

In myproject/settings.py, import dj_database_url and sys and update the DATABASES setting. You’ll also add a new directive, DEVELOPMENT_MODE, to determine local and production development modes. If DEVELOPMENT_MODE is True, use SQLite. Otherwise, use the App Platform PostgreSQL database.

The sys Python standard library module gives you access to argv, a list of command-line arguments passed to the script being currently run. sys.argv[1] allows you to check if the collectstatic command is being run. If collectstatic is being run, you do not attempt a data connection.

# myproject/settings.py
# ...
import dj_database_url
import sys

# ...

DEVELOPMENT_MODE = os.getenv("DEVELOPMENT_MODE", "False") == "True"

if DEVELOPMENT_MODE is True:
    DATABASES = {
        "default": {
            "ENGINE": "django.db.backends.sqlite3",
            "NAME": os.path.join(BASE_DIR, "db.sqlite3"),
        }
    }
elif len(sys.argv) > 0 and sys.argv[1] != 'collectstatic':
    if os.getenv("DATABASE_URL", None) is None:
        raise Exception("DATABASE_URL environment variable not defined")
    DATABASES = {
        "default": dj_database_url.parse(os.environ.get("DATABASE_URL")),
    }

# ...

Create a remote code repository for your app

To deploy to App Platform, you need a remote code repository. In this step, you’ll initialize a Git repository and push it to GitHub.

Start by initializing a Git repo:

git init

Create a .gitignore file and add the following:

# .gitignore
venv
*.pyc
db.sqlite3

# macOS file
.DS_Store

.DS_Store is a macOS-specific file that doesn’t need to be present on other operating systems.

Add your changes:

git add .

Commit those changes:

git commit -m 'Create Django app for App Platform'

Now your code is committed, you’ll push it to GitHub.

In your browser, log in to GitHub and create an empty repository called django-memcache. The repo can be public or private.

Back in the terminal, add your GitHub repo as a remote origin, replacing your_username with your actual GitHub username:

git remote add origin https://github.com/your_username/django-memcache.git

That command tells Git where to push your code to.

Rename the default branch:

git branch -M main

And push your code to GitHub:

git push -u origin main

Your app’s code is now on GitHub, ready to be deployed to App Platform.

Deploy Django to App Platform

You can now set up your app with App Platform.

First, log in to your DigitalOcean account. From the Apps dashboard, click Create, then Apps.

On the Create Resource From Source Code screen, select GitHub as the Service Provider. Then, give DigitalOcean permission to access your repository. The best practice is to select only the repository you want to be deployed. You’ll be prompted to install the DigitalOcean GitHub app if you still need to do so. Select your repository from the list and click Next.

Add a run command

After clicking Next, you’ll be taken to the Resources screen. Click the Edit button beside your django-memcache Web Service. Find the Run Command setting and click its Edit button. Add the following run command and click Save:

gunicorn --worker-tmp-dir /dev/shm myproject.wsgi

That command starts your Django application using Gunicorn. The --worker-tmp-dir /dev/shm option specifies the directory for temporary files. /dev/shm is a temporary filesystem stored in memory and is often much faster than disk-based filesystems. Setting the --worker-tmp-dir option can avoid Gunicorn excessively blocking.

Click Back at the bottom of the django-memcache Settings screen.

Choose a plan

Back on the Resources screen, click Edit Plan to select your plan size. This tutorial will use the Basic Plan with the smallest size Web Services (512 MB RAM | 1 vCPU) for the app-platform-memcache resource. The Basic Plan and smallest web service offer enough resources for this sample Django app.

Once you have set your plan, click Back.

Attach a database

To create a database for your app, click the Add Resource (Optional) toggle, then select Database as the Resource Type, and click Add.

On the Configure your database screen, you’ll create a development database. Name it db. Then, click the Create and Attach button.

Back on the Resources screen, click Next to go to the Environment screen.

Configure environment variables

Next, you’ll configure the environment variables necessary for your app to run. You can set environment variables at the Global and Web Service level. You’ll set these variables at the Web Service level.

Beside django-memcache, click Edit.

Notice DATABASE_URL-${db.DATABASE_URL} was automatically set when you attached your database.

Set the following variables:

• DJANGO_ALLOWED_HOSTS: ${APP_DOMAIN}
• DEBUG: True
• DJANGO_SECRET_KEY: Generated a strong password (check encrypt to encrypt for safety)

${APP_DOMAIN} and ${db.DATABASE_URL} are App Platform specific variables. To learn more about these different variables, read the App Platform Environment Variable Documentation.

Click Save, then, click Next.

Select a region

Choose a region for your app. I’ll use New York.

Click Save, then click Next to continue to the Review screen.

Create resources

On the Review tab, click the Create Resources button to build and deploy your app. It will take a little while for the build to run. When it is finished, you will receive a success message with a link to your deployed app.

Run migrations

Once your deployment is live, you need to run migrations. Go to your app’s Console tab and run python manage.py migrate.

Open up your app’s App Platform URL in your browser. It will look something like https://clownfish-app.ondigitalocean.app/. The app should work as it does locally, apart from one thing. Static assets need to be served in production.

Optional step - Serving your static files

In production, when DEBUG is False, Django static files must be served explicitly. When DEBUG is True, they are served automatically by runserver. This tutorial doesn’t use any static files, so you can skip this step if you like.

Static files are configured to be collected and placed in a folder, ready to serve, but they still need to be deployed. Follow the DigitalOcean documentation Deploying your static files.

Next, you’ll implement caching in Django.

Set up caching in Django

To set up caching in Django, you will do the following steps:

• Configuring an Object Cache with MemCachier
• Configure Django to use your cache

Configuring an Object Cache with MemCachier

In this step, you’ll create and configure an object cache. Any memcached-compatible cache would work for this tutorial. You’ll provision one with the MemCachier Add-On from the DigitalOcean Marketplace.

MemCachier is a fully managed caching service that simplifies setting up and using Memcached in your web applications. Built for seamless integration with cloud platforms like DigitalOcean App Platform, MemCachier offers developers a streamlined and hassle-free way to implement high-performance caching in their applications without worrying about the complexities of managing a cluster of Memcached servers.

First, you’ll add the MemCachier Add-On from the DigitalOcean Marketplace. Visit the MemCachier Add-On page and click Add MemCachier. On the next screen, give your cache a name, choose the Free Developer plan, and make sure to select the same region your App Platform app is in. I’ll use nyc1.

Your app and cache should be in the same region so that latency is as low as possible. You can view your App Platform app’s settings if you need to find the region again. Then, click Add MemCachier to provision your cache.

Next, you’ll set your cache’s configuration values as environment variables for your Django Web Service component. Click the name of your MemCachier Add-On. Then, click the Show button for Configuration Variables to display the values for MEMCACHIER_USERNAME, MEMCACHIER_PASSWORD, and MEMCACHIER_SERVERS.

You’ll now save your MemCachier configuration variables as environment variables for your app. Return to your App Platform app’s dashboard and click Settings. Then, under Components, click django-memc…. Scroll to the Environment Variables section, click Edit, and then add your MemCachier configuration variables with the three keys (MEMCACHIER_USERNAME, MEMCACHIER_PASSWORD and MEMCACHIER_SERVERS) and the corresponding values you got from the MemCachier dashboard. For MEMCACHIER_PASSWORD, check Encrypt because the value is a password.

Click Save to update the app. Next, you’ll configure Django to use your cache.

Configure Django to use your cache

To begin, install the django-bmemcached Django cache backend. That package uses the bmemcached module, which supports the Memcached binary protocol with SASL authentication, allowing you to connect to your cache with a username and password.

(venv) $ pip install django-bmemcached

And update your dependencies list:

(venv) $ pip freeze > requirements.txt

Configure Django to use your MemCachier cache by adding the following to the end of myproject/settings.py:

# myproject/settings.py
# ...
def get_cache():
    try:
        servers = os.environ['MEMCACHIER_SERVERS']
        username = os.environ['MEMCACHIER_USERNAME']
        password = os.environ['MEMCACHIER_PASSWORD']

        return {
            'default': {
                'BACKEND': 'django_bmemcached.memcached.BMemcached',
                # TIMEOUT is not the connection timeout! It's the default expiration
                # timeout that should be applied to keys! Setting it to `None`
                # disables expiration.
                'TIMEOUT': None,
                'LOCATION': servers,
                'OPTIONS': {
                    'username': username,
                    'password': password,
                }
            }
        }
    except:
        return {
            'default': {
                'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'
            }
        }

CACHES = get_cache()

This code configures the cache for both development and production. If the MEMCACHIER_* environment variables exist, the cache will be set up with django_bmemcached, connecting to MemCachier. Whereas, if the MEMCACHIER_* environment variables don’t exist—hence development mode—Django’s simple in-memory cache is used instead.

Implement caching strategies in Django

You’ll now implement several caching techniques in Django: caching expensive database queries, template fragments, entire views, and sessions.

Cache expensive database queries

Memcache is often used to cache expensive database queries. Our simple task list app doesn’t include any resource-intensive queries, but for the sake of learning, let’s imagine that getting all tasks from the database is an expensive operation.

The code to fetch the task list from the database can be modified to check the cache first. In myapp/views.py, update the following:

# myapp/views.py
# ...
from django.core.cache import cache
import time

TASKS_KEY = 'tasks.all'

def index(request):
    tasks = cache.get(TASKS_KEY)
    if not tasks:
        time.sleep(2) # simulate a slow query
        tasks = Task.objects.order_by('id')
        cache.set(TASKS_KEY, tasks)
    context = {'tasks': tasks}
    return render(request, 'index.html', context)

# ...

The above code first checks the cache to see if the tasks.all key exists in the cache. A database query is executed, and the result is cached if it does not. Subsequent page loads will use the cached value instead of querying the database. The time.sleep(2) exists to simulate a slow query.

Redeploy the app to App Platform with the following:

(venv) $ git add .
(venv) $ git commit -m 'Add query caching'
(venv) $ git push

View Memcache statistics

To help demystify Memcache caching operations, it’s helpful to visualize what’s going on under the hood.

Though very cumbersome, one way to do that is to telnet into a Memcached server and run the stats command to see changes as operations are performed on your cache.

With MemCachier, however, you get an analytics dashboard that displays your cache’s statistics so you can monitor performance and troubleshoot issues quickly and easily.

To open your MemCachier analytics dashboard, go to your Add-Ons Dashboard and click the View MemCachier dashboard link.

Add a task, and the following should happen:

1. cache.get(TASKS_KEY) is called, but the task list is not yet in the cache (get misses +1)
2. cache.set(TASKS_KEY, tasks) then stores the task list in the cache (Set Cmds +1)

Now, if you refresh the page:

1. cache.get(TASKS_KEY) successfully gets the task list from the cache (get hits +1)

The cache is working, but there is still a significant problem. Add a new task and see what happens. No new task appears on the current tasks list! The new task was created in the database, but the app serves the stale task list from the cache.

Clear stale data

There are many techniques for dealing with an out-of-date cache. You’ll learn about four options:

• Time-based expiration.
• Deleting the cached value.
• Key-based expiration.
• Updating the cached value.

1. Time-based expiration

The easiest way to make sure the cache does not get stale is by setting an expiration time. The cache.set method can take an optional third argument, which is the time in seconds that an item should stay in the cache. If this option is not specified, it defaults to the TIMEOUT argument of the appropriate backend in the CACHES setting in settings.py.

You could modify the cache.set method to look like this, caching the item for 5 seconds:

cache.set(TASKS_KEY, tasks, 5)

This functionality only works when it is known how long the cached value is valid. In our case, however, the cache gets stale upon adding or removing a task, so this technique is inappropriate.

2. Delete the cached value

A straightforward strategy is to invalidate the tasks.all key when you know the cache is out of date—namely, to modify the add and remove views to delete the tasks.all key:

# myapp/views.py
# ...
def add(request):
    if 'name' in request.POST:
        task = Task(name=request.POST['name'], notes=request.POST['notes'])
        task.save()
        cache.delete(TASKS_KEY)
        return redirect('/')
    return render(request, 'add.html')

def remove(request):
    task = Task.objects.get(id=request.POST['id'])
    if task:
        task.delete()
        cache.delete(TASKS_KEY)
    return redirect('/')

Now, whenever a task is added or removed, the cached tasks will be deleted and re-cached the next time the task list is viewed.

3. Key-based expiration

Another technique to invalidate stale data is to change the key:

# myapp/views.py
# ...
import random
import string

def _hash(size=16, chars=string.ascii_letters + string.digits):
    """
    Generate a random string of specified length and character set.

    Args:
        size (int): Length of the string to be generated. Default value is 16.
        chars (str): Set of characters to be used for generating the string. Default is the set of ASCII letters and digits.

    Returns:
        str: A random string of specified length and character set.
    """
    # Use random.choice() to randomly select characters from the specified character set
    # Repeat this process for the specified length of the string
    # Use join() to concatenate all the randomly selected characters into a single string
    return ''.join(random.choice(chars) for _ in range(size))

def _new_tasks_key():
    return 'tasks.all.' + _hash()

TASKS_KEY = _new_tasks_key()

# ...

def add(request):
    if 'name' in request.POST:
        task = Task(name=request.POST['name'], notes=request.POST['notes'])
        task.save()
        global TASKS_KEY
        TASKS_KEY = _new_tasks_key()
        return redirect('/')
    return render(request, 'add.html')

def remove(request):
    task = Task.objects.get(id=request.POST['id'])
    if task:
        task.delete()
        global TASKS_KEY
        TASKS_KEY = _new_tasks_key()
    return redirect('/')

The upside of key-based expiration is that you do not have to interact with the cache to expire the value. The LRU (Least Recently Used) eviction of Memcache will eventually clean out the old keys. When the memory limit is reached, LRU eviction is used in Memcache to remove the least recently used items from cache memory.

4. Update the cached value

Instead of invalidating the key, a cached item’s value can be updated:

# myapp/views.py
# ...

def add(request):
    if 'name' in request.POST:
        task = Task(name=request.POST['name'], notes=request.POST['notes'])
        task.save()
        cache.set(TASKS_KEY, Task.objects.order_by('id'))
        return redirect('/')
    return render(request, 'add.html')

def remove(request):
    task = Task.objects.get(id=request.POST['id'])
    if task:
        task.delete()
        cache.set(TASKS_KEY, Task.objects.order_by('id'))
    return redirect('/')

This approach is more straightforward to implement than key-based expiration but as performant. And it means the task list is immediately re-cached, so the next time it is loaded, database query and cache.set calls are unnecessary.

Choose option 2, 3, or 4 to ensure the cache is never out-of-date. I’m using option 4, updating the cached value, for this tutorial.

As usual, commit your changes and redeploy the app:

(venv) $ git add .
(venv) $ git commit -m 'Add query caching invalidation'
(venv) $ git push

Now, when you add or remove a task, all the tasks you’ve added since implementing caching will appear.

If you’ve implemented option 4 to clear stale cache data, each time you add or remove a task, the following should happen:

1. cache.set(TASKS_KEY, ...) updates the cached task list (Set Cmds +1)
2. cache.get(TASKS_KEY) fetches the updated cached task list (get hits +1)

Cache template fragments

Django allows you to cache template fragments. This is similar to snippet caching in Flask. Add {% load cache %} to the top of your template to enable fragment caching.

To cache a rendered set of task items, we use a {% cache <timeout> <key> <optional arguments> %} statement in myapp/templates/index.html:

<!-- myapp/templates/index.html -->
{% load cache %}
<!DOCTYPE html>
<!-- ... -->
  <ul>
    {% for task in tasks %}
      {% cache None taskfragment task.id %}
      <li>
        <a href="/detail/{{ task.id }}">{{ task.name }}</a>
      </li>
      {% endcache %}
    {% endfor %}
  </ul>
<!-- ... -->
</html>

Here the timeout is None and the key is taskfragment. Because this fragment has dynamic data (the task ID and name), you pass an additional argument task.id to uniquely identify each fragment.

As long as task IDs are never reused, this is all there is to cache fragments in Django. If you use a database that does reuse IDs, you must delete the fragment when its respective task is deleted. You can do this by adding the following code to the remove view in myapp/views.py:

# myapp/views.py
# ...
from django.core.cache.utils import make_template_fragment_key

# ...

def remove(request):
    task = Task.objects.get(id=request.POST['id'])
    if task:
        task.delete()
        # ...
        key = make_template_fragment_key('taskfragment', [request.POST['id']])
        cache.delete(key)
    return redirect('/')

Deploy your changes:

(venv) $ git add .
(venv) $ git commit -m 'Add fragment caching'
(venv) $ git push

Let’s see the effect of caching the fragments in our application. The next time you refresh the tasks list view, the following will happen:

1. Django checks if each fragment is in the cache. No fragments are currently cached (get misses +1 per task)
2. Django stores each fragment in the cache (Set Cmds +1 per task)

And on subsequent page refreshes:

1. Django checks if each fragment is in the cache. All fragments are now cached (get hits +1 per task)

Cache views

We can go one step further and cache the output of entire views in addition to fragments. This should be done carefully because it can result in unintended side effects if a view frequently changes or contains forms with CSRF tokens, as explained earlier. The task list in our app changes each time a task is added or deleted, so the cached view must be cleared after both actions.

You can cache the task list view with the @cache_page(<timeout>) decorator in myapp/views.py:

# myapp/views.py
# ...
from django.views.decorators.cache import cache_page

# ...

@cache_page(None)
def index(request):
    # ...

# ...

We must delete the cached view whenever we add or remove a task. This is not trivial. We need to learn the key when the view is cached to delete it:

# myapp/views.py
# ...
from django.utils.cache import learn_cache_key

# ...

VIEW_KEY = ''

@cache_page(None)
def index(request):
    # ...
    context = {'tasks': tasks}
    response = render(request, 'index.html', context)
    global VIEW_KEY
    VIEW_KEY = learn_cache_key(request, response)
    return response

def add(request):
    if 'name' in request.POST:
        # ...
        cache.delete(VIEW_KEY)
        return redirect('/')
    return render(request, 'add.html')

def remove(request):
    # ...
    if task:
        # ...
        cache.delete(VIEW_KEY)
    return redirect('/')

We use the Django cache util learn_cache_key to get the key when the view is cached. Then, delete the item from the cache when adding or deleting a task.

Deploy the changes:

(venv) $ git add .
(venv) $ git commit -m 'Add view caching'
(venv) $ git push

@cache_page stores the page in two items, one for headers and one for the page. The keys for those items look something like this:

# Headers
:1:views.decorators.cache.cache_header..7822c3f267e735391510ed308a3a0176.en-us.UTC

# Page
:1:views.decorators.cache.cache_page..GET.7822c3f267e735391510ed308a3a0176.d41d8cd98f00b204e9800998ecf8427e.en-us.UTC

On the first refresh, the following will happen:

1. @cache_page tries to fetch the header item, but it’s not yet cached (get misses +1)
2. @cache_page stores the header item in the cache (Set Cmds +1)
3. @cache_page stores the page item in the cache (Set Cmds +1)
4. learn_cache_key() also stores the header item in the cache (Set Cmds +1)

On subsequent page loads, the following happens:

1. @cache_page fetches the header item from the cache (get hits +1)
2. @cache_page fetches the page item from the cache (get hits +1)

Note view caching does not make the caching of expensive operations or template fragments redundant. It’s good practice to cache smaller operations within larger cached operations or smaller fragments within larger fragments. This technique (called Russian doll caching) helps with performance if a larger operation, fragment, or view is removed from the cache because the building blocks do not have to be recreated from scratch.

Advanced Memcache Debugging

With three caching strategies in place (query, fragment, and view caching), it can be challenging to determine precisely what cache activity is happening.

Our analytics dashboard also comes with advanced cache Introspection features (Prefixes, Keys, and Recent Requests), available starting with our Advanced plan (1 GB+). These tools can be invaluable for debugging, troubleshooting, and ensuring things are working as expected.

When writing this tutorial, I regularly used the Recent Requests view to verify the executed cache commands. That view shows the last 100 lines of logs for each of your cache servers.

Here are some logged commands you would expect to see with Django, with my comments added:

# Try to get tasks. "Key not found", not in cache (`get misses` +1)
2023-02-23T14:18:12Z GETK Key not found 12B  :1:tasks.all
# Store tasks. "OK", stored successfully (`Set Cmds` +1)
2023-02-23T14:18:14Z SET  OK            1KB  :1:tasks.all

# Try get fragment. Not in cache (`get misses` +1)
2023-02-23T14:39:17Z GETK Key not found 63B  :1:template.cache.taskfragment.691513953ea19e01a0e7881a339ce106
# Store fragment. Stored successfully (`Set Cmds` +1)
2023-02-23T14:39:17Z SET  OK            144B :1:template.cache.taskfragment.691513953ea19e01a0e7881a339ce106

# Try get view. Not in cache (`get misses` +1)
2023-02-23T14:18:12Z GETK Key not found 82B  :1:views.decorators.cache.cache_header..7822c3f267e735391510ed308a3a0176.en-us.UTC
# Store view header. Stored successfully (`Set Cmds` +1)
2023-02-23T14:18:14Z SET  OK            95B  :1:views.decorators.cache.cache_header..7822c3f267e735391510ed308a3a0176.en-us.UTC
# Store view page. Stored successfully (`Set Cmds` +1)
2023-02-23T14:18:14Z SET  OK            1KB  :1:views.decorators.cache.cache_page..GET.7822c3f267e735391510ed308a3a0176.d41d8cd98f00b204e9800998ecf8427e.en-us.UTC

Use Memcache for session storage

Memcache works well for storing information for short-lived sessions that time out. However, because Memcache is a cache and therefore not persistent, long-lived sessions are better suited to permanent storage options, such as your database.

For short-lived sessions, configure SESSION_ENGINE to use the cache backend. Add the following to the end of myproject/settings.py:

# myproject/settings.py
# ...
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'

Django allows you to use a write-through cache backed by a database for long-lived sessions. This is the best option for performance while guaranteeing persistence. To use the write-through cache, configure the SESSION_ENGINE in myproject/settings.py like so:

SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'

For more information on how to use sessions in Django, please see the Django Session Documentation.

Clean up

The app you have deployed in this tutorial will incur charges, so you can optionally destroy the app and the MemCachier Add-On when you have finished working with them.

From the app’s dashboard, click Actions, then Destroy App. Destroying the app will also destroy the attached database.

To clean up your MemCachier Add-On, click Add-Ons, then the name of your MemCachier Add-On. Next, click on Settings and Destroy.

Further reading and resources

• MemCachier Django Documentation
• App Platform Django Documentation
• Django Caching Documentation