Testing a Django third party application with pytest and tox

Introduction

Let’s say we’ve got an idea for a third-party application for Django. We’ve written all the code. We’ve run it against a small Django project to test it. Now we are ready to release it to PyPI for others to use.

However, there’s a chance that some issues can pop up:

• The package could be based on some underlying code in Django. That code might change at some point. This can lead to an incompatibility.
• How do we know it’s not already incompatible with previous versions of Django?

For these reasons, we will have to write automated tests for our application. In this post, I’ll give you a quick guide on how to set them up.

If you need to reference some example code, you can take a look at django-enum-choices (which we recently announced).

Let’s get started

Once we already have our application up and running, its structure should look something like this.

── our_django_third_party
│   ├── __init__.py
│   ├── requirements.txt
│   ├── __version__.py
├── LICENSE
├── README.md
├── setup.py

The first thing we can do is to unit test any logic in the application itself using Python’s unittest.TestCase.

This can be done with standard python unit tests, but what happens when we want to automate tests for our integration with a Django project?

How do we test model behaviour, Django admin’s behaviour, or what if our package can be integrated with other third-party Django applications, like Django Rest Framework, which require Django in order to function?

Well, we need a Django project set up for that, but we want to use it only inside our tests.

Setting up a test-only Django project

The File Structure

Let’s create a new directory containing our tests inside the package:

── our_django_third_party
│   ├── __init__.py
│   ├── requirements.txt
│   ├── __version__.py
│   ├── tests
│   │   ├── __init__.py
│   │   ├── some_unit_tests.py
│   │   ├── model_integration_tests.py
├── LICENSE
├── README.md
├── setup.py

We named the file with the standard unit tests some_unit_tests.py and we added a new file called model_integration_tests.py, which will contain our tests that use Django models.
First thing we need to do if we want to have tests that are using models and the database, is to make all subclasses of unittest.TestCase inherit from django.test.TestCase instead.

Since we want to have models now we will need to do the following:

1. Create a Django project
2. Create a Django app inside the project
3. Add a models.py in the new app where we will store our models
4. Add a settings.py  file where we will register our installed app which will let Django run migrations for us automatically when running the test suite

After following the steps above, the package structure should look like this:

── our_django_third_party
│   ├── __init__.py
│   ├── requirements.txt
│   ├── __version__.py
│   ├── tests
│   │   ├── __init__.py
│   │   ├── some_unit_tests.py
│   │   ├── model_integration_tests.py
│   │   ├── settings.py
│   │   ├── testapp
│   │   │   ├── apps.py
│   │   │   ├── models.py
├── LICENSE
├── README.md
├── setup.py

We have added settings.py in the tests directory, and we’ve created a Django app, called testapp which contains our models and apps.py.

Inside apps.py we have our Django app config:

from django.apps import AppConfig


class TestAppConfig(AppConfig):
    name = 'our_django_third_party.tests.testapp'
    verbose_name = 'TestApp'

Inside settings.py we have the setup for our databases, the default INSTALLED_APPS which come from django-admin startproject with the addition of our newly added app config:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": "mem_db"
    }
}


INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.sites",
    "our_django_third_party.tests.testapp.apps.TestAppConfig"
]

Now in models.py we can create any models that we need to use in our tests.

Switching the test runner

Since we’re not using unittest.TestCase anymore, we need to run our tests with a more comprehensive test runner, such as nose or pytest or even Django’s test runner with python manage.py test. My personal preference is to use pytest as we’re not using a standard Django project.

For running our tests with pytest we’re going to have to install 2 external libraries with  pip : pytest being the first one, of course, and pytest-django for running tests that require Django (our model tests in this case).

What we need to do after installing our pytest requirements is to create our pytest configuration file. Here’s how things look like after creating pytest.ini

── our_django_third_party
│   ├── __init__.py
│   ├── requirements.txt
│   ├── __version__.py
│   ├── tests
│   │   ├── __init__.py
│   │   ├── some_unit_tests.py
│   │   ├── model_integration_tests.py
│   │   ├── settings.py
│   │   ├── testapp
│   │   │   ├── apps.py
│   │   │   ├── models.py
├── LICENSE
├── README.md
├── setup.py
├── pytest.ini

And the contents of pytest.iniare:

[pytest] 
DJANGO_SETTINGS_MODULE = our_django_third_party.tests.settings 
django_find_project = false

DJANGO_SETTINGS_MODULE points pytest-django to the settings moudle that should be used when executing tests.

By default pytest-django also expects an actual Django project with a manage.py file inside it, which we don’t have since we’re using Django only in our tests. Because of that we need to set django_find_project to false. This will tell pytest-django not to automatically search for manage.py.

We can now run our tests safely!

Plugging in tox

Why should we use tox?

In the introduction, I mentioned that there is a possibility for third party Django applications to become incompatible with Django at some point or they might already be incompatible with previous versions of it as well. The same applies to Python versions. In order to be sure that our package isn’t creating issues for its users, using a different Python, Django, or any other library that our package extends we need to have different environments with different versions of these dependencies.

If we wanted to do that manually, we would have to create separate Python virtual environments with different versions for each dependency and then run the tests separately for each one. Let’s say we want our package to be compatible with Python 3.5, 3.6, and 3.7 as well as Django 1.11, 2.1, and 2.2. This means we need to create environments equal to the number of combinations between Python and Django versions that we  want to support, resulting in 9 different python environments (3 for each Python version).

Creating 9 python environments, installing different dependencies in each one of them, switching between them, and running pytest 9 times, sounds like a lot of pain, doesn’t it?

That’s where tox comes in. It provides an easy way to run all your tests in different python environments with just a few lines of configuration.

Overview

The tox configuration that we need has 2 main points:

• [tox] – This is where we define our testing environments.  We’re going to use the ones, listed in the previous paragraph:
• Django 1.11 with Python 3.5, 3.6, 3.7
• Django 2.1 with Python 3.5, 3.6, 3.7
• Django 2.2 with Python 3.5, 3.6, 3.7
• An environment for linting our code (Will use our local development dependencies)
• [testenv] – This is where we can list our dependencies and commands to run in the environments.
• deps – A list of all dependencies that our environment needs (if we have only one environment) or mapping of environment names their corresponding dependencies (if we have multiple environments)
• commands – The commands to execute after setting up each environment

Creating the configuration file

The first thing we should do is add a tox.ini file in the root directory of our project:

── our_django_third_party
│   ├── __init__.py
│   ├── requirements.txt
│   ├── __version__.py
│   ├── tests
│   │   ├── __init__.py
│   │   ├── some_unit_tests.py
│   │   ├── model_integration_tests.py
│   │   ├── settings.py
│   │   ├── testapp
│   │   │   ├── apps.py
│   │   │   ├── models.py
├── LICENSE
├── README.md
├── setup.py
├── pytest.ini
├── tox.ini

Our tox.ini will contain only the [tox] section with our environment list and an empty [testenv] section for now:

[tox]
envlist =
    lint-py{37}
    django22-py{37,36,35}
    django21-py{37,36,35}
    django111-py{37,36,35}

[testenv]

What’s going on here:

We’re listing our environments in the envlist variable of the first section. The environments that contain both Django and py in their definitions (djangoX-py{Y}) are made of 2 different parts. Inside the curly braces, we have defined the Python versions we want our project to be tested within the listed environment, 37 meaning Python 3.7, etc. Outside of the braces, stands the name prefix for the listed environment. For example, django22-py{37,36,35} means we want an environment named django22 which will be created with 3 different python versions.

The next step is to add our dependencies and commands.

First, we’re going to define 2 new sections in our configuration. The first will contain all dependencies we need for every environment we need and the second will contain variables with the dependencies for the different Django versions that we need to support:

[base]
deps =
    pytest
    pytest-django

[django]
2.2 =
    Django>=2.2.0,<2.3.0
2.1 =
    Django>=2.1.0,<2.2.0
1.11 =
    Django>=1.11.0,<2.0.0

Every environment will need pytest and pytest-django in order to run the tests, but each one will require different Django versions.

Now we can define our dependencies in the [testenv] section, using the 2 we just created. Here is our test.ini file at this point:

[tox]
envlist =
    lint-py{37}
    django22-py{37,36,35}
    django21-py{37,36,35}
    django111-py{37,36,35}

[testenv]
deps =
    {[base]deps}[pytest]
    django22: {[django]2.2}
    django21: {[django]2.1}
    django111: {[django]1.11}
commands = pytest

[base]
deps =
    pytest
    pytest-django

[django]
2.2 =
    Django>=2.2.0,<2.3.0
2.1 =
    Django>=2.1.0,<2.2.0
1.11 =
    Django>=1.11.0,<2.0.0

We have also added the commands variable to the [testenv] section. It tells tox to  run the pytest command in every environment.

What we’ve done in the deps variable:

1. We’ve told tox to install all dependencies, listed in the [base], on every combination that it creates.
2. After that install specific dependencies from the [django] section to the according environments

There’s one last thing we need to add to our tox configuration – the commands and dependencies for the lint environment. We don’t need Django for running a linter on our package so we’re going to create a new branch from the [testenv] section and add our dependency for linting and its execution command.

This is what our final tox.ini looks like:

[tox]
envlist =
    lint-py{37}
    django22-py{37,36,35}
    django21-py{37,36,35}
    django111-py{37,36,35}

[testenv]
deps =
    {[base]deps}
    django22: {[django]2.2}
    django21: {[django]2.1}
    django111: {[django]1.11}
commands = pytest

[testenv:lint-py37]
deps =
    flake8
commands = flake8 our_django_third_party/

[base]
deps =
    pytest
    pytest-django
    
[django]
2.2 =
    Django>=2.2.0,<2.3.0
2.1 =
    Django>=2.1.0,<2.2.0
1.11 =
    Django>=1.11.0,<2.0.0

Now,  we can execute tox in our terminal to run the tests in the different environments. One important thing to note – all the Python versions that have been specified in tox.ini  must be present in the environment where tox is executed.

Testing against multiple databases

If your third-party application is interacting in some way with models, there is a chance that you might need to support multiple databases, like we needed to do in django-enum-choices. Here’s how we made our testing setup.

Adding a new database in the settings

This step is simple, you just add your new database configuration to  the DATABASE dictionary in settings.py. In our case, we had to test with PostgreSQL, so we just added the database URL:

import environ

env = environ.Env()

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": "mem_db"
    },
    'postgresql': env.db('DATABASE_URL', default='postgres:///our_test_database')
}

DATABASE_ROUTERS = ['our_django_third_party.tests.testapp.database_routers.DataBaseRouter']

You can see we’ve also added an extra variable to our settings which points to a database router. So let’s explain what it’s for.

The database router

Now that we’ve defined that our Django project will use 2 different databases, we need a way for Django to know which database to use and when. This is what the database router is for. In our case we want the postgresql database to be used only when database operations (migrations, read, write, etc.) are preformed on models, containing PostgreSQL specific fields. And this is what our database router looks like:

from django.apps import apps
from django.db import models
from django.contrib.postgres import fields as pg_fields


POSTGRES = 'postgresql'
DEFAULT = 'default'


class DataBaseRouter:
    def _get_postgresql_fields(self):
        return [
            var for var in vars(pg_fields).values()
            if isinstance(var, type) and issubclass(var, models.Field)
        ]

    def _get_field_classes(self, db_obj):
        return [
            type(field) for field in db_obj._meta.get_fields()
        ]

    def has_postgres_field(self, db_obj):
        field_classes = self._get_field_classes(db_obj)

        return len([
            field_cls for field_cls in field_classes
            if field_cls in self._get_postgresql_fields()
        ]) > 0

    def db_for_read(self, model, **hints):
        if self.has_postgres_field(model):
            return POSTGRES

        return DEFAULT

    def db_for_write(self, model, **hints):
        if self.has_postgres_field(model):
            return POSTGRES

        return DEFAULT

    def allow_relation(self, obj1, obj2, **hints):
        if not self.has_postgres_field(obj1) and not self.has_postgres_field(obj2):
            return True

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if model_name is not None and \
           db == DEFAULT and \
           self.has_postgres_field(apps.get_model(app_label, model_name)):
            return False

        return True

The important things here are the methods in the router that actually determine if the operation should be performed and what database should be used:

• db_for_read – Use postgresql only if the model has PostgreSQL fields inside it, otherwise use default
• db_for_write – Works the same way as db_for_read, but for writing operations instead
• allow_relation –  Use postgresql only if both the objects that the relation must be made between containing PostgreSQL fields.
• allow_migrate – This method is called every time our tests try to build a database. Since we have 2 databases. The first time it’s called with the default database, and the second – with the postgresql database. What we do here is we don’t allow migrations for the default database if these migrations refer to models with PostgreSQL fields. Otherwise, we allow them.

If you need multiple databases for an actual Django project, your router will probably be quite different and it might not be just one. Our requirement here is only to use the databases for testing purposes, so it’s pretty straightforward.

Now that we have our database router, every time a query is performed on a model that has PostgreSQL fields, we will point that query to the postgresql database.

All that is left is to write our tests for the models which require a different database.

from django.test import TestCase


class ModelIntegrationTests(TestCase):
    databases = ['default', 'postgresql']

    def test_model_without_pg_fields(self):
        self.assertIsNotNone(NormalModel.objects.create())

    def test_model_with_pg_fields(self):
        self.assertIsNotNone(ModelWithPgFields.objects.create())

We need to explicitly define the databases that will be used in this test case, otherwise we will receive an error, telling us:  AssertionError: Database queries to 'postgresql' are not allowed in this test .

That’s all there is to it.

Once again, if you need an actual project for reference, check out django-enum-choices .

Thanks for reading and happy testing!