Using Docker For Python Web Development - Why And How

• “What’s the advantage of using Docker instead of virtualenv?”
• “Why run a development server inside a container. It works just fine as it is.”
• “This looks like unnecessary complexity. Why would anyone do that?”

That’s what you sometimes hear from smart, busy people who are first introduced to the idea of using containers for local development.

Before diving in: If you have worked with Vagrant to setup reproducible, automated development environments, you don’t have a lot to gain from switching to Docker locally. You’re fine, and can stick to the workflow you have.

The Pain

Do you know the feeling, when your development machine ends up with a crapload of software, which you installed and don’t use often, but needed for a project a while back? How about having to reinstall everything for each of your active projects when you get a new laptop, switch environments or just decide to try a new Linux distro?

It feels messy. At some point, everywhere you look is cluttered up with dependencies, and at some point it’s hard to tell if upgrading one of them for one project, will cause your other work to start exhibiting weird bugs.

Getting an old project to run on a different OS can be challenging, depending on how lucky you are with stuff like compilation dependencies and libraries being similar.

No Pain For Me

I’m doing my development work on two different machine: one Ubuntu laptop, chosen for the computational power and convenience, and a tiny MacBook for travel, mobility and working without having to worry about the battery being drained too fast. Still, all I have to do when I switch between machines, is doing a git pull and I’m ready to go.

There’s no different configuration files for those environments, and I don’t have to write two sections in each README, to tell future-me how exactly to install this particular database version on OSX, and how to do it on RedHat or Ubuntu. When I’ll finally get around to installing Manjaro, I will be able to start working on my projects without finding out everything from scratch and losing time.

Here’s How

When done right, Vagrant with Ansible or Salt to provision VMs can achieve many of the benefits for you, as using Docker for local development does:

• The development environments is host-machine agnostic and reproducible.
• You can set it up with a single command - completely with functional database data and running dev server from scratch (you can, right?).
• Certainty that you KNOW the steps needed to get your project into a dev-able state, or can make sure with little effort.
• The app is accessible on a local port, which is convenient to reach.
• You can mount the local code folder in the other environment without much effort. This way you can use your favourite editor and browser, and the development server can react to changes.

Using Docker differs slightly from the above approach when looking at the upsides. Those are why I prefer using it for new projects:

• It’s not as resource hungry as a VM - this matters for battery lifetime and when using a tiny laptop.
• The provisioning is quicker than a VM, once the images are pulled.
• The skills and configs around Docker can be used as a starting point for an eventual future deployment/production setup. The same is true for Ansible, but the things they do only overlap and are not completely interchangeable.

The benefits of using Docker (or Vagrant) for development are too good to miss out on. The main objections are IDE interaction being hindered, slow iteration cycles and the daunting learning curve. The last one is easy to bridge, and slow iteration cycles are only due to using the tool wrong. With the approach described below, you can avoid the most common pitfalls and see for yourself if it suits your current needs.

The Setup

I have a Docker Compose powered setup for each project which can benefit from it. A project qualifies as soon as it’s out of toy status, uses a database or depends on system-specific libraries.

This setup fixes pains with:

• Clutter.
• Working on multiple development machines.
• Risking multiple local projects interfering with each other.
• Stuff breaking because of your development machine changes.
• Having outdated dev setup docs or missing important points.

With Docker, it’s easy to try to get started, run into a situation where you are disappointed by the tool and never go further. Usually that’s due to lacking an overview of the basics, or trying to use it badly for the wrong reasons.

The case-for-using-Docker only really starts to emerge once you have to compile dependencies, need paricular system libraries or access to a local backing service (such as a PostgreSQL database) in a particular version. Before those, virtualenv and maybe management scripts got you completely covered. If that changes however, you’re not using Docker instead of virtualenv, but rather with it. More on that below.

Think of Docker, as a mixture of git and virtualenv for everything which virtualenv does not cover. You can use it, to capture more than project dependencies on the python module level. You can have your dependencies, pinned to a particular version and configured correctly as code.

You can see the source for the following examples in an in-progress toy project here.

First, we need a container which is built to be the running environment for our Python app. Here’s an example Dockerfile for a Flask app.

EDIT: this article is kinda old. The following Dockerfile works, but it has a few shortcomings. Check out this article for a review of a better Flask app Dockerfile, what it does right and what could be improved on.

FROM ubuntu:16.04

# bring system up-to-date
RUN apt-get update -qq && \
    apt-get upgrade -qqy

# install a particular version of Python and other stuff
RUN apt-get install -qqy \
    python-virtualenv \
    libpq-dev \
    python3=3.5.* \
    python3-dev=3.5.*

# copy scripts into the container for convenience (you could mount a folder as well)
RUN mkdir -p /srv
ADD start.sh /srv/start.sh
RUN chmod +x /srv/start.sh
ADD maintenance.sh /srv/maintenance.sh
RUN chmod +x /srv/maintenance.sh

# tell the container to execute the start script
ENTRYPOINT ["/bin/bash", "/srv/start.sh"]

Starting with an Ubuntu 16.04 container, the system is updated and dependencies are installed. The app will need libpq-dev to build psycopg2 for communicating with PostgreSQL, and Python 3.5 as well as virtualenv. Afterwards, convenience scripts are copied into the imager. We could also mount a directory with those when running the container instead. As this is a development setup we can be less strict on security best practices without risk or downsides. .

The image created with the above Dockerfile, is used in a Docker Compose file, which brings up multiple interconnected containers and configures them using an easy-to-read docker-compose.yml file. Here is the file used for a simple Flask app:

version: '2'

volumes:
  postgres_data: {}
  env: {}

services:
  app:
    build: ./compose/app_dev
    volumes:
      - ./baa:/srv/app
      - env:/srv/env
    working_dir: /srv/app
    env_file:
      - env_dev
      - env_dev_secret
    ports:
      - "127.0.0.1:5000:5000"
    depends_on:
      - db
      - redis
    # uncomment for debugging the service - container does not try to start dev server
    #entrypoint: ["sh", "-c", "sleep infinity"]
  db:
    #https://hub.docker.com/_/postgres/
    image: postgres:9.6.3
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: dbpw
    ports: #make db accessible locally
      - "127.0.0.1:5432:5432"
  redis:
    #https://hub.docker.com/_/redis/
    image: redis:3.2.9
    ports: #make redis accessible locally
      - "127.0.0.1:6379:6379"

The file consists of two main blocks: volumes and services. Each entry in the ‘services’ block corresponds to one container which is going to run once we issue the command docker-compose up. The name of each service is arbitrary, but can be used from the other containers to access via the network which is brought up by Docker Compose and connects them.

The content of each service specifies a Docker image to use when starting the container, as well as settings to configure it and environment variables to set. The env_file for example, is a file with lines of variables which is read from. Docker variables and all the way you can use them need some getting used to.

In the ‘volumes’ block, we define something like mountable directories handled by Docker, which are used to store data between container restarts. Every state and content in a container is lost once it’s removed, but the volumes persist unless deleted. This way we can use the virtualenv data from a previous container run as it is preserved in a volume. The database is configured with credentials, and made accessible locally. The local code folder (baa) is mounted into the container from the local machine. This way, you can work on the code locally, and see changes in the development container immediately.

The env and db data is saved in volumes. This way it survives restarts, This also helps to keep the startup time as low as possible.

The volume data can be wiped with

$ docker-compose down -v

The exact steps needed to setup a dev environment, is described in the README, as well as maintenance commands.

Conclusion

So, that’s what I use to develop Python applications, once virtualenv does not cover the complete setup anymore. This makes it possible to develop multiple projects on a single machine without causing lots of clutter over time or disrupting other projects. Moving to other operating systems on a single machine, or working on multiple different ones becomes easy and painless using Docker. While you can get similar upsides when using Vagrant, my personal choice falls to Docker as the skills you use are useful beyond development environments in the long term and you can save on resources.

If you want, you can try adapting the setup I have described for one of your simpler projects and see if it suits for your current workflows.