Speed Up The Build of Your Python Docker Image

September 28, 2017

Are you waiting for ages, every time you’re changing a tiny detail in your dockerized Python project and need to build an image? Literally any comment change, makes it necessary to install every. single. pip. dependency. Over and over! Aaaargh.

That’s not an issue if you’re working in your dev environment, as you’re simply mounting your code and have a volume for your virtualenv directory - but building a new image for testing, staging or production takes a long time. It almost feels like you’d rather say ‘I’m not slacking off, my Docker image is building’ and go for a coffee every time that happens. Why did you even start using Docker? The eternal waiting times are not worth it. Or is there a way around it?

The Problem

Your Dockerfile probably contains something like this:

ADD code /app # executed on every small change
RUN pip install -r /app/requirements.txt
# and here we go again...

You’re adding your project code (Flask or Django project?) after installing the necessary libraries and setting up a virtual environment. Then, you’re running pip to install the exact versions of every Python dependency needed for the project in a “requirements.txt” file.

You’re not using the Docker cache as well as you could. The good news is: there’s a simple way to fix that.

Use The Docker Cache

You can prevent the perpetual re-execution of the dependency-installation step if there were not actual changes to the stuff you’re using. There’s no tricky volume mounting or multi-stage build kung-fu needed.

The ADD directive only needs to run if the referenced file changed since the last time it was executed. If it did, every single build step needs to run again, but if it’s the same you can just use a version from the Docker cache and skip to the next one.

If you add the requirements.txt file before your other code, and run the pip install step right after it, both will only be executed if the file changes. Not on every build.

ADD code/requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# the steps above only depend on the requirements.txt file!
ADD code /app

This way, you can skip the expensive operation if nothing changed and reuse a cached state. You Docker image build will be faster in most cases. Once that’s not enough anymore, there are more elaborate ways to improve on the process.

Join the mailing list!

Subscribe to get notified about future articles and stay in touch via email.

I write about Django, Kubernetes, Docker, automation- and deployment topics, but would also like to keep you up to date about news around the business-side of things.

Privacy and your data: You can get more information about the usage of your data, the storage of your registration, sending out mails with the US-provider ConvertKit, statistical analysis of emails sent and your possibility to unsubscribe in my Privacy Policy.

I use the US-provider ConvertKit for email automation. By clicking to submit this form, you acknowledge that the information you provide will be transferred to ConvertKit for processing in accordance with their Privacy Policy and Terms.

We won't send you spam. Unsubscribe at any time. Powered by ConvertKit