A Simple Fix If Your Dockerized App Crashes Because The Database Container Isn't Ready Yet
It’s frustrating to work with a fragile docker-compose development setup.
One frequent issue, is if the dockerized application crashes because the database isn’t ready early enough. All you see is an error message and a terminated container.
Racing conditions are pretty annoying. They don’t always happen, and it’s easy to get caught in the trap of “retrying until it works again” once too often.
Here’s an easy way to work around your failing dependency quickly, and get a fix in place without breaking out of your context, blocking out time to identify the exact issue and fix the root cause.
“Depends On” And Why It Doesn’t Work
If you want containers to start in a particular order, depends_on may seem like an easy fix.
However, this configuration option won’t help you. depends_on only makes sure the containers are started in the right order,
not that they are ready in the right order. And “ready” is what you want your database to be.
This still results in crashes, because your application is ready first, and sometimes fails to access the (still starting) database service.
An Easy Fix
Configure your application container to be restarted upon failure. It’s a single configuration, and can be easily added to your docker-compose file.
This is a snippet of what it looks like in your docker-compose.yml:
version: "3"
services:
app:
# (...)
restart: on-failure
Yep, that’s all. You can read more about restart in the docs. This will work if Docker isn’t running in Swarm mode (restart_policy can be used otherwise case).
The upside: it’s really simple to do, you don’t need to rebuild your Docker image and your docker-compose services will come up eventually. Sometimes right away, sometimes after a few retries.
The downside: it’s a simple and naive approach. You can hit edge cases, such as exceeding the retry limit if there’s too much going on. This also doesn’t look too pretty - the docker-compose output might be flooded with error messages from the application container. Especially if crashes occur later on.
A First Step, But You Can Do Better
Simply restarting on failure is a simple way to make your docker-compose development environment a little bit less brittle when it comes to the startup order of your services.