The Reason Why You Need to Wait When Stopping Your Docker Compose Services
Working with docker-compose can feel like a chore if it makes you wait again and again. Waiting for containers to stop or when restarting can cost a lot of time if you’re using docker-compose for development.
If you need to wait for your containers to be “stopped gracefully”, chances are that docker-compose isn’t at fault. Rather, it’s because of your dockerized apps don’t react to a SIGTERM signal and time out instead.
Stopping Containers
When you tell docker-compose to stop containers with docker-compose down, docker-compose stop, docker-compose restart or just by pressing ctrl+c in the terminal where docker-compose up is running, docker-compose tries to stop your containers as gently as possible.
It lets them know, by sending a SIGTERM signal to each container. This is a polite way to let a program know that it’s supposed to shut down at its own pace, but do so quickly. Well-behaved programs that receive a SIGTERM are supposed to finish anything which is important, don’t take on new work and exit as soon as possible on their own terms.
Docker has limited patience when it comes to stopping containers. There’s a timeout, which is 10 seconds by default for each container. If even one of your containers does not respond to SIGTERM signals, Docker will wait for 10 seconds at least. If your containers depend on each other, docker-compose can’t shut them down all at once. It will ask them to stop in the right order, waiting for each one. This waiting time can add up.
If the timeout period passes, and the container still hasn’t managed to exit on its own, Docker sends a SIGKILL signal, which causes the process to be stopped by the operating system immediately.
A Problem Of Communication
What causes a process to use up all of the timeout after getting a SIGTERM? There are four options:
- The graceful shutdown just takes a long time.
- The process would like to get another signal than SIGTERM.
- The process gets the signal, but ignores it. (Bad process. Bad!)
- The process doesn’t receive ANY signals.
The first case needs some fixing. Either by optimizing the app or by extending the timeout. You can configure this using the docker-compose stop_grace_period. If that’s the case, your waiting issue won’t be fixed, but is the right thing to do.
Nginx is an example for the second option - it would much rather get a SIGQUIT signal. You can configure the right signal for your dockerized application via the stop_signal or directly in the Dockerfile. This is the exception - most apps should react to SIGTERM and SIGINT at least.
With option 3, your dockerized process simply ignores the signal. Handling signals is quite an important task to be a good process-citizen. Not being able to do so should be seen as a bug.
Option 4 is most likely however. The most probable cause, is that your app simply doesn’t get the signals.
Where Are My Signals?
Chances are, an unintended shell process has them. This can be caused by multiple small mistakes.
If you have defined your ENTRYPOINT (or CMD, you should care about the difference), in shell form instead of exec form, there’s a /bin/sh -c running as PID 1 in your container. That shell process does not pass any signals to your actual command. Check out this table for an overview how different CMD and ENTRYPOINT formats interact with each other, and watch out for the /bin/sh -c part.
# Shell form - DANGER! This becomes "/bin/sh -c your_command"
ENTRYPOINT "your_command"
# Exec form - GOOD! No "/bin/sh -c" is added
ENTRYPOINT ["your_command"]
Another possible cause, is if you run your process from an entrypoint script without using exec. With exec, your process “takes the place” and gets all signals, otherwise they are received and kept by the entrypoint script.
# Run the long-running command in your script with exec:
exec your_command
If your entrypoint script is running continuously, with a while loop for example, you should tell it how to react to incoming signals before starting to loop.
# Run the "exit" command when receiving SIGNAL
trap "exit" SIGINT
trap "exit" SIGTERM
You should also consider to use an init system inside of your containers - like dumb-init, tini or by using the init: true docker-compose option on newer Docker versions (nothing magical here, tini is used by Docker by default with this option). Handling signals and processes as PID 1 properly has many edge cases, and using a valid init in your containers is a great idea.
In Conclusion
Handling signals will make your containers respond to the request of being stopped. If the graceful shutdown of your docker-compose services takes at least 10 seconds, you might want to investigate whether you might be ignoring signals like SIGTERM.
By avoiding the above pitfalls, you’ll make your experience of interacting with your containers nicer. Less waiting time, less surprises through ungraceful shutdowns and another step towards using Docker right.
I hope this articles has helped you diagnose the cause of your long waiting times, and that you will be able to speed up your workflows with this knowledge.