What does it take to make your containerized application easier to operate in production? What can help to keep them running and make them more reliable?
Here are a few things to look out for, which will make your container ops job easier, both inside and outside of your running containers.
Output Logs To Stdout
Most logging solutions for containers are built around the assumption, that log s are written to stdout. That’s the stuff you see when you run docker logs CONTAINER_ID_OR_NAME.
Sometimes, non-containerized applications prefer to write to disk. That makes it a little more tedious to configure them to play nicely with common approaches.
More Observability
Monitoring is useful! When running containers, you can get a lot of stuff out of the box with the right setup - how much memory each container is using, how CPU hungry it is, the network traffic and much more.
If you have application-specific metrics you’d like to collect over time, that’s usually done by providing a http endpoint which can be scraped regularly by tools like Prometheus. They collect that exposed data, and make it accessible for other tools like Grafana, which make it possible to display cool graphs and dashboards based on the available metrics.
Make Sure Your Containers React To Signals
Badly configured containers don’t care about signals. They either are not configured to receive them (sometimes by a hungry shell process swallowing them), or they simply are ignored. Just one of many things to watch out for when building Docker images. (But it’s fixable.)
That makes it really hard for orchestrators to deal with this kind of containers! Well, maybe not exactly hard but tedious. Imagine telling a container nicely that it needs to terminate and has time to clean up. Instead of the container follewing suit quickly, the orchestrator needs to wait for a complete (long!) termination grace period just to send a kill signal by the end.
Reacting to signals is a necessary prerequisite for a well-configured container.
Expect That Your Containers Will Need To Terminate
If deployed on a cluster with multiple nodes, you containers should expect that they won’t stick around forever. For a multitude of reasons, even on a single host, your containerized application might need to shut down or is just stopped suddenly. It may get a friendly heads-up warning, but it’s not guaranteed.
Make sure that you take into account the volatile environment containers exist in, and that most tooling around them is not made to treat them too gently.
Make Your Containers Configurable
Hard-coding values of a containerized application, and assuming they’ll always be this way makes it hard to use the same images across different environments.
Using environment variables to configure application, where possible (well, except for really sensitive secrets), is something which makes a containerized application easier to operate.
While it’s possible to pass a configuration file, environment variables are currently one of the most frequently used way to pass configuration values to web applications.
Make sure your containerized application handles them if possible.
One Service Per Container
The rule of thumb is that it’s okay to group non-loosely-coupled services into a single container.
Usually, that means that you want to have a single service per container. Instead of going the VM-like (or bare-metal machine-like) approach, and packing as much as possible into a single container, you want to split up your applications into single containers.
The benefit here is, that you can operate your applications easier if there’s one per container.
When any of the applications crashes, you will notice (due to healthchecks), and can restart the whole container to try and fix the issue after a sufficient timeout.
Make Your Containers Quick To Startup
A Docker image is a great build artifact. It’s got everything you need to run it (apart from backing services) baked into it. You don’t need to download any big files, or install missing packages.
Building your images in a way, that containers are ready to run and can be started quickly is another useful trait.
Provide Healthchecks
It’s really useful to have some way to tell from outside the container, whether the application inside is ready to receive requests.
You can configure a HEALTHCHECK in your Docker image. Usually, it makes sense to configure even more fine-granular settings specific to your orchestrator of choice (k8s distinguishes between readinessProbe, startupProbe and livenessProbe for example).
Having at least a healthcheck in place makes load balancing and detecting issues a lot easier.
Use Properly Versioned Images
Using the :latest tag is the antithesis of this.
You want to have a good versioning scheme for your images. This way, you can have control over what version of a containerized application will be deployed at any given time.
If your images are not well tagged, then containers with different versions may look the same from the outside to your orchestrator. This can cause issues and strange behaviour at the most unfortunate times.
Be Up To Date
And last but not least, the contents of your containers matter as well. Security updates matter. If your image is reasonably up to date, it will be less exposed.
It’s a surprisingly frequent misconception, that you don’t need to worry about security when you start using containers.
In Conclusion
I hope this overview has helped you to get a few ideas for what you can do to make your containers more ops-friendly!
It’s a long way to go, and it’s okay to aim for incremental improvements instead of getting it exactly right on the first try. Best of luck herding those containers!