Deployment Properties You Want And Tasks You Should be Able to Do

Your web app is shipped, online and running smoothly. You had some struggle at first, before choosing what platform and tech to go with, and the learning curve was a bit harsh. Now you’re still a bit unsure - did you miss something?

Will the app go ka-boosh in the middle of the night, and you’ll be wondering what happened in the morning? What if there’re too many users all at once? Did you forget to take something into account which will bite you later? Is there something you should be doing every week, or else? No matter what way you chose to bring your web application online - there’s always a fixed set of tasks which you should be able to do, and things you should have in place, no matter whether it’s a single server, or a fancy scalable setup.

Here is a list of those concerns, so you can check if you haven’t missed something important, together with ways to implement them.

Ship New Versions of Your App

The steps needed to do when you have just improved your app, and want to share it with your users. The app is ready, so the only thing standing between shipping and you, depends on your deployment processes. You don’t want to have to break a leg each time, guess at the commands needed and be nervous that everything breaks. If you have tests, you want to make sure they are still passing, if there’s changes to the data models you want to be sure that they are applied, will not break everything and leave the old data intact.

If you’re feeling nervous when deploying because of the tech, you’re not doing it right. Also, you’ll want to have a way to go back to an older version of the code if needed, given that bad comes to worse.

Be Sure Your App is Up And Functional

In the short-term, you want to know whether the last shipped version does the things it’s supposed to be doing. What people usually do once they deploy their app, is to browse to the site, and test the basic and newest functionality manually. Clicking around nervously, hoping that nothing’s broken.

Wouldn’t it be good to be pretty-darn-sure everything is still working, and save the minutes wasted on exciting, hasty smoke tests? Having a staging environment, to deploy changes before production and having automated tests set up will tell you that not everything is broken.

In the long-term, you should be the first to know when your site is unavailable, or not really responsive. Without checking every hour or so, that is. This can be achieved by having monitoring in place, with sane alerts which go off if there’s a problem. You will get an email from a friendly bot when problems start to occur, and not from angry customers.

Make Sure Your Services Survive Reboots

In case of a single server, you should have configurations in place which help your services restart after a reboot of the machine. The first step in this direction, is to have a section in the deployment readme, specifying which commands need to be executed to bring up your deployment from scratch. Once that’s in place, the information can be translated into supervisord and systemd configs.

Be Up to Date

Operating system packages become outdated - usually when there’s a good reason for them to be, such as nasty bugs and vulnerabilities leading to all kinds of issues if they’d be left unattended to. You should have a system in place, to automatically install security updates, or be notified that you should take care of them and do so in time. Apart from services, this also applies to the frameworks and libraries you’re using to develop your application.

Be Notified When There Are Issues With Your App

This one is a bit like monitoring, but not quite. While the above is for letting you know that the app is unavailable or slow to respond, this point is for letting you know when errors occur and what caused them. If you’re not yet using Sentry to get crash reports with all the details you need, do check it out.

Be Able to Make Backups. Do It Regularly

Have you considered what happens, if your current deployment setup simply disappears overnight? Maybe due to an attacker or one of your scripts executed by accident. Would you be able to restore your setup? What about configurations? Most important - would your users lose more than a day of data? In case your app gets compromised, or simply goes “poof”, you should have a recent snapshot of the database and everything else which is not easy to restore at hands.

Having offsite backups is a good idea. You can use your development machine, a set of external hard drives or a paid service. Anything is better than nothing. If you’re using AWS and making backups to S3 which are owned by the same account as the one which runs your deployment - you might want to reconsider. There’s a reason why more prudent companies have two independent AWS accounts, one of which only receives backups to S3.

Restore From Backups

Have you tried to restore data from your backups recently? If not, how do you know that they’re not completely broken? You should check your backups regularly, and make very sure that they are sufficient to bring up a production setup without major data loss. You should have a procedure in place, describing exactly what steps need to be performed.

Collect Application Logs And Make Use of Them

Logs are great. Talk to anybody who has worked with building in-depth analytics for a larger web application - they can be a great source for detailed data on how your users are interacting with your application. You can’t get this level of detail simply from your drop-in analytics solution or by looking at data in the operational database.

That is, if you’ve put any effort into what you are logging and how it’s stored. You should make sure, that your application logs are useful. In addition, you should know what you’re logging stuff for. What’s the point of writing out information which nobody is going to look at anyway?

Once your logs are useful, make sure they are preserved regularly and are not rotated out of existence.

Execute Admin Commands For Your Application

If you just started deploying your app, you might have missed this point. Eventually, you’ll need to make changes to the database or write useful administration commands.

Those can be used to perform database migrations, export data for analytics or to your CRM, loading data into the app, or computing derived data every once in a while.

You should be able to interact with your app via a command line, or through a custom admin interface in your app at least. If it’s a regular task, you might want to run something like cron to make it automatically recurring.

Monitor Stats, to Notice Issues and Prevent Them

Hosting a web app, is a bit like gardening. Sometimes you just go out there, and take a look to see if everything is in order, or if you’re missing an issue. If you notice the signs early, you may fix it before it causes a mess or at least start preparing.

You may have alerts in place, and are monitoring metrics like the response time or whether the app responds at all.

Taking an interested look at stuff you are not keeping a keen eye on, and look at weekly or daily trends and outliers is worth a few minutes of your time at least. This will help you spot issues before they occur (such as running low on disk space, because of Docker). You can also take the time to reevaluate your alerting game, and set up new ones which you have missed until then.

Lower The Entry Barrier

This one is not really a feature of your deployment setup, but a point which you will be very grateful eventually. You should document all your tasks, be it regular or one-off during initial setup. This way, future-you can execute them without surprises and re-discovering all the pitfalls once again. This will come in very handy to help other people start working on the project.

The next step of having docs in place, is making them automatic via code. In case of a development environment, this might be a Vagrant machine which is provisioned using Salt or Ansible, so a new dev env can be brought up from scratch with a single command on a new machine.

To make it convenient to join the project development, tests are even more important than a reproducible dev environment. Otherwise, there’s no easy way to tell if your latest commit just broke something important in the app.

Conclusion

I hope that this list will help you evaluate your current deployment setup, and take note what can be improved in the future. If you want to go through all of those points in a structured fashion and make notes - drop me your email address below, and you’ll get a handy checklist with room for notes.