After finishing the first version of your Django project, it’s just a question of time until you stumble into things you didn’t prepare for. Those are real-world challenges which are usually not covered in beginner tutorials. However, all successful Django projects need to deal with these topics sooner or later.
Let’s talk about eleven of these challenges. Knowing about those will not resolve them automatically, but you’ll have a bit more time to prepare instead of being disappointed that you didn’t know about them at an earlier point in time.
Deploying Your Project
Obvious, right? But it’s still a surprising hurdle for many.
My Django project is almost done - now, how do I deploy it?
Bringing your Django project online can come as a surprisingly difficult task.
You can prepare for it by keeping the eventual deployment in mind while building your application. Once it’s time to deploy our project, it’s okay to go with a straightforward option, where you don’t have to jump through too many hoops. Heroku can be an excellent choice, and there’s plenty of successful & big projects running on it.
To get some basic knowledge in, you can read up on the 12 factor app, which provides great guidelines for modern web application architecture.
Automating the Deployment Process
Bringing your app online can feel like a big task. But you’ll want to be able to deploy your application without making it a big deal in the long run.
If your project is a success, you’ll keep working on it. You’ll be deploying new features and bug fixes. Imagine spending 15 minutes each time, and getting a bit nervous that you might mess up. That’s not a thing you want to be doing over the long haul.
Deploying manually is error-prone and can be a very tedious task. One of your goals should be, to make deployments as boring and uneventful as possible.
Automating your deployment workflow more and more is a good use of your time if you’re working with an active project. Check out this xkcd if you want for a rough estimate of how much time you can save. If you’re wondering where to start, here’s a rough action plan:
- Figure out how to deploy manually
- Write down instructions for yourself about the process
- Ask yourself: are you happy with your current process? Now’s a good time to tweak steps
- Try to deploy based on those instructions. Is anything missing?
- Automate a small part of it using a .sh script, or take a look at Fabric
- Try to use the new automation, and enjoy the saved time
- Add the script-usage to your instructions and continue using them together
- Reiterate if needed
Know When to Upgrade Dependencies
Once your project is online, you’ll want to keep it up-to-date. But how often should you take care of it? When do you need to upgrade system dependencies, Python dependencies, do security updates or upgrade the Django version?
There’s tooling which can notify you and automate dependency-bumping. You’ll want to have tests in place to be sure that any given upgrade did not break something without you noticing.
Backups
If your data is of any importance, you’ll want to be doing regular backups. But only creating backups is not enough - you need to use those backups on a regular basis as well.
Restoring your latest backup into a fresh environment on a regular schedule is the only way to be sure that your backups are any good.
If you’re haven’t tried restoring from your backups in a while, you can’t be sure that you have usable backups. Part of this topic, is deciding on a backup scheme. Working with them regularly is an important part of being responsible for a deployed web application.
If the thought of restoring from your backups feels like a lot of work, you might want to rethink your use of automation - both for your infrastructure and your maintenance processes.
Developer Onboarding
You’ll notice the need for this, once people start joining your team. Setting up a development environment quickly, and giving people everything they need to start working on the project should not take up hours of your time.
There’s always at least a bit of arcane knowledge which isn’t mentioned anywhere.
Oh yeah, to do X you just need to do A B C but watch out for Y
How easy is it to go from cloning the project’s repository to a working development setup? Are there proper docs in place? Are steps missing? Do they require arcane knowledge and sitting together for an hour trying to come up with the right commands? Have you decided on tooling for linting, automated tests, project style guides?
Observability
When your project is online, you’ll want to know whether everything is working as expected. Monitoring, logging and some error-alerting are what you can rely on for this.
This is not something you do once, and are done with. Working on observability is a constant and iterative effort. Depending on your project, Being proactive is a pretty good idea.
If something breaks, you’ll want to be notified, and be able to find out what happened. In the best case, you have monitoring in place to figure out future issues and prepare for them in advance - before they become a big deal and lead to unexpected downtimes.
Improve Performance Where It Matters
Does your application have performance issues?
Basic observability is a requirement. You can’t dig into this topic properly without it. Trying to speed up random aspects, without having diagnosed the issue in the first place, likely boils down to premature optimization.
However, once you find out what your bottlenecks are, you’ll be able to choose the right approach to address them. There’s a lot you can do from here, but most likely it’ll be:
- Fix plain-broken ORM queries (hello prefetching)
- Provisioning more resources
- Add caching in the right places
- Tweaking and optimizing your ORM queries
Worker Processes
Eventually, you’ll want to add regular or long-running tasks to your project. After all, requests to your application are under time pressure.
To take care of that, worker processes are your friends. Celery is a cool tool to take some load off of your application server, provide some reliability to your deployment and make it possible to trigger long-running tasks.
The S Word
Nooo, don’t make yourself nervous. You probably won’t have to deal with this the way you think. Most applications never have to go to scale. Basic performance tweaks are all you’ll probably need.
Scaling is a nice-to-have problem. Way too many people think they need to take care of it way too early in the lifecycle of a project. The result is second-guessing everything and choice paralysis because you want to know “but will this scale” without really knowing what your requirements are.
If you’re lucky enough to build something which needs scaling, you’ll figure it out.
You can prepare in advance by making sure that you’re following architecture best-practices. See the the 12 factor app guidelines. Don’t put too much effort into this topic from the get-go. A bit of vertical scaling (getting a more powerful server) will buy you enough time, if you ever need it.
Rewriting the Whole Thing
If only I could start from scratch.
Resisting the urge to rewrite the project can become challenging as the codebase grows.
The project starts out nice and clean. Then business needs, user requests, and time constraints happen. The structure grows to be messy, there are parts which were never expected to be created and things feel out of control.
That’s normal unfortunately! Business requirements are not completely known from the get-go. You build something which you believe to be right, but you’re working with incomplete data while the business changes. New information is uncovered and the business evolves. The real world is messy.
You’re not the first person to stumble into this. The best thing to do, is to look at how other companies have handled the “complete rewrite” topic, and learn from their experience: I can’t recommend this article by Herb Caudill enough.
Most probably, you should stick to refactoring your now-legacy project.
Recover From Catastrophic Failure
This one goes hand-in-hand with automation and having backups.
Being able to recover from catastrophic failure is something your project hopefully will never need, but it’s good to be prepared in advance.
You’ll need to be able to rely on your backups, and you’ll be happy if most of your deployment / infrastructure setup is either very well documented or preferably automated.
You can test your procedures, by bringing up a completely independent production-like environment every once in a while. Just in case.
In Conclusion
Those were eleven real-world challenges which every successful Django project eventually will encounter.
Knowing about them in advance can help you to level-up your Django skills, and prepare for those things which are usually left out of common tutorials.