Congratulations! You’re finally ready to automate a part of your deployment pipeline.
Did the team size hit that magical 3-dev number? Is the product you’re working on ready to become more than a prototype?
Or do you just really really want to switch away from Jenkins?
In any case, you’re ready to move on, and there’s a lot to gain from doing so. Go you!
But where to start?
There’s more tools out there than you can list, let alone learn about, and doing so is confusing as hell.
Most tools will be very eager to convince you that, yes, of course, you should give them a try, why not. Also, sometimes it’s not really obvious that a tool will be a good fit for the tasks you have in mind.
So do you just have to try each and see if they work for you?
You can do better
The good news is, you don’t have to try all of them. Not even most. Of course, you could just pick one and give it a try, until you’re happy with what you see, but you can do better by doing an approximate evaluation first.
The trick is not to try to be convinced by each tool’s landingpage and list of features, but to see if it’s good in what’s important for you.
First, you have to make up your mind: what do you need right now?
What do you want to achieve?
That really depends on you, your team, your tech stack and your company.
Just as with jobs, there’s a lot of possible things which might be important to one person, but wouldn’t really matter to another.
To help you choose the right things to think about: here’s a list of questions you can consider, and reasons why they might be important for you.
Read through them, and pick three to five.
1. What’s the usecase this tool is designed for?
If you can’t answer this, the tool might be not sure what it’s good for itself. Or the documentation/marketing page are doing a bad job.
Don’t think about features the tool has, but the problem it’s meant to solve.
Is it one problem, or multiple ones bundled together? Are you experiencing the problem it’s communicating right now?
2. Does the tool do one thing?
Similar to the first question, but not quite.
Some tools, are made with a very specific task mind. They are opinionated, and want to be used in a certain way.
Others can be used for any task, but you’ll have to do more thinking and tweaking. I tend to prefer specialized tools, as they usually do the thing they’re designed to do, and do it well without needing major setup efforts.
If you’re looking for a tool to run unit tests and build Docker images, you might want to prefer something specialized, rather than a generic task runner.
3. Has it been in use for a while?
Do you know of other companies, with a similar size, who are using this tool? Could you think of specific people to get in touch?
If something has been around, it can be a nice validation. There will be less bugs for you to find, and it will probably stick around for a bit more.
Something which has been around for too long however might have collected a bit of cruft.
4. Is the interface nice?
What will developers see and interact with in their day-to-day work? Some tools are frustrating to navigate, and not a pleasure to use.
This one might seem shallow, but seriously consider the future user experience.
Does your team value well-designed software, which offers exactly the information needed at one glance?
5. Can you configure it with code?
Will you need a GUI to interact with it? Does it offer a straightforward way to be configured without using the mouse and arcane click rituals?
How much work will it take to automate it?
I prefer tools which can be brought up with a single command (using a mix of Terraform, Ansible and YAML files?) and only use GUI for displaying results & simple user interactions.
6. What about complexity?
Is it a complex conglomerate of 5 tools working together under the hood?
Does it do a lot of magic, which is nice to have in the beginning but a nightmare to fix once things get a little more complicated?
If the solution has a lot of moving parts, you should be ready to understand each of them to make it work once it breaks.
7. What about documentation?
Are the docs well structured and easy to navigate?
What about other sources of information? Is it easy to find information and solutions to common issues on the web? Will you need to dive into the source code eventually?
8. Is the tool designed for interacting with publicly-facing system?
This could be a major dealbreaker if you want to use the tool for internal-only systems.
Do you really want a hosted tool to get credentials and access to internal parts of your infrastructure?
9. Does your team know how to use it already?
This one is easily discarded, but can be a great advantage. Is there a technology many people on your team know how to use? Have you used something in previous projects? You could make use of your existing expertise.
What’s the team (or one of the teams) already using and comfortable with? Maybe you don’t need to shop around if there’s an already-adopted solution. At the very least, you can learn more about what’s important by asking questions about their experience.
Choosing the newest, fanciest thing out there can be very tempting. But do you need to? In the end, you want to have something which you can make work reliably.
10. Can it be configured with YAML?
YAML is currently in favor for all kinds of configuration tasks, as it easy to read and convenient to write. Many people (who have had to write YAML for a while) would strongly disagree with this sentiment however.
In the end it’s a question of taste, and YAML is one of the lesser evils. Your team will probably prefer YAML over XML, JSON or a DSL they haven’t used yet when it comes to configs.
In the case of CI tools, it’s a convenient thing to have your pipeline defined as code along your project. GitLab CI/CD uses gitlabci.yml files. Jenkins, on the other hand, can be configured to use Jenkinsfiles per repo, which need to be written in Groovy.
11. Is it designed for containers?
Are you relying on Docker for your dev and deployment workflows?
If you’d like to spin up your setup with docker-compose to run integration tests, you’d best pick a tool which supports this out of the box.
Newer tools tend to be designed with currently-in workflows in mind.
12. Does it fit your tech stack?
Not just the languages you are developing in, but also the platform you’re deploying on and the IDEs your team is using.
If you’re developing Java applications using IntelliJ IDEA, you should seriously consider TeamCity. There’s a lot of potential for happy synergies if you choose a tool which fits the rest of your stack.
Do you want to deploy to Kubernetes? A k8s-first tool might be the best choice for you.
13. Does it play well with your other tools?
Where do you host your code? If you’re on GitHub, some tools really want you to use them. If you’re self-hosting a GitLab repository, you might not get a smooth experience.
14. Will scale be an issue for you?
A question which is most likely pretty much irrelevant for you. That’s good, right? One less thing to worry about.
This consideration only should come up if you’re working with more than a dozen of teams on hundreds of projects. Running lots of simultaneous build jobs can become an issue, and architecture choices really matter depending on your workflows.
If you are not a big-ish company, you won’t have those kinds of issues.
15. Can we host it on-prem?
This is really important to companies which are working with sensitive data and under regulation constraints.
You won’t be able to use a hosted service, and the ability to run something in-house without breaking a leg can be the most significant factor to consider.
16. What about license fees?
Tools can get very pricey when your company grows. If your team is small and won’t grow explosively you will be fine.
But what about when you need more users or task runners? Could you be better-off hosting something for yourself with a permissive company license?
By considering growth issues early and choosing accordingly, you can cut your future spending by 10x in some cases.
17. Will you be able to customize it for your needs?
If the tools is open-source? How easy is it to contribute to it, request features or write plugins?
18. Is it easy to throw money at the problem?
Is it an open source tool, which is backed by a company? Can you get convenient paid support and a flexible license?
If you’re well funded, on what matters, don’t lose momentum on tasks which are not central to your company
19. What about multi-tenancy?
If you have several teams which are working independently of each other, and should not disrupt other’s work, you might need to host one and the same tool several times independently.
Will it be a divergent GUI click-fest, or can you create something which is easy to reproduce and automate?
20. How much effort will it require?
Can you host it on your current infrastructure? Is it a hosted solution?
Will you need to put constant effort into scaling, updating and other maintenance tasks? Will everybody need to read the docs to use it?
Time and attention are very precious resources to invest into a tool which is not central to your business.
Next steps
Have you picked three to five questions which matter most to you? Did you come up with one which is not listed? Great!
Now that you have your questions, you can try to answer them for the tools you currently have in mind. Funny enough, it’s easier to collect information when you’re looking for a specific piece of it.
You will spend less time “getting convinced” by each tool and can focus on what really matters to you.
You don’t need to find the perfect one. You can move towards a closer evaluation round once you have found two tools which tick most of the boxes you chose, and you’ll be just fine.
I hope you can use this article to save time, make a decision which works for you and move beyond analysis paralysis towards working on what will make a difference for your company!