Being New to Luigi, How Do I Get Started Without Hitting Roadblocks?
You have read about Luigi and now would like to put it to use. You could just jump into it, Trying to get it to work as you need. If you do though, it might turn out that concepts you assumed to be true turn out to be not quite correct, that you have been reinventing the wheel instead of using a tried-and-tested approach or that your pipelines are broken in subtle ways and need to be reworked completely.
If you don’t have time or don’t want to commit yet: here is a quick tour through often-missed or wrongly understood concepts, check out this non-exhaustive selection of important and often neglected points. If you want to learn Luigi quickly, save yourself time down the line and use at its complete potential, please carry on. It comes down to investing a bit of time up-front and approaching the topic thoroughly.
- Read the README of the GitHub Luigi repository.
- Watch one or both of the following talks by one of original creators of the library: Elias Freider or Erik Bernhardsson to get first-hand information and an insight behind the scenes.
- Read the first part of the online documentation up to the API reference, to get in-depth information. It’s not too long and you will learn a lot.
- Set up the scheduler locally, and give it a try on some simple tasks. You can use the approach described in my “Yry Luigi With Vagrant article or head straight for the repository.
- Give the API reference a quick overview and keep the contrib part of it in mind when something seems not to be included.
- If you encounter difficulties down the line, search the official luigi-user mailing list and ask your question if nothing suitable can be found.
Join the mailing list!
Subscribe to get notified about future articles and stay in touch via email.
I write about Kubernetes, Docker, automation- and deployment topics, but would also like to keep you up to date about news around the business-side of things.