vsupalov

Create Luigi Task Dependencies Without Having to Handle Their Output

December 4, 2015

You are in a situation, where you have a Luigi task which already requires() data from another task, but should wait until a set of other tasks are done beforehands as well. No need to get on the hacky side of things!

The _requires() method can be overridden in a task, to help specify other tasks which should finish. However unlike the tasks listed in requires(), you will not have to handle the data in this case. Here is an example of using the _requires() function:

# this function is used to collapse a
# potentially multi-nested item collection (lists, ...)
from luigi import flatten

# in a Luigi Task based class
def _requires(self):
    # this method needs to return an iterable
    # which contains the _requires of the superclass
    return flatten([
        # a few tasks you'd like to run before this one but 
        # honestly don't want to use their output data here
        SomeTaskWhichYouWantToRunBeforeThisOne()
        SomeOtherTask(),
        YetAnotherTask(),
        # important! "CLASSNAME" needs to be adjusted
        super(CLASSNAME, self)._requires()
        ])

You can find the original function and documentation directly in the Luigi Github repository. The final call to the superclass _requires() function ensures that tasks which are specified in your usual requires() function are handled and nothing breaks in interesting ways.

Join the mailing list!


Subscribe to get notified about future articles and stay in touch via email.

I write about Kubernetes, Docker, automation- and deployment topics, but would also like to keep you up to date about news around the business-side of things.

Privacy and your data: You can get more information about the usage of your data, the storage of your registration, sending out mails with the US-provider ConvertKit, statistical analysis of emails sent and your possibility to unsubscribe in my Privacy Policy.

I use the US-provider ConvertKit for email automation. By clicking to submit this form, you acknowledge that the information you provide will be transferred to ConvertKit for processing in accordance with their Privacy Policy and Terms.

We won't send you spam. Unsubscribe at any time. Powered by ConvertKit