Allow a global variable modified in one class to be read by another class

38 Views Asked by At

I've inherited a series of luigi tasks in a python script, and I am attempting to do some basic logging of the execution times within the various classes, but I'm having a hard time figuring out how to do that.

The script itself contains four classes:

class Pull(luigi.Task):
class Validate(luigi.Task):
class Upload(luigi.Task):
class Finalize(luigi.Task):

Another script executes the Finalize class, and within that, in order for it to execute, the Upload class has to have finished, and for the Upload class to run, the Validate has to have finished, and for the Validate class to run, the Pull class has to have finished. All of the classes are structured in this same manner:

class Finalize(luigi.Task):

    environment = luigi.Parameter(default='dev')

    def requires(self):
        return [Upload(environment=self.environment)]

    def output(self):
        return luigi.LocalTarget(f'.out/{CUR_DATE}_{self.environment}_Finalize.txt')

    def run(self):
        # ....code goes here

What I've tried is, outside of the classes, up at the top of the page is to set up some global variables: (I've also tried defining this code block within the Finalize class itself, AND within the run function of the Finalize class. All 3 attempts yield the exact same thing that I explain below)

import time

global pull_time
global validate_time
global upload_time 
global finalize_time

pull_time   = 0
validate_time = 0
upload_time = 0
finalize_time = 0

Then within each of these classes, write data to the global var in question, like this, from the Pull function:

global pull_time 
pull_time = round(time.time() - self.start_time1, 2)
print(f"Time to pull = {pull_time}")

This seems to work, locally within this class, as it prints out Time to pull = 11.42, however my ultimate goal is to get all of these read by the Finalize class, so I can save the data into mysql and log it, but no matter what I try, I cannot seem to get these variables to have any value but 0 in the Finalize class, which currently prints out:

global finalize_time
finalize_time = round(time.time() - self.start_time4, 2)
print(f"Time to finalize = {finalize_time}")

print(f"all together now: pull = {pull_time}, validate = {validate_time}, upload = {upload_time}, finalize = {finalize_time}")

# all together now: pull = 0, validate = 0, upload = 0, finalize = 133

I am not putting global pull_time etc, within the Finalize class, as I am not modifying them - I just want their updated values, but I cannot figure out how to do that.

If there is another method by which I could pass this data in between classes, I would be fine doing that too - I don't have to do it with global variables.

0

There are 0 best solutions below