How to use evironment variables to set cron like schedule using apscheduler?

Question

I have put together a ETL job written in Python. I have to run it through different environments (LOCAL, DEV, TST, PROD) so I need to find a way to set the job schedule using environment variables so that I can set different schedules for each environment without touching the code from one environment to another. Today I am using apscheduler. I have something like this in the code:

from apscheduler.schedulers.blocking import BlockingScheduler

def etl_job():


sched = BlockingScheduler(daemon=True)
sched.add_job(etl_job,'interval',minutes=7)
sched.start()

Does anyone know now to solve this? Thanks!

Answer 1

Here how I've solved this one:

Environment variable (json format)

schedule='{"jobsSchedule": [{"job_name": "job_1","schedule": "*/1 * * * *"},{"job_name":"job_2","schedule": "*/2 * * * *"}]}'

Read the env variable

jobCrontabSchedule = json.loads(os.getenv('schedule'))

Schedule each one of the jobs

for job in jobCrontabSchedule['jobsSchedule']:
    print ("Scheduling the job: "+job['job_name']+" with the schedule: "+job['schedule'])   
    sched.add_job(globals()[job['job_name']], CronTrigger.from_crontab(job['schedule']))

Answer 2

Sample of setting an env variable and reading it with python (I assume this is what you're asking).

export INTERVAL=5

import os
print(os.environ.get("INTERVAL"))

However, few things you may want to remember:

• Setting environment variables from one OS to another will be different
• Make sure you are setting the variable for the same user you will run your process with
• Make sure you provision migrations of these variables as you move across environments