I've multiple azkaban flows in a project. Most of the commands are python scripts and all of them are running in a scheduled way. job nodes look like this :
nodes:
- name: Start
type: command
config:
command: printenv
- name: Install Python Requirements
type: command
config:
command: pip3 install -r requirements.txt
dependsOn:
- Start
- name: CustomLogic
type: command
config:
command: python3 test.py
dependsOn:
- Install Python Requirements
There are intermittent errors in the scheduled executions where the second step fails. I was wondering it might be because i'm using global python in my commands.
Is there a nice extensible way i can spawn a new python virtualenv before all my executions and run them in it?