python virtual env specific to every execution of an azkaban flow

141 Views Asked by At

I've multiple azkaban flows in a project. Most of the commands are python scripts and all of them are running in a scheduled way. job nodes look like this :

nodes:
  - name: Start
    type: command
    config:
      command: printenv

  - name: Install Python Requirements
    type: command
    config:
      command: pip3 install -r requirements.txt
    dependsOn:
      - Start

  - name: CustomLogic
    type: command
    config:
      command: python3 test.py
    dependsOn:
      - Install Python Requirements

There are intermittent errors in the scheduled executions where the second step fails. I was wondering it might be because i'm using global python in my commands.

Is there a nice extensible way i can spawn a new python virtualenv before all my executions and run them in it?

0

There are 0 best solutions below