Azure ML Cluster: changing folders during job

Question

I am working with Azure ML to train and test an ML model. I have a main script which calls different functions (.py files in different subfolders) to load data, preprocess, train and test model. While running it in VSCode interactive window it is able to change the folders and find the files. But, when I try to send it as a job on a compute target it is unable to find the folder.

While running in VSCode it is able to access: '/mnt/batch/tasks/shared/LS_root/mounts/clusters/<cluster_instance_name>/code/Users/<user_name>/<dir_name>'

While running as a job on a compute target it is in the following directory: '/mnt/azureml/cr/j//exe/wd' As it is in an entirely different folder it cant access the function scripts.

My question is, how can I access the function files when I send the main script as job?

I have tried the following:

1. I have tried changing the directory in the main script to where the function folders lie but it cant access the folder during the job.
2. I have tried checking all the folders present in the directory '/mnt/azureml/cr/j//exe/wd' but it contains only the following folders: outputs, main_script.py, user_logs, azureml_logs, logs
3. In the job command, for the code attribute filled in the root directory containing all the files but it takes a lot of time to create the job.

I believe that there should be a a solution with option 3. Else I will have to have all the function files in the same folder as the main script.

Answer 1

In command job you can provide code directory with your scripts including main script and other function scripts. Below is sample code for reference:

job = command(
    code="./src",  
    command="pip install -r requirements.txt && python main.py --iris-csv ${{inputs.iris_csv}} --epochs ${{inputs.epochs}} --lr ${{inputs.lr}}",
    inputs={
        "iris_csv": Input(
            type="uri_file",
            path="https://azuremlexamples.blob.core.windows.net/datasets/iris.csv",
        ),
        "epochs": 10,
        "lr": 0.1,
    },
    compute="cpu-cluster",
    environment="AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest",
    display_name="pytorch-iris-example-with-function",
    description="Running with Function",
)

All the required files are under src directory.

 |-src
 | |-function
 | | |-network.py
 | |-main.py
 | |-requirements.txt

once you create the job, it will upload all the code file in src directory.

With above setup, I was able to execute job with main script calling function from other .py file.

For detailed sample code, you can check this notebook.