Azure Machine Learning SDK V2 changer version of python in Compute Cluster

Question

In SDK V1, I am using a environment for compute cluster with a dockerfile string like this:

azureml_env = Environment("my_experiment")
azureml_env.python.conda_dependencies = CondaDependencies.create(
    pip_packages=["pandas", "databricks-connect==10.4"],
)
dockerfile = rf"""
FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest
RUN mkdir -p /usr/share/man/man1
RUN apt-get -y update \
&& apt-get install  openjdk-19-jdk -y \
&& rm -rf /var/lib/apt/lists/*
"""
azureml_env.docker.base_image = None
azureml_env.docker.base_dockerfile = dockerfile

So I am using mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest where it gives me a python 3.8.

But when I switch to SDK V2, I get a python 3.10, which is not compatible with my databricks runtime that need python 3.8.

Here is my dockerfile:

FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest

RUN mkdir -p /usr/share/man/man1

RUN apt-get -y update \
    && apt-get install  openjdk-19-jdk -y \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install -r requirements.txt && rm requirements.txt

# set command
CMD ["bash"]

I call it like this in python:

azureml_env = Environment(
    build=BuildContext(
        path="deploy/utils/docker_context", # Where is my dockerfile and other file to copy inside
    ),
    name="my_experiment",
)
azureml_env.validate()
self.ml_client.environments.create_or_update(azureml_env)

Why don't I get a python 3.8 but a python 3.10?

Answer 1

You cannot provide a conda.yaml file when using a Docker build context. Thus, you need to create a conda environment and a Dockerfile as shown below:

FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest

WORKDIR /

ENV CONDA_PREFIX=/azureml-envs/sklearn-1.0
ENV CONDA_DEFAULT_ENV=$CONDA_PREFIX
ENV PATH=$CONDA_PREFIX/bin:$PATH

# This is needed for MPI to locate libpython
ENV LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Create conda environment
COPY conda_dependencies.yaml .
RUN conda env create -p $CONDA_PREFIX -f conda_dependencies.yaml -q && \
    rm conda_dependencies.yaml && \
    conda run -p $CONDA_PREFIX pip cache purge && \
    conda clean -a -y

RUN mkdir -p /usr/share/man/man1

RUN apt-get -y update \
    && apt-get install openjdk-19-jdk -y \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install -r requirements.txt && rm requirements.txt

# Set command
CMD ["bash"]

Here, I am creating a conda environment first with Python version 3.8 and running the remaining commands.

conda_dependencies.yaml

name: pydata-example
channels:
  - conda-forge
dependencies:
  - python=3.8
  - pip=21.2.4
  - pip:
    - numpy==1.22
    - scipy==1.7.1
    - pandas==1.3.0
    - scikit-learn==0.24.2
    - adlfs==2021.9.1
    - fsspec==2021.8.1

env_docker_context = Environment(
    build=BuildContext(path="docker-contexts/tst"),
    name="docker-context-example-1",
    description="Environment created from a Docker context."
)
ml_client.environments.create_or_update(env_docker_context)

Output:

And in the environment:

If you want to avoid these commands, you can try using other images with Python 3.8.