AzureML Studio Prompt Flow- "No Module named 'azureml.fsspec'

Question

I am trying to work with file data in Azure ML prompt flow and resorted to this article:

Access data from Azure cloud storage during interactive development

It said to run this command in my runtime compute instance to install azureml.fsspec and mltable: pip install -U azureml-fsspec mltable

After everything was installed, I kept getting this error whenever I ran the python prompt flow step that had fsspec

===============================

I actually had to install fsspec and mltable with specific versions: fsspec 1.0.0 and mltable 1.3.0. This is because the updated versions doesn't work with the azureml-dataprep library together. This is what happened with the updated versions:

ERROR: azureml-dataset-runtime 1.51.0 has requirement azureml-dataprep<4.11.0a,>=4.10.0a, but you'll have azureml-dataprep 4.11.1 which is incompatible.

Or these two errors:

ERROR: mltable 1.4.1 has requirement azureml-dataprep[parquet]<4.12.0a,>=4.11.1, but you'll have azureml-dataprep 4.10.9 which is incompatible.
ERROR: azureml-fsspec 1.1.1 has requirement azureml-dataprep<4.12.0a,>=4.11.1, but you'll have azureml-dataprep 4.10.9 which is incompatible.

So apparently dataprep version has to be <4.11 for dataset-runtime but 4.11.1 for fsspec and mltable.

=============================

So I have fixed that issue by using older versions of azureml-fsspec and mltable but I still get the same error saying that there's no fsspec module whenever I run the python prompt flow step.

Has anyone ever run into this issue or know how to solve it?

Answer 1

Based on the provided information, I have reproduced the issue.

When we are creating runtime for Prompt flow it will assign a default runtime environment which doesn't contain the required package.

To solve this, you can install the packages using subprocess.

import subprocess
package_name = "azureml-fsspec"
subprocess.check_call(["python", "-m", "pip", "install", package_name])

With this I was able to resolve the issue.

Another possible solution is to use custom environment with required packages. For more details, please refer to this documentation.