ModuleNotFoundError: No module named 'pandas.core.indexes.numeric' using Metaflow

Question

I used Metaflow to load a Dataframe. It was successfully unpickled from the artifact store, but when I try to view its index using df.index, I get an error that says ModuleNotFoundError: No module named 'pandas.core.indexes.numeric'. Why?

I've looked at other answers with similar error messages here and here, which say that this is caused by trying to unpickle a dataframe with older versions of Pandas. However, my error is slightly different, and it is not fixed by upgrading Pandas (pip install pandas -U).

Answer 1

This issue is caused by the new Pandas 2.0.0 release breaking backwards compatibility with Pandas 1.x, although I don't see this documented in the release notes. The solution is to downgrade pandas to the 1.x series: pip install "pandas<2.0.0"

Answer 2

Try using the pandas.read_pickle() method to load the file instead of the pickle module:

import pandas as pd

df = pd.read_pickle("file.pkl")

The pandas method should provide compatibility to read older files, and "is only guaranteed to be backwards compatible to pandas 0.20.3 provided the object was serialized with to_pickle." My tests with pandas-1.x show it can also read some files written from the pickle module too.

Answer 3

Try using pd.compat.pickle_compat.load() as that was only solution in my case:

import pandas as pd

df = pd.compat.pickle_compat.load('file.pkl') 

Answer 4

So i dont know why this works but joblib.load was failing to read the pickle with the same error "module named 'pandas.core.indexes.numeric'" then i installed prefect and simple_salesforce and some how it now works... not sure why but i think worth mentioning

Answer 5

try this if you using pandas > 2.0.0

Import necessary libraries:

import os
import pandas as pd

Get the absolute path to the current directory:

current_directory = os.path.abspath(os.getcwd())

Construct the absolute path to the pickle file ('file.pkl' in this case):

file_path_file = os.path.join(current_directory, 'file.pkl')

Use pandas to read the pickle file directly:

file_df = pd.read_pickle(file_path_file)