pandas FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version

Question

In order to print dataframes nicely using tabulate, so that NaN and NaT are printed as empty cells, I've been using this successfully:

print(tabulate(df.astype(object).fillna("")))

Now, this causes the following warning:

FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead.

I don't know what I should do instead now. I certainly don't see how infer_objects(copy=False) would help as the whole point here is indeed to force converting everything to a string representation and filling in missing values with empty strings.

Answer 1

I encountered the same issue today and I'm still looking for a fix myself. In your case, I must admit I don't know what calling astype(object) twice does. However, as the error message mentions, you can try avoiding this and cast your data to strings instead (if that's feasible for the data you're handling). Then, you can use pd.DataFrame.replace like so:

df.astype(str).replace(to_replace=["nan", "None"], value=(""))

Of course, you probably want to adapt to_replace to include the NaN and NaT you're searching for. On my system, astype(str) converts NaN to nan, so it might be similar for you and NaT.

Answer 2

Convert the DataFrame/Series type first

Example:

df.astype(float).fillna(value)

Infer the objects' type with infer_objects

df.infer_objects(copy=False).fillna(value)

Where value is a compatible type of the inferred objects:

---

Setting pandas.set_option("future.no_silent_downcasting", True) seems to work to remove the warning too, but I don't know if this is the correct behavior (as also pointed out by @mrgou in the comments).

---

Explanation

The arrays are of the type object and, when you call fillna, it tries to infer the objects' type which issues this warning.

Answer 3

I just was in a similar situation. This line of code has worked for years and I just now find out its being deprecated.

df = pd.read_excel(excel,dtype='str')
df.replace({r'\n': ' '}, regex=True)

I had to update it to this:

df = pd.read_excel(excel,dtype='str')
df.astype(str).replace(to_replace=['\n', ' '], value=(''))

As I had received this warning:

FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed 
in a future version. To retain the old behavior, explicitly call 
`result.infer_objects(copy=False)`. To opt-in to the future behavior, set 
`pd.set_option('future.no_silent_downcasting', True)`
  df = df.replace({r'\n': ' '}, regex=True)

Answer 4

In case of following distinct examples:

ser1 = pd.Series([False, True, float("nan")])
ser1 = ser1.fillna(False)

and

ser2 = pd.Series(["Zero", 5, 2.3])
ser2 = ser2.replace("Zero", 0)

the use of an option context combined with infer_objects at the end seems to be the most generic solution to get rid of the FutureWarning:

with pd.option_context("future.no_silent_downcasting", True):
    ser1 = ser1.fillna(False).infer_objects(copy=False)

and

with pd.option_context("future.no_silent_downcasting", True):
    ser2 = ser2.replace("Zero", 0).infer_objects(copy=False)

Probably better is to be more specific and use astype(bool) and astype(float) instead of infer_objects(copy=False) in the above.

Remark that other proposed solutions don't work in this case:

1. The use of infer_objects(copy=False) before fillna or replace:

ser1.infer_objects(copy=False).fillna(False)
ser2.infer_objects(copy=False).replace("Zero", 0)

doesn't get rid of the FutureWarning.

2. The use of astype before fillna or replace is even more dangerous as it returns the wrong result for the first example:

ser1.astype(bool).fillna(False)

and raises a ValueError for the second example:

ser2.astype(float).replace("Zero", 0)

3. I would not recommend setting pandas.set_option("future.no_silent_downcasting", True) as this may hide issues elsewhere.