Conditional elementwise multiplication of two pandas dataframes

Question

I am struggling doing a basic conditional elementwise multiplication between two dataframes. Let's assume I have the following two dataframes:

df1 = pd.DataFrame({'A': [-0.1,0.3,-0.4, 0.8,-0.5,-0.1,0.3,-0.4, 0.8,-1.2],'B': [-0.2,0.5,0.3,-0.5,0.1,-0.2,0.5,0.3,-0.5,0.9]},index=[0, 1, 2, 3,4,5,6,7,8,9])
df2=pd.DataFrame({'C': [-0.003,0.03848,-0.04404, 0.018,-0.1515,-0.02181,0.233,-0.0044, 0.01458,-0.015],'D': [-0.0152,0.0155,0.03,-0.0155,0.0151,-0.012,0.035,0.0013,-0.0005,0.009]},index=[0, 1, 2, 3,4,5,6,7,8,9])

The idea is to multiply df1 and df2.shift(-1) (elementwise, not matrix multiplication) depending on the values of df1. If (df1>=0.50 or df1<=-0.50) then I multiply df1 and df2.shift(-1). Otherwise, I just put 0.

The desired result in this example should be the following (with columns names being the column names of df1 as well as df1 index):

df3=pd.DataFrame({'A': [0,0,0, -0.1212,0.010905,0,0,0, -0.012,'NaN'],'B': [0,0.015,0,-0.00755,0,0,0.00065,0,-0.0045,'NaN']},index=[0, 1, 2, 3,4,5,6,7,8,9])

I tried the following code:

import numpy as np
import pandas as pd
df3=np.where((df1>=0.50 or df1 <=-0.50),df1*df2.shift(-1),0)

And I get The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). Thank you.

Answer 1

Use | for bitwise OR with DataFrame constructor:

arr = np.where((df1>=0.50) | (df1 <=-0.50),df1*df2.shift(-1),0)
df3 = pd.DataFrame(arr, index=df1.index, columns=df1.columns)
print (df3)
          A        B
0  0.000000  0.00000
1  0.000000  0.01500
2  0.000000  0.00000
3 -0.121200 -0.00755
4  0.010905  0.00000
5  0.000000  0.00000
6  0.000000  0.00065
7  0.000000  0.00000
8 -0.012000 -0.00450
9       NaN      NaN

Numpy solution should be faster:

arr2 = np.concatenate([df2.values[1:, ], 
                       np.repeat(np.nan, len(df2.columns))[None, :]])

arr = np.where((df1.values>=0.50) | (df1.values <=-0.50),df1.values*arr2,0)
df3 = pd.DataFrame(arr, index=df1.index, columns=df1.columns)
print (df3)
          A        B
0  0.000000  0.00000
1  0.000000  0.01500
2  0.000000  0.00000
3 -0.121200 -0.00755
4  0.010905  0.00000
5  0.000000  0.00000
6  0.000000  0.00065
7  0.000000  0.00000
8 -0.012000 -0.00450
9       NaN      NaN