I have a rather big dataframe (df) containing arrays and NaN in each cell, the first 3 rows look like this:
df:
A B C
X [4, 8, 1, 1, 9] NaN [8, 2, 8, 4, 9]
Y [4, 3, 4, 1, 5] [1, 2, 6, 2, 7] [7, 1, 1, 7, 8]
Z NaN [9, 3, 8, 7, 7] [2, 6, 3, 1, 9]
I already know (thanks to piRSquared) how to take the element-wise mean over rows for each column so that I get this:
element_wise_mean:
A [4.0, 5.5, 2.5, 1.0, 7.0]
B [5.0, 2.5, 7.0, 4.5, 7.0]
C [5.66666666667, 3.0, 4.0, 4.0, 8.66666666667]
Now I wonder how to get the respective standard deviation, any idea? Also, I don't understand yet what groupby() is doing, could someone explain its function in more detail?
df
np.random.seed([3,14159])
df = pd.DataFrame(
np.random.randint(10, size=(3, 3, 5)).tolist(),
list('XYZ'), list('ABC')
).applymap(np.array)
df.loc['X', 'B'] = np.nan
df.loc['Z', 'A'] = np.nan
element_wise_mean
df2 = df.stack().groupby(level=1)
element_wise_mean = df2.apply(np.mean, axis=0)
element_wise_sd
element_wise_sd = df2.apply(np.std, axis=0)
TypeError: setting an array element with a sequence.
Applying np.std using lambda with converting to
numpy array
is working for me :Or solution from comment:
I try explain more:
First reshape by
stack
- columns are added toindex
andMultiindex
is created.Then
groupby(level=1)
means group by first level ofMultiindex
- (by valuesA
,B
,C
) andapply
some function. Here it isnp.std
.Pandas not working with
array
s or lists very nice, so converting is necessary. (It looks like bug)