pandas grouped with aggregation stats across all dataframe columns

Question

I am grouping data in a pandas dataframe and using some aggregation functions to generate results data. Input data:

A  B  C  D  E  F
0  aa  5  3  2  2  2
1  aa  3  2  2  3  3
2  ac  2  0  2  7  7
3  ac  9  2  3  8  8
4  ac  2  3  7  3  3
5  ad  0  0  0  1  1
6  ad  9  9  9  9  9
7  ad  6  6  6  6  6
8  ad  3  3  3  3  3

The pandas grouped function seems to only operate on one column at a time but I want to generate the statistic on all columns in my df. For example, I can use the function grouped['C'].agg([np.mean, len]) to generate the statistics on column 'C' but what if I want to generate these statistics on all columns A - F?

The output from this is:

    A  count_C    mean_C
0  aa        2  2.500000
1  ac        3  1.666667
2  ad        4  4.500000

But what I want is:

    A  count_B    mean_B  count_C    mean_C  count_D  mean_D  etc...
0  aa        2  4.000000        2  2.500000        2     2.0  etc...
1  ac        3  4.333333        3  2.500000        3     4.0
2  ad        4  4.500000        4  2.500000        4     4.5

Is there any easy way to do the group by with aggregation in a single command? If not, is there an easy way to iterate over all columns and merge in new aggregation statistics results for each column?

Here's my full code so far:

import pandas as pd
import numpy as np
import pprint as pp

test_dataframe = pd.DataFrame({
    'A' : ['aa', 'aa', 'ac', 'ac', 'ac', 'ad', 'ad', 'ad', 'ad'],
    'B' : [5, 3, 2, 9, 2, 0, 9, 6, 3],
    'C' : [3, 2, 0, 2, 3, 0, 9, 6, 3],
    'D' : [2, 2, 2, 3, 7, 0, 9, 6, 3],
    'E' : [2, 3, 7, 8, 3, 1, 9, 6, 3],
    'F' : [2, 3, 7, 8, 3, 1, 9, 6, 3]
})

#group, aggregate, convert object to df, sort index
grouped = test_dataframe.groupby(['A'])
grouped_stats = grouped['C'].agg([np.mean, len])
grouped_stats = pd.DataFrame(grouped_stats).reset_index()
grouped_stats.rename(columns = {'mean':'mean_C', 'len':'count_C'}, inplace=True)
grouped_stats.sort_index(axis=1, inplace=True)

print "Input: "
pp.pprint(test_dataframe)

print "Output: "
pp.pprint(grouped_stats)

Answer 1

You don't have to call grouped['B'] grouped['C'] one by one, simply pass your entire groupby object and pandas will apply the aggregate functions to all columns.

import pandas as pd

test_dataframe = pd.DataFrame({
    'A' : ['aa', 'aa', 'ac', 'ac', 'ac', 'ad', 'ad', 'ad', 'ad'],
    'B' : [5, 3, 2, 9, 2, 0, 9, 6, 3],
    'C' : [3, 2, 0, 2, 3, 0, 9, 6, 3],
    'D' : [2, 2, 2, 3, 7, 0, 9, 6, 3],
    'E' : [2, 3, 7, 8, 3, 1, 9, 6, 3],
    'F' : [2, 3, 7, 8, 3, 1, 9, 6, 3]
})
agg_funcs = ['count', 'mean']
test_dataframe = test_dataframe.groupby(['A']).agg(agg_funcs)

columns = 'B C D E F'.split()
names = [y + '_' + x for x in columns for y in agg_funcs]
test_dataframe.columns = names

Out[89]: 
    count_B  mean_B  count_C  mean_C  count_D  mean_D  count_E  mean_E  count_F  mean_F
A                                                                                      
aa        2  4.0000        2  2.5000        2     2.0        2    2.50        2    2.50
ac        3  4.3333        3  1.6667        3     4.0        3    6.00        3    6.00
ad        4  4.5000        4  4.5000        4     4.5        4    4.75        4    4.75