Seaborn showing values not found in Pandas columns

Question

Original dataframe:

dp.head(10)

Creating new dataframe using recommended selection method:

dtest = pd.DataFrame(dp[dp['numdept'].isin([3,6,8,10])]).dropna()
dtest.reset_index(drop =True, inplace = True)
dtest.head(10)

Testing to make sure that only the values in [3,6,8,10] are in dtest['numdept']:

print "numdept is 5:", dtest[dtest["numdept"].isin ([5])]
print "set of distinct values in the numdept column:", sorted(set(dtest['numdept'].tolist()))

>> numdept is 5: Empty DataFrame
>> Columns: [numgrade, numyear, numdept]
>> Index: []
>> set of distinct values in the numdept column: [3, 6, 8, 10]

Plotting:

plt.figure(figsize=(16, 8))
sb.boxplot(x="numyear", y="numgrade", hue="numdept", data=dtest)

Question: Why are the "nummdept" categories in the plot legend showing values other than 3,6,8,10?

Problem surfaced in an ipython notebook, but recurs even when I carry the code to a regular environment. Also tried to avoid seaborn related issues by using the suggestion here, to no avail.

Using Canopy 1.7.4.3348, jupyter 1.0.0-15, pandas 0.19.0-1 matplotlib 1.5.1-9 and seaborn 0.7.0-6

EDIT: On an impulse, inserted the following before the plotting code:

grouped = dtest.groupby(['numdept', 'numyear'])
grouped.mean()

The output has numdept values that should not exist in dtest.

Does this make it a pandas bug?

Answer 1

You are using a categorical variable. It appears the legend is based on the categories in the categorical variable, not the values that are actually present. A categorical variable may represent categories that don't actually occur in the data, and these categories are still shown in the legend.

As suggested in the documentation, you can do dtest.numdept.cat.remove_unused_categories() to remove the empty categories.

Answer 2

Why this is happening I am not certain, but there is an easy way to get it to use the desired [3, 6, 8, 10] legend you want.

#Create mock data
dp = pd.concat([pd.DataFrame(np.random.randint(1, 4, [100, 1])),
                pd.DataFrame(np.random.randint(1, 14, [100, 1])),
                pd.DataFrame([3.0]*20 + [6.0]*20 + [8.0]*20 + [10.0]*20 + [11.0]*20)], axis=1)
dp.columns = ["numyear", "numgrade", "numdept"]

dtest = pd.DataFrame(dp[dp['numdept'].isin([3,6,8,10])]).dropna()
dtest.reset_index(drop=True, inplace=True)

sns.boxplot(x="numyear", y="numgrade", hue="numdept", data=dtest,
            hue_order=[10, 3 , 8, 6])

Here I have added a hue_order and specified the order (I chose non-numeric order to emphasise this) and exact values I'd like to see. If specified [1, 2, 3, 6, 8, 10] it would give these as the legend.

Finally, you could generalise this nicely using the following,

sns.boxplot(x="numyear", y="numgrade", hue="numdept", data=dtest,
            hue_order=dtest.numdept.unique().sort(), width=0.2)