Python plot multiple z-test result with confidence interval (visualize A/B test results)

Question

what I want to plot

Hi, I want to visualize results for one A/B test. The experiment tracks 4 metrics, and I want to show them in one plot altogether. The schema of my dataframe is:

test_control | metric1 | metric2 | metric3 | metric4

Does anyone know how to plot, by matplotlib, pandas or seaborn?

Thanks in advance!

Answer 1

I found it's probably easier to be done in R. In python, I calculated the error bar and then used matplotlib.pyplot.errorbar to plot:

get CI

kpi_map = {'kpi':[], 'mean_diff':[], 'err':[], 'pval':[]}
for col in metrics:
    sp1 = df.loc[df['test_control']=='test'][col]
    sp2 = df.loc[df['test_control']=='control'][col]
    std1 = np.std(sp1, ddof=1)
    std2 = np.std(sp2, ddof=1)
    mean_diff_std = (std1**2/len(sp1) + std2**2/len(sp2)) **0.5
    mean_diff = sp1.mean() - sp2.mean()     
    kpi_map['kpi'].append(col)
    kpi_map['mean_diff'].append(mean_diff)
    kpi_map['err'].append(1.96*mean_diff_std)

plot

df_kpi = pd.DataFrame(data = kpi_map)
plt.errorbar(y=df_kpi['kpi'], x=df_kpi['mean_diff'], xerr=df_kpi['err'], fmt='o', elinewidth=2, capsize=4, capthick=2)