What test can I use to calculate significance of optimization lift?

Question

Given the following data for 12 users: username, number of deals for control, revenue from test, revenue from control

Here's an example of how the data looks like

Can you help me figure out how I can calculate the significance of the hypothesis that the test is more profitable (preferably using excel)? The measure I was thinking of using was the % of lift in revenues for each customer.

P.s. I have a background in statistics but not an expert so please keep it as simple as possible.

Answer 1

Since each pair of incomes refers to the same individual, you can perform a paired t-test.

Variable 1: Control income

Variable 2: Deals income

Then follow these instructions (copied here for posterity):

• In Excel, click Data Analysis on the Data tab.
• From the Data Analysis popup, choose t-Test: Paired Two Sample for Means.
• Under Input, select the ranges for both Variable 1 and Variable 2.
• In Hypothesized Mean Difference, you’ll typically enter zero. This value is the null hypothesis value, which represents no effect. In this case, a mean difference of zero represents no difference between the two methods, which is no effect.
• Check the Labels checkbox if you have meaningful variables labels in row 1. This option helps make the output easier to interpret. Ensure that you include the label row in step #3.
• Excel uses a default Alpha value of 0.05, which is usually a good value. Alpha is the significance level. Change this value only when you have a specific reason for doing so.
• Click OK.

Alternatively, you can indeed calculate the difference between the two incomes, and then perform a one sample t-test (assuming that the difference is zero). However, such a test is not available out-of-the-box in Excel; the procedure is described here.