Significantly different values for MAPE and MAE on the same dataset

Question

I'm currently running a regression with various forecasting methods on the same dataset. For DT, the MAE is a little higher than for the AB model, while the MAPE is significantly higher for the AB model. How is this possible? I realize that a lower MAE does not necessarily lead to a lower MAE, but this difference is quite significant and got me wondering if I'm doing something wrong.

Decision Tree - MAE: 13.85

Decision Tree - MAPE: 59.77%

AdaBoost - MAE: 11.53

AdaBoost - MAPE: 76.23%

Here are the formulas that I use for the calculation:

MAE:

mae = sklearn.metrics.mean_absolute_error(y_pred, y_test)

MAPE:

def percentage_error(actual, predicted):
   res = np.empty(actual.shape)
   for j in range(actual.shape[0]):
       if actual[j] != 0:
           res[j] = (actual[j] - predicted[j]) / actual[j]
       else:
           res[j] = predicted[j] / np.mean(actual)
   return res

def mean_absolute_percentage_error(y_test, y_pred): 
   return np.mean(np.abs(percentage_error(np.asarray(y_test), np.asarray(y_pred)))) * 100

Source for the MAPE formula: https://stackoverflow.com/a/59033147/10603410

I hope someone is able to help with this! Thanks!

Answer 1

From my perspective, you can expect that sort of result as it depends on what the shape of your error distribution for each model looks like.

If your AB regressor is making bigger mistakes at low values of y_true than your DT regressor, but smaller mistakes at higher values of y_true, then you could easily expect to see a larger MAPE and potentially a lower MAE for the AB regressor.

As a motivating (and hugely contrived) example, let's say we have two models: model A and model B.

Model A makes larger mistakes at low values of y_true, but then smaller mistakes at high values of y_true:

#----- set up dummy y_true -----
def f(x):
    '''
    Returns y_true for a given input, x
    '''
    return 5*x**2

xs = np.linspace(-1,1,20)
y_true = f(xs)
#-------------------------------

def modelA_pred(x):
    '''
    Returns the predicted value for model A for a given input, x
    
    We're going to simply use a constant offset from y_true which will
    give us larger relative errors at lower values of y_true.
    '''
    return f(x) - 0.5

y_pred = modelA_pred(xs)

fig,(ax1,ax2) = plt.subplots(1,2, figsize=(10,5))
ax1.plot(xs,y_true,label='true')
ax1.plot(xs,y_pred,label='pred')
ax1.set_xlabel('x'); ax1.set_ylabel('y')
ax1.legend()

ax2.plot(xs,np.abs(y_pred-y_true),label='absolute error')
ax2.plot(xs,np.abs(y_pred-y_true)/y_true,label='absolute % error')
ax2.set_xlabel('x'); ax2.set_ylabel('abs(error)')
ax2.legend()

plt.plot();

print('MAPE: ', mean_absolute_percentage_error(y_true, y_pred))
print('MAE: ', mean_absolute_error(y_true, y_pred))

# MAPE:  436.3483933148212
# MAE:  0.5

While model B makes smaller mistakes at low values of y_true, but then larger mistakes at high values of y_true:

def modelB_pred(x):
    '''
    Returns the predicted value for model B for a given input, x
    
    We're going to use a relative offset here which will
    depend on the value of y_true - this will give us smaller
    absolute differences at lower values of y_true.
    '''
    return 0.5*f(x)

y_pred = modelB_pred(xs)

fig,(ax1,ax2) = plt.subplots(1,2, figsize=(10,5))
ax1.plot(xs,y_true,label='true')
ax1.plot(xs,y_pred,label='pred')
ax1.set_xlabel('x'); ax1.set_ylabel('y')
ax1.legend()

ax2.plot(xs,np.abs(y_pred-y_true),label='absolute error')
ax2.plot(xs,np.abs(y_pred-y_true)/y_true,label='absolute % error')
ax2.set_xlabel('x'); ax2.set_ylabel('abs(error)')
ax2.legend()

plt.plot();

print('MAPE: ', mean_absolute_percentage_error(y_true, y_pred))
print('MAE: ', mean_absolute_error(y_true, y_pred))

# MAPE:  50.0
# MAE:  0.9210526315789472